Data Processing Evolved: OpenLineage with Willy Lulciuc

Willy Lulciuc (@wslulciuc) is a pioneer in data engineering and one of the creators of OpenLineage, the open-source framework for data lineage collection and analysis. It enables consistent collection of lineage metadata, giving engineers a better perspective on how data is produced and used, so they can better solve complex problems. Join us to learn about the evolution of data processing, the role of AI, and the future of data engineering.Subscribe to Contributor on Substack for email notifications!In this episode we discuss:Willy’s journey from WeWork to oleanderThe role of AI in data engineeringChallenges in data processing and observabilityA vision for the future of data platformsLinks:OpenLineageoleanderPeople mentioned:Julien Le Dem (@J_)

Andrew Smith (Github: @asmith-plainsight) dives deep into OpenFilter, the open-source framework for building computer vision workflows. Andrew is the CTO of Plainsight which is a leader in modern computer vision infrastructure. Tune in to find out how OpenFilter is simplifying and revolutionizing computer vision applications.Subscribe to Contributor on Substack for email notifications!In this episode we discuss:The challenges in deploying computer vision, particularly in edge environmentsHow Andrew’s time at Google influenced the development of OpenFilterThree problems Andrew sought to solve before open-sourcing the projectBest use cases for OpenFilterThe surprising nascence of computer vision applicationsLinks:OpenFilterPlainsightOpenLineagePeople mentioned:Kit Merker (@KitMerker)Willy Lulciuc (@wslulciuc)

Pete Hunt (@floydophone) joins Eric Anderson (@ericmander) to talk about Dagster, the open-source data orchestration platform built for productivity. In 2021, Eric interviewed Dagster’s founder, Nick Schrock, about an earlier iteration of the product. Now, four years later, Pete has become the CEO of Dagster Labs and has all the updates on how this platform has changed along with the industry and the world.Subscribe to Contributor on Substack for email notifications!In this episode we discuss:How and why Dagster has invested in software-defined data assetsNailing a layered approach between full-code engineering and low-code solutionsThe changing role of AI in data pipelinesDagster’s renewed effort to build observability toolsLinks:DagsterPeople mentioned:Nick Schrock (@schrockn)Other episodes:Dagster with Nick Schrock

Mark Fussell (@mfussell) is the co-creator of Dapr, the open-source runtime system designed to support cloud native and serverless computing. Dapr provides APIs that simplify the development of distributed applications, providing essential functionalities like service invocation, pub/sub messaging, and observability. Subscribe to Contributor on Substack for email notifications!In this episode we discuss:The difference between Dapr and a service meshHow Dapr is beneficial for platform engineering teamsFrom Microsoft to Diagrid in the startup worldThe significance of Dapr’s graduation from the CNCFLinks:DaprDiagridKEDAPeople mentioned:Yaron Schneider (@yaronschneider)Other episodes:Istio with Sven MawsonLinkerD with William MorganRobust Observability: OpenTelemetry with Austin Parker

Kee Jefferys (@JefferysKee) is the technical co-founder of Session, the end-to-end encrypted messenger that emphasizes user confidentiality and anonymity. Session uses a blockchain-based decentralized network for message transmission and is open-source, so the system can be run entirely by its community. In this episode, Kee explains the importance of reducing metadata, the challenges of centralization, and the role of incentives in creating a decentralized network. Contributor is looking for a community manager! If you want to know more, shoot us an email at [email protected] to Contributor on Substack for email notifications!In this episode we discuss:Why the demand for better privacy has increased How Session differs from other encrypted messaging platforms Combining encryption and incentive modelsWhy open-source is so critical to messaging applications trying to prove their securityHow Session could outlast its developersLinks:Session

Clément Salaün (@superzamp) is the co-founder and CTO of Formance, the open-source platform which is building an agnostic infrastructure for the future of the financial Internet. Formance is divided into several modules which allow fintech engineers to build and operate complex flow of funds, weaving together multiple payment rails with internal ledger accounts. Tune in to find out what it takes to build a scalable and extensible financial core system. Contributor is looking for a community manager! If you want to know more, shoot us an email at [email protected]. Subscribe to Contributor on Substack for email notifications! In this episode we discuss: How Clem got into fintech through Minecraft Thoughts on the European open-source scene  Formance’s ratio of community size to Github stars The influence of crypto on Formance What it means to split an open-source project into many repositories Links: Formance

Gideon Mendels (Github: @gidim) is the co-founder and CEO of Comet, the end-to-end model evaluation platform for AI developers. Among the tools in the Comet ecosystem is Opik, an open-source solution for evaluating, testing and monitoring LLM applications. Opik allows users to log traces and spans, define and compute evaluation metrics, score LLM outputs, compare performance across app versions, and more. As a true open-source project, its full featureset is available for use by anyone, completely free. Contributor is looking for a community manager! If you want to know more, shoot us an email at [email protected]. Subscribe to Contributor on Substack for email notifications! In this episode we discuss: How Opik’s popularity blew up beyond the Comet team’s expectations Why CI/CD is especially important in an end-to-end platform Gideon’s “severe allergy” to “fake open-source” offerings Why the number of dedicated machine learning engineers is actually going down Eric’s thoughts on what it means for venture capital to invest in the LLM space Links: Opik Comet

Hanson Ho (@bidetofevil) is an Android Architect at Embrace, the mobile-first observability solution built on OpenTelemetry. Embrace began as a proprietary platform but went open-source at the end of 2023. Hanson shares about how the project is still at the beginning of its open-source journey and why his team is committed to collaborative development with the larger community of OpenTelemetry. Contributor is looking for a community manager! If you want to know more, shoot us an email at [email protected]. Subscribe to Contributor on Substack for email notifications! In this episode we discuss: The missing piece in mobile observability tooling How Embrace switched its perspective after going open-source 3 use cases for Embrace Links: Embrace OpenTelemetry Other Episodes: Robust Observability: OpenTelemetry with Austin Parker

Stephan Ewen (@StephanEwen) is the co-founder of Restate, the open-source workflow-as-code engine. Restate is lightweight, simple, and provides durable execution. Before Restate, Stephan co-created Apache Flink, the open-source stream processing framework. Lessons learned from Flink have heavily influenced the development of Restate, although Stephan says they have exact opposite use cases. Contributor is looking for a community manager! If you want to know more, shoot us an email at [email protected]. Subscribe to Contributor on Substack for email notifications! In this episode we discuss: The history of Flink and the impact of the 2016 U.S. election Why tooling for real-time transactional problems has historically had room for improvement What constitutes “modern” workflow engines Can you use Restate for any use case? Moving from a large company to a small startup as an open-source developer Links: Restate Apache Flink People mentioned: Kostas Tzoumas (@kostas_tzoumas) Other episodes: Temporal with Maxim Fateev

Eyal Solomon (@EyalSolomo44643) is the CEO and co-founder of Lunar, an open-source platform which bills itself as the “first reverse API gateway.” Lunar allows engineering teams to monitor, manage, and optimize API consumption. According to Eyal, it’s very easy to integrate with APIs, but difficult to keep them maintained, and there was a clear need for a generic solution to control and scale every API consumed in production.  Contributor is looking for a community manager! If you want to know more, shoot us an email at [email protected]. Subscribe to Contributor on Substack for email notifications! In this episode we discuss: How most companies think their API maintenance is a unique problem The importance of managing API consumption in the face of the AI revolution Why Eyal and his team decided to open-source Lunar Future plans for Lunar, including the development of autonomous optimization and pre-built flows Eyal’s thoughts on how to start conversations with potential enterprise clients Links: Lunar People mentioned: Roy Gabbay (LinkedIn)