The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Generic AI coding tools consistently fall short on Airflow work. Only 9% of users in last year's Airflow survey said they were satisfied with them. In this episode, Yetunde Dada, Senior Director of Product Management at Astronomer, and Stephanie Niu, Senior Product Manager at Astronomer, join Kenten to walk through Otto, Astronomer's data engineering agent built specifically for Airflow, and how it approaches investigations, upgrades, and evaluation. Otto is available now in Labs and will enter Preview on November 17th.Key Takeaways:00:00 Introduction.02:12 Why generic LLMs struggle with Airflow: version and provider matrices, operator deprecations, and training data skewed toward older public patterns.04:55 What Otto is: an agent harness built for Airflow and Astro, with system prompts, upgrade knowledge bases, and investigation expertise.06:38 Where Otto lives today: the Astro UI, Astro IDE, Astro CLI, VS Code and Cursor extensions, and the Astro MCP.10:34 How Otto runs investigations by mirroring what a senior data engineer checks: task logs, Dag code, worker and scheduler state, and run history.12:12 Event-driven investigations and self-healing: Otto kicks off as soon as a Dag fails and can trigger downstream processes via API.15:36 Evaluating Otto in production: golden datasets, evidence utilization, reasoning chains, and abstention quality.17:01 The Astronomer Data Engineering Benchmark (DEBench): why existing SQL and dbt benchmarks miss orchestration, and how Otto compares to Claude Code and Codex.20:04 What's next: an agent-friendly Astro CLI, Otto in the Airflow UI, and Routines for scheduled and event-triggered Otto work.Resources Mentioned:AstronomerOtto, Astronomer's data engineering agentAstro ObserveDEBench on GitHubThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Orchestration #Airflow

Semiconductor chip design is not where most people expect to find Apache Airflow, but Qualcomm's CPU team runs millions of pre-silicon simulations and post-silicon validation jobs through it every day. In this episode, Kenten Danas is joined by Dheeraj Turaga, Principal Engineer at Qualcomm and an Apache Airflow committer, to talk through two very different Airflow use cases: orchestrating semiconductor design workflows across on-prem compute farms, and driving robotic handlers in post-silicon validation labs.Key Takeaways:00:00 Introduction.01:36 What Qualcomm does and where the CPU team fits in the Snapdragon product line.04:07 Reframing Airflow as a general-purpose workflow orchestration platform, not just a data pipeline tool.05:26 How semiconductor design workflows look as Dags: simulations, power analysis, gate-level conversion, and silicon placement.09:28 Bringing Airflow in-house and bridging the 20% gap between data pipeline assumptions and chip design needs.11:36 Using celery with small persistent workers that spawn ephemeral workers to dynamically consume on-prem compute.13:25 Orchestrating silicon validation labs: limited chips, limited workstations, and multiple teams timesharing hardware.16:40 Targeting specific lab machines using the Edge Executor with machine-specific queues and pools.18:45 Managing failures in physical hardware workflows using the new task state store and pluggable retry policies in Airflow 3.3.21:12 Contributing a new Dag drain state to Airflow 3.4 that lets existing runs finish before pausing a Dag.Resources Mentioned:Apache AirflowEdge ExecutorAirflow Summit 2025 session: Semiconductor Chip Design Workflow Orchestration with AirflowAirflow Summit 2026 session: Dags Move Robots - Closed-Loop Orchestration for Silicon Validation Labs with AirflowQualcomm SnapdragonDheeraj Turaga on LinkedInThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Orchestration #Airflow

Large-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational controls. In this episode, Kenten Danas is joined by Bhanu Prakash, Technical Lead at HCL Tech, to talk through how his team uses Airflow to orchestrate end-to-end data and platform migrations to the cloud, generate Dags dynamically for micro-batch workloads, and automate incident response through a ServiceNow integration.Key Takeaways:00:00 Introduction.01:19 Bhanu's role as an Airflow and ETL SME at HCL Tech, leading architecture and delivery across data engineering engagements.02:30 Six years of Airflow use cases: data migration, scheduled analytics, customer segmentation refreshes, and web-application-adjacent data ops.03:19 When to recommend Airflow to clients: everything except real-time, with cloud-agnostic portability as the deciding factor over ADF and Glue.06:19 Where Airflow fits in the current cloud migration project: watermarking, cross-reference tables, on-prem to cloud movement, and downstream Databricks and Snowflake processing.08:01 Running through Airflow versions 2.3, 2.11, and now 3.2, and how backwards compatibility kept 99% of the code unchanged.11:10 Building a Dag generation utility driven by JSON config, plus runtime Dag creation for micro-batching (illustrated with an e-commerce example).14:45 Making Dags configurable for clients with pull and push triggering, and the blue-green pattern for promoting Dags through dev, stage, pre-prod, and prod.16:28 Using Airflow's ServiceNow integration to build an auto-resolution engine that uses LLMs to build context from past incidents and apply gated fixes.18:12 A wish-list feature: making Airflow an AI-first platform where cross-tool debugging happens inside the Airflow UI.Resources Mentioned:HCL TechApache AirflowServiceNowDatabricksSnowflakeThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Orchestration #Airflow

At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken Obata, Senior Software Engineer at Lyft, joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it.Key Takeaways:00:00 Introduction.02:49 How Airflow responsibility is split at Lyft between the infrastructure org (scaling, version control, Astronomer partnership) and product teams that write Dags, maintain metadata, and orchestrate ETL and MLOps.04:00 Contrasting data problems at large ownership-first orgs (data reconciliation) versus mid-size lean-data orgs like Lyft (data engineers as the bottleneck).07:04 Why Lyft treats Airflow as a service layer rather than a scheduled ETL system, including productionizing Jupyter notebooks from science teams.10:14 The experience running Airflow on Astro, including the 2 to 3 upgrade and consolidated observability in one place.12:13 The two problems driving self-service Dags: supporting 30+ users with a small data engineering team while meeting SOX compliance on driver pay changes.14:38 How the GitHub Actions and Airflow integration works bidirectionally, blocking PRs until validation completes.18:28 Why Lyft chose event-driven over polling: easier upstream-to-downstream context passing and avoiding wasted compute on on-demand work.20:23 The two hardest production challenges: GitHub's tight coupling of check IDs to commit SHAs, and multi-hour simulations racing user rebases. How Lyft built an external context store to solve both.25:15 Why Airflow 3.3's pluggable retry policy and task state store would unlock efficiency gains for transient GitHub failures and long-running Spark simulations.26:39 Looking ahead to task loops and agentic operators for optimization problems with multiple correct answers.Resources Mentioned:LyftApache AirflowAstro by AstronomerGitHub ActionsThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap.Key Takeaways:00:00 Introduction.02:22 Insignis data team as a "newspaper" for the business. Aniruddha describes the data team's role: consuming data from operational systems and external sources like Salesforce and Ringover, and delivering insight without owning business logic.08:07 Airflow as the orchestrator, not the executor. Airflow sits at the heart of the platform, orchestrating SQL pipelines, API ingestion, and even acting as an observability layer over real-time pipelines built elsewhere.12:19 Observability as definition of done. Why a green Airflow UI is not observability, and how Insignis builds independent dashboards to catch silently dropped data.14:38 Choosing managed Airflow. How abstracting away the scheduler, workers, and database frees a small team to focus on code and documentation.20:04 DAG patterns that keep costs down. CI/CD enforcement, failure callbacks, mindful DAG boundaries, and periodic purging of logs and metadata.22:06 Tariff and Maestro. Two internal tools Insignis built: Tariff to surface SQL pipeline costs, and an AI-powered tool called Maestro to recommend worker, scheduler, and database sizing.25:40 Exploring AI with Airflow. MLOps use cases around client behavior and the appeal of self-healing pipelines.28:20 Wishlist for Airflow. Event-based DAG parsing so code changes surface in the UI closer to instantaneously.Resources Mentioned:Annual Airflow survey- Insignis- Apache Airflow- Astronomer- Building a Low-Cost Scalable Airflow Platform for Small Teams- Google BigQueryThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI.Key Takeaways:00:00 Introduction.01:40 Idelic's mission: collecting data from North American trucking fleets and using it to identify risky drivers so fleets can intervene before accidents happen.02:59 Where Airflow fits: Idelic runs on Astro, with each customer/integration combination as its own DAG.03:56 The reality of vendor integrations: APIs, SFTP, email, screen scraping, and even executables installed on customer servers to push data out.06:11 How the DAG count grew and where things started to hurt as a small team tried to keep up.08:49 The standardized framework: abstract methods that wrap common Airflow functions and generate task groups for each ETL phase, plus standardized logging across vendors.11:15 CI/CD with Jenkins: per-PR Airflow deployments that QA can spin up on demand through a Jira hook and tear back down when done.13:31 Letting customers create their own DAGs through a wizard in Safety Suite, with plans for customer-triggered runs, filtered result review, and human-in-the-loop approval.15:37 Using the Airflow API to trigger customer DAGs, and why Idelic pushes run data into its own database instead of relying on the Airflow logs API.18:58 What Matt wants next from Airflow: task and DAG level trend tracking, and more from Otto on self-healing DAGs.Resources Mentioned:Airflow Survey: astronomer.typeform.com/airflowsurvey26Astro by Astronomer: astronomer.ioApache Airflow: airflow.apache.orgOtto (Astronomer's data engineering agent): https://www.astronomer.io/product/otto/Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#ai #orchestration #airflow

Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to manage LoRA fine-tuning cycles.Key Takeaways:00:00 Introduction.01:47 From modeling to pipelines. Chhayank shares how leading a readmission risk model across hospitals with different EHR systems turned him into a "pipeline person."03:14 Day-to-day as an LLM engineer. Serving with ONNX Runtime, failure design, and agentic tooling with LangGraph. The model is only about one-fifth of the work.05:03 The sensitive data problem. Why hosted LLM APIs are off the table for PHI and payments data, and why the real challenge is running Llama 3 nightly with enough checks that downstream models can trust the output.07:34 Where Airflow fits. Sensors per site, dependency ordering, retries and backfills, and run records as the "paper trail" for audits.10:03 DAG structure. Four stages: ingest and normalize, inference on GPU pods via the KubernetesPodOperator, validation, and load to the feature store.12:32 Loud vs silent failures. OOM kills and timeouts are easy. Silent failures where the output looks clean but is wrong are the scary category, handled with Pydantic schemas, cross-checks against structured records, and distribution monitoring.16:45 LoRA fine-tuning in its own DAG. A separate monthly or quarterly DAG pulls labeled data, fine-tunes with Hugging Face PEFT on top of Llama 3, and only writes a new adapter if it beats the frozen holdout.20:33 Hardest production lessons. Pin everything (model version, temperature, features, seed), match parallelism to GPU capacity, and invest in traceability so you can answer "which adapter produced this row" six months later.23:00 Airflow wishlist. First-class GPU awareness so teams stop reinventing it through pools and queues, and treating model versions as Airflow assets.Resources Mentioned:- Annual Airflow survey- Airflow Summit talk: From Hours to Minutes: Orchestrating Local LLMs for Sensitive Data Pipelines with Apache Airflow- Llama 3- Hugging Face PEFT- ONNX Runtime- LangGraph- PydanticThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Orchestration #Airflow

Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope.Key Takeaways:00:00 Introduction.01:16 Anish's inside-out path into Airflow, starting from source code and docs rather than running pipelines in production.03:13 Why security engineering flips the usual software question from "what can this do?" to "what can anyone make this to?"04:49 How connection testing worked before, why it was disabled in Airflow 2.7 after a CVE, and how the redesign moves execution off the API server.06:26 Running connection tests on the worker in Airflow 3 so credentials never leave the API server boundary.07:50 What the mailing list and architecture review looked like, including a first-round rejection and eventual consensus.10:00 The JWT logout problem in Airflow 3 and the revocation table that makes logout an actual logout.13:12 A production incident with a GPU pipeline task hitting 403 after 15 minutes, and why it required splitting the token in two.15:41 How the workload token and the auth token now have different lifetimes and scopes, tied to the task queue timeout.19:53 Advice for new Airflow contributors: start small, don't fear the PR review, and treat feedback as mentorship.Resources Mentioned:Annual Airflow surveyApache AirflowAirflow on GitHubAirflow SlackAnish Giri at Airflow SummitThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Orchestration #Airflow

Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates without knowing Python.Key Takeaways:00:00 Introduction.01:47 What itti and Grupo Vázquez do, and the Data Engineering Center of Excellence's mandate to build a 360-degree view of the customer across more than 30 companies.02:56 How Airflow fits in as a central task orchestrator (not a processing engine) across around 300 production DAGs.04:25 Managing enterprise-scale Airflow: preferring Airflow-as-a-service, plus enabling self-service for non-technical users through YAML.05:44 Why itti built a second, more opinionated YAML framework after DAG Factory-style customization created a code review bottleneck.07:14 More than 80% of new DAGs are now created with the new framework because it's simply faster.07:54 How the framework works end to end: Python DAGs, Jinja templates, YAML configs, and CI/CD compilation.09:10 A Google Sheets ingestion example that shows how prevalidation, download, and processing tasks are hidden behind a simple YAML config to preserve reliability.10:14 Building an in-house data quality tool that tests per partition instead of full-scanning tables, triggered via custom Airflow operators.11:17 Custom operators for dbt, the in-house data quality tool, and AWS services like QuickSight dashboard refreshes, and how OSS Airflow makes them portable across instances.15:35 Spec-driven development with a fork of GitHub Spec Kit so business users can describe what they want and let agents generate DAGs against certified templates.17:24 Slack-native error routing: every DAG has an owner team, common errors ship with explanations, and only deep issues escalate to the central team.19:29 Where they're heading next: agent-triggered pull requests for self-healing pipelines.20:22 Airflow 3 wishlist: backfill improvements, event-driven orchestration for streaming pipelines, and Human in the Loop for generative AI DAGs.Resources Mentioned:Apache AirflowApache SparkDAG FactoryGitHub Spec KitThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow

Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components.Key Takeaways:00:00 Introduction.02:29 How ATC Drivetrain uses Airflow across thousands of DAGs to orchestrate ETL/ELT jobs, data quality checks, and production reporting for a complex automotive remanufacturing environment.04:08 Defining self-healing: pipelines that identify a known failure, decide whether they can recover safely, execute an approved action, and validate the result, all without paging an engineer.07:03 The five-layer self-healing architecture: observe, classify, policy, recover, and validate.10:00 Where AI fits in the recovery loop: classification, context gathering, and recommendations, but never bypassing operational policies or approval steps.13:00 A concrete before/after: a currency-exchange failure caught overnight by AI-assisted recovery that saved four hours of downtime and roughly 120K.14:27 Confidence levels and success rates: about 95% of small failure modes recover on their own.15:34 AI-assisted troubleshooting at scale: how contextual log analysis and recommended actions save engineers from digging through thousands of log lines.20:13 Orchestrating AI agents through Airflow: treating agents as bounded, governed workflow components with human approval and confidence-based stop conditions.23:45 What Kumuda wants next from Airflow: stronger AI agent governance, standardized tracking of prompts and tool calls, and more flexible event-driven execution.Resources Mentioned:Orchestrate EverythingApache AirflowATC DrivetrainThanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.#AI #Automation #Airflow