0:00
27:45
Recuar 15 segundos
Avançar 15 segundos

Airflow 3.4 is here, and it brings task loops, multi-language Dag SDKs, and a stable 1.0 release of the Common AI provider. Marc Lamberti, Education Content Lead at Astronomer, joins Kenten to walk through what's new, what it means for AI orchestration, and which smaller features are worth paying attention to.


Key Takeaways:


00:00 Introduction.
01:12 Task loops (AIP-111) let you repeat tasks in a task group until a condition is met or a max iteration count is reached.
01:56 How task loops work under the hood: a loop method on a task group, max iterations plus an until condition, and task instances recreated per iteration like dynamic task mapping. A Dag is still a Dag.
04:34 Use cases for task loops: pagination, agent feedback loops, model training until a metric converges, and data quality retries.
06:26 Multi-language Dag SDKs in 3.4. You can now write full Dags in Java, Go, or TypeScript, with a brand new TypeScript SDK.
09:41 Why the TypeScript SDK matters for AI orchestration. Agent code written in TypeScript can be orchestrated directly in Airflow.
11:17 Common AI provider 1.0: durable execution, an object storage toolset, and cost controls for agent runs. Supports Airflow 2.11 and Airflow 3.
14:38 Inside the Astronomer EDU agent, built with the LLM decorator, agent decorator, and durable execution to triage and draft answers for support tickets.
18:58 Smaller features worth a look: Dag draining, multiple Dag folders in the UI, AssetAndTimeSchedule, and the return of custom graph colors.
23:06 Advice for teams still on Airflow 2: Airflow 2 has reached end of life. Use ruff rules, the Airflow CLI, or the Ador data engineering agent to upgrade.


Resources Mentioned:



Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Orchestration #Airflow

Mais episódios de "The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI"