Talk Python To Me podcast

#330: Apache Airflow Open-Source Workflow with Python

0:00
1:07:50
Recuar 15 segundos
Avançar 15 segundos
If you are working with data pipelines, you definitely need to give Apache Airflow a look. This pure-Python workflow framework is one of the most popular and capable out there. You create your workflows by writing Python code using clever language operators and then you can monitor them and even debug them visually once they get started.

Stop writing manual code or cron-job based code to create data pipelines check out Airflow. We're joined by three excellent guests from the Airflow community: Jarek Potiuk, Kaxil Naik, and Leah Cole.

Links from the show

Jarek Potiuk: linkedin.com
Kaxil Naik: @kaxil
Leah Cole: @leahecole

Airflow site: airflow.apache.org
Airflow on GitHub: github.com
Airflow community: airflow.apache.org
UI: github.com
Helm Chart for Apache Airflow: airflow.apache.org
Airflow Summit: airflowsummit.org
Astronomer: astronomer.io
Astronomer Registry (Easy to search for official and community Providers): registry.astronomer.io
REST API: airflow.apache.org
Contributing: github.com
Airflow Loves Kubernetes talk: airflowsummit.org
Watch this episode on YouTube: youtube.com
Episode transcripts: talkpython.fm

---------- Stay in touch with us ----------
Subscribe on YouTube (for live streams): youtube.com
Follow Talk Python on Twitter: @talkpython
Follow Michael on Twitter: @mkennedy

Sponsors
Talk Python Training
AssemblyAI

Mais episódios de "Talk Python To Me"

  • Talk Python To Me podcast

    #338: Using cibuildwheel to manage the scikit-HEP packages

    1:17:44

    How do you build and maintain a complex suite of Python packages? Of course, you want to put them on PyPI. The best format there is as a wheel. This means that when developers use your code, it comes straight down and requires no local tooling to install and use. But if you have compiled dependencies, such as C or FORTRAN, then you have a big challenge. How do you automatically compile and test against Linux, macOS (Intel and Apple Silicon), Windows, and so on? That's the problem cibuildwheel is solving. On this episode, you'll meet Henry Schreiner. He is developing tools for the next era of the Large Hadron Collider (LHC) and is an admin of Scikit-HEP. Of course, cibuildwheel is central to this process. Links from the show Henry on Twitter: @HenrySchreiner3 Henry's website: iscinumpy.gitlab.io Large Hadron Collider (LHC): home.cern cibuildwheel: github.com plumbum package: plumbum.readthedocs.io boost-histogram: github.com vector: github.com hepunits: github.com awkward arrays: github.com Numba: numba.pydata.org uproot4: github.com scikit-hep developer: scikit-hep.org pypa: pypa.io CLI11: github.com pybind11: github.com cling: root.cern Pint: pint.readthedocs.io Python Wheels site: pythonwheels.com Build package: pypa-build.readthedocs.io Mac Mini Colo: macminicolo.net scikit-build: github.com plotext: pypi.org Code Combat: codecombat.com clang format wheel: github.com cibuildwheel examples: cibuildwheel.readthedocs.io Cling in LLVM: root.cern New htmx course: talkpython.fm/htmx Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Talk Python Training AssemblyAI
  • Talk Python To Me podcast

    #337: Kedro for Maintainable Data Science

    1:03:14

    Have you heard of Kedro? It's a Python framework for creating reproducible, maintainable and modular data science code. We all know that reproducibility and related topics are important ones in the data science space. The freedom to pop open a notebook and just start exploring is much of the magic. Yet, that free-form style can lead to difficulties in versioning, reproducibility, collaboration, and moving to production. Solving these problems is the goal of Kedro. And we have 3 great guests from the Kedro community here to give us the rundown: Yetunde Dada, Waylon Walker, and Ivan Danov. Links from the show Waylong on Twitter: @_WaylonWalker Yetunda on Twitter: @yetudada Ivan on Twitter: @ivandanov Kedro: kedro.readthedocs.io Kedro on GitHub: github.com Join the Kedro Discord: discord.gg Articles about Kedro by Waylan: waylonwalker.com Kedro spaceflights tutorial: kedro.readthedocs.io “Hello World” on Kedro: kedro.readthedocs.io Kedro Viz: quantumblacklabs.github.io Spaceflights Tutorial video: youtube.com Dynaconf package: dynaconf.com fsspec: Filesystem interfaces for Python: filesystem-spec.readthedocs.io Neovim: neovim.io Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Tabnine Talk Python Training AssemblyAI
  • Talk Python To Me podcast

    Não percas um episódio de Talk Python To Me e subscrevê-lo na aplicação GetPodcast.

    iOS buttonAndroid button
  • Talk Python To Me podcast

    #336: Terminal magic with Rich and Textual

    59:12

    Have you heard of the package Rich? This library allows you to create very, well, rich terminal-based UIs in Python. When you think of what you can typically build with basic print statements, that may seem quite limited. But with Rich, imagine justified tables, progress bars, rendering of markdown, and way more. This is one of the fastest growing projects in the Python space these days. And the creator, Will McGugan is here to give is the whole history and even a peak at the future of Rich and a follow on library called Textual. Links from the show Will on Twitter: @willmcgugan Rich: github.com Textual: github.com Pyfilesystem: pyfilesystem.org A Look At – and Inside – Textual Video: youtube.com ObjExplore: reposhub.com ghtop: ghtop.fast.ai Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Shortcut Talk Python Training AssemblyAI
  • Talk Python To Me podcast

    #335: Gene Editing with Python

    58:20

    Gene therapy holds the promise to permanently cure diseases that have been considered life-long challenges. But the complexity of rewriting DNA is truly huge and lives in its own special kind of big-data world. On this episode, you'll meet David Born, a computational biologist who uses Python to help automate genetics research and helps move that work to production. Links from the show David on Twitter: @Hypostulate Beam Therapeutics: beamtx.com AWS Cloud Development Kit: aws.amazon.com/cdk Jupyter: jupyter.org $1,279-per-hour, 30,000-core cluster built on Amazon EC2 cloud: arstechnica.com Luigi data pipelines: luigi.readthedocs.io AWS Batch: aws.amazon.com/batch What is CRISPR?: wikipedia.org SUMMIT supercomputer: olcf.ornl.gov/summit Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Shortcut Talk Python Training AssemblyAI
  • Talk Python To Me podcast

    #334: Microsoft Planetary Computer

    59:46

    On this episode, Rob Emanuele and Tom Augspurger join us to talk about building and running Microsoft's Planetary Computer project. This project is dedicated to providing the data around climate records and the compute necessary to process it with the mission of help use all understand climate change better. It combines multiple petabytes of data with a powerful hosted Jupyterlab notebook environment to process it. Links from the show Rob Emanuele on Twitter: @lossyrob Tom Augspurger on Twitter: @TomAugspurger Video at example walkthrough by Tom if you want to follow along: youtube.com?t=2360 Planetary computer: planetarycomputer.microsoft.com Applications in public: planetarycomputer.microsoft.com Microsoft's Environmental Commitments Carbon negative: blogs.microsoft.com Report: microsoft.com AI for Earth grants: microsoft.com Python SDK: github.com Planetary computer containers: github.com IPCC Climate Report: ipcc.ch Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Shortcut Talk Python Training AssemblyAI
  • Talk Python To Me podcast

    #333: State of Data Science in 2021

    1:03:59

    We know that Python and data science are growing in lock-step together. But exactly what's happening in the data science space in 2021? Stan Seibert from Anaconda is here to give us a report on what they found with their latest "State of Data Science in 2021" survey. Links from the show Stan on Twitter: @seibert State of data science survey results: know.anaconda.com A Python Data Scientist’s Guide to the Apple Silicon Transition: anaconda.com Numpy M1 Issue: github.com A Python Developer Explores Apple's M1 (Michael's video): youtube.com Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Shortcut Masterworks.io AssemblyAI Talk Python Training
  • Talk Python To Me podcast

    #332: Robust Python

    1:11:56

    Does it seem like your Python projects are getting bigger and bigger? Are you feeling the pain as your codebase expands and gets tougher to debug and maintain? Patrick Viafore is here to help us write more maintainable, longer-lived, and more enjoyable Python code. Links from the show Pat on Twitter: @PatViaforever Robust Python Book: oreilly.com Typing in Python: docs.python.org mypy: mypy-lang.org SQLModel: sqlmodel.tiangolo.com CUPID principles @ relevant time: overcast.fm Stevedore package: docs.openstack.org Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Shortcut Masterworks.io AssemblyAI Talk Python Training
  • Talk Python To Me podcast

    #331: Meet the Python Developer in Residence: Lukasz Langa

    1:06:21

    Python is a technology and community built upon the goodwill and volunteer time of 1,000s of contributors from the core devs inside CPython to the authors of 100,000s of external packages on PyPI. Until recently, the only full time folks have been at the PSF doing very important work but that work has been largely outside of CPython the technology. In July, 2021, the PSF created the Python Developer in Residence position. The first person in that role is Łukasz Langa and he's here to tell us how it's going and how it will benefit Python at large. Links from the show Łukasz Langa on twitter: @llanga Black: github.com/psf/black CPython PRs: github.com Weekly reports: lukasz.langa.pl Visionary Sponsors: python.org/psf/sponsorship/sponsors What do you get when you sponsor the PSF?: www.python.org/sponsors/application Brett Canon's PyCascades talk: youtube.com Django fellowship program: djangoproject.com Lukasz's prior episodes: Gradual Typing of Production Applications: talkpython.fm/151 Dive into CPython 3.8 and beyond: talkpython.fm/214 Python Language Summit 2018: talkpython.fm/179 Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Talk Python Training AssemblyAI
  • Talk Python To Me podcast

    #330: Apache Airflow Open-Source Workflow with Python

    1:07:50

    If you are working with data pipelines, you definitely need to give Apache Airflow a look. This pure-Python workflow framework is one of the most popular and capable out there. You create your workflows by writing Python code using clever language operators and then you can monitor them and even debug them visually once they get started. Stop writing manual code or cron-job based code to create data pipelines check out Airflow. We're joined by three excellent guests from the Airflow community: Jarek Potiuk, Kaxil Naik, and Leah Cole. Links from the show Jarek Potiuk: linkedin.com Kaxil Naik: @kaxil Leah Cole: @leahecole Airflow site: airflow.apache.org Airflow on GitHub: github.com Airflow community: airflow.apache.org UI: github.com Helm Chart for Apache Airflow: airflow.apache.org Airflow Summit: airflowsummit.org Astronomer: astronomer.io Astronomer Registry (Easy to search for official and community Providers): registry.astronomer.io REST API: airflow.apache.org Contributing: github.com Airflow Loves Kubernetes talk: airflowsummit.org Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Talk Python Training AssemblyAI
  • Talk Python To Me podcast

    #329: Geekout: Renewable Energy

    48:18

    We're back with another GeekOut episode. Richard Campbell, a developer and podcaster who also dives deep into science and tech topics, is back for our third GeekOut episode. This time around, we're diving into renewable energy, energy storage, and just what do we do to keep the lights on with our frying our beloved Earth? Links from the show Richard on Twitter: @richcampbell IEA report 2021: iea.org Flywheel storage: blogspot.com Crane storage: eni.com Pumped hydro storage: eurekalert.org Tesla battery utility-scale: tesla.com The US’s largest solar farm is canceled because Nevada locals don’t want to look at it: electrek.co DEVintersection conference (run by Richard): devintersection.com .NET Rocks Podcast (Richard's a cohost, many geekout episodes): dotnetrocks.com Prior Geekouts on Talk Python #276: Geekout: Life in the solar system and beyond: talkpython.fm #253: Moon base geekout: talkpython.fm Watch this episode on YouTube: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Talk Python Training AssemblyAI

Descobre o mundo dos podcasts com a app gratuita GetPodcast.

Subscreve os teus podcasts preferidos, ouve episódios offline e obtém recomendações fantásticas.

iOS buttonAndroid button