S01E05 - Web scraping deployments, proxy infrastructure and sourdough

AI can build a web scraper, but can it keep it running when the website changes? In episode 14, John Rooney and Ayan Pahwa discuss what self-healing scrapers can realistically do, why reducing maintenance matters more than chasing full autonomy, and how to keep agent costs under control. We also recap Extract Summit Austin, explore Zyte MCP and better web access for coding agents, discuss Markdown and shared agent context, and look at John’s self-hosted browser service: CDP connections, queues, browser contexts and the limits of a mini PC.

In Ep 13, we discuss where probabilistic AI tools like JEV fit alongside large language models, how they might help make faster and cheaper decisions in web scraping pipelines, and what we learned from experimenting with JEV. We also talk about building scraping workflows with Scrapy and notebooks, and the practical trade-offs of scraping product variants at scale. Topics include: • JEV and probabilistic classification • Using AI to decide when a website needs browser rendering • Quality of input signals, cost, speed, and debugging • Scrapy workflows, notebooks, and data review • Product variants, extraction, and the cost of extra requests Subscribe for more conversations about web scraping, data extraction, and the tools and techniques behind them. #WebScraping #AI #DataExtraction

When should you use a browser for web scraping? John and Ayan join Zyte’s Paweł to explore CDP, Playwright, and the trade-offs between running browsers locally and using managed infrastructure. From fingerprinting and session management to pricing and AI agents, discover what it takes to run browser-based scraping at scale.

In this episode, Ayan and John explore the rapidly evolving world of AI agents, coding harnesses and custom developer tools. They discuss emerging AI models, modular agent ecosystems, web scraping, prompt-to-data workflows, AI evaluation and the security risks of putting agents into production. They also share new Scrapy projects, reflect on the power of open-source software and Linux, and look at the rise of proactive desktop agents. Plus, there’s time for model pricing, developer workflows and a well-earned appreciation of croissants.

Episode 10 of the Zyte podcast is out, and it is one of the widest-ranging conversations we have recorded this year. John Rooney sits down with Neha Setia Nagpal and Ayan Pahwa for an unfiltered, all-in-style chat covering everything the team has been building, reading, and arguing about lately.

This episode Ayan and John talk to Ian Johnson, staff engineering and author of the Harness Engineering book. We talk through the subject, what it is and what it means to us as builders. Ian Johnson: https://tacoda.dev Harness Engineering book: https://leanpub.com/harness-engineering Upcoming Webinar with Ian Johnson: https://www.eventbrite.com/e/how-to-grow-from-junior-to-senior-in-the-age-of-ai-tickets-1994040543945 Access the web's data. Clean, ready, and at scale | https://zyte.com

In this episode, John, Neha, and Ayan trade origin stories (requests and Beautiful Soup, a runaway crawl at IBM back in 2016, a wall of JavaScript that broke everything) and get into why a lean, extensible core is exactly what modern data extraction needs. They cover the anti-bot arms race, reverse-engineering back-end APIs, the recent httpx download handler and the move toward an async reactor, and the four plugins John reaches for on every production build.

Ayan teaches us about AI harnesses, what the system prompt is and what it does, how it affects what you actually get from your model back, and we talk about what models we actually use for our day-to-day work. 

The team meet up to chat deploying spiders, both managed and self hosted, proxies infrastructure and monitoring, and how sourdough cultures can tell us a lot about code.