Extract Pod by Zyte

In this episode, Ayan and John explore the rapidly evolving world of AI agents, coding harnesses and custom developer tools. They discuss emerging AI models, modular agent ecosystems, web scraping, prompt-to-data workflows, AI evaluation and the security risks of putting agents into production.
They also share new Scrapy projects, reflect on the power of open-source software and Linux, and look at the rise of proactive desktop agents. Plus, there’s time for model pricing, developer workflows and a well-earned appreciation of croissants.

Episode 10 of the Zyte podcast is out, and it is one of the widest-ranging conversations we have recorded this year. John Rooney sits down with Neha Setia Nagpal and Ayan Pahwa for an unfiltered, all-in-style chat covering everything the team has been building, reading, and arguing about lately.

This episode Ayan and John talk to Ian Johnson, staff engineering and author of the Harness Engineering book. We talk through the subject, what it is and what it means to us as builders.Ian Johnson: https://tacoda.devHarness Engineering book: https://leanpub.com/harness-engineering Upcoming Webinar with Ian Johnson: https://www.eventbrite.com/e/how-to-grow-from-junior-to-senior-in-the-age-of-ai-tickets-1994040543945 Access the web's data. Clean, ready, and at scale | https://zyte.com

In this episode, John, Neha, and Ayan trade origin stories (requests and Beautiful Soup, a runaway crawl at IBM back in 2016, a wall of JavaScript that broke everything) and get into why a lean, extensible core is exactly what modern data extraction needs. They cover the anti-bot arms race, reverse-engineering back-end APIs, the recent httpx download handler and the move toward an async reactor, and the four plugins John reaches for on every production build.

Ayan teaches us about AI harnesses, what the system prompt is and what it does, how it affects what you actually get from your model back, and we talk about what models we actually use for our day-to-day work. 

The team meet up to chat deploying spiders, both managed and self hosted, proxies infrastructure and monitoring, and how sourdough cultures can tell us a lot about code.

In this episode we talk about the projects we have been working on, from learning hands on javascript challenges, to automating scraping with N8n and MCP servers we find interesting.