Last updated:
Bots & Humans: maintaining AI agents and automation in an AI-bot web
The web your scrapers and agents depend on is closed by default: CAPTCHAs, AI crawl blockers, TLS fingerprinting. Meanwhile AI agents are becoming the biggest readers of the web. This talk covers why DIY scraping decays into anti-bot infrastructure, and what a self-healing, API-first alternative looks like, with a live demo.
Talk by Anil Kumar Krishnashetty at devWorld Conference, Amsterdam, May 2026.
Key takeaways
- Your docs’ biggest reader is no longer a developer, it is the agent the developer sends. Mintlify measured nearly 199 million documentation requests from Claude Code in one month (March 2026), more than Chrome on Windows.
- The web is closed by default. Cloudflare’s one-click AI crawl blocker, enabled by millions of sites, blocked over 400 billion bot requests in its first five months.
- “Ask the agent to browse for it” does not scale: one task at a time, per-task cost multiplied by millions of pages, and the agent hits the same CAPTCHAs and blocks your scraper does.
- DIY scraping decays into anti-bot infrastructure: Playwright, then rotating proxies, then CAPTCHA solvers, then TLS fingerprint evasion. You wanted data, you’re now maintaining an evasion stack.
- The architectural fix is treating web data like a database or CDN: give a URL, get an API back, with self-healing scrapers that detect DOM changes and patch their own code.
Transcript
Transcript lightly edited for clarity.
Your biggest reader is now an AI agent
Hi, welcome everyone. I want to start this talk with a quick number. How many of you here are using Claude? That’s great. Here is a fresh insight from Mintlify, a documentation platform. In March they published one month of data: Claude Code alone made nearly 199 million requests to documentation, way more than Chrome on Windows. So what I want to share is this: if you’re building dev tools, your biggest reader is not a developer anymore. It’s the agent the developer is sending to your documentation.
I’m Anil Kumar, a Technical Product Marketing Manager at Bright Data. Before that I spent a decade building large scale B2B applications at enterprise companies. If you don’t know Bright Data: it is the world’s number one web data infrastructure platform, and more than 70% of the world’s AI labs are customers, just to give you a sense of scale.
In this talk I’ll set out the landscape and the challenges, and in the second part I’ll do a live demo: pick one website and build an API for it instantly.
The web we have today is a closed web
I want to start with a statement: the web we have today is a closed web. Bots are defended against everywhere. Crawlers are blocked by default. There are two contrasting things here: Claude Code can get data from documentation, yet the web is closed. For a glimpse of the closed web, think of CAPTCHAs, checking whether you are a human. Wherever you try to access content, there is a block for bots.
A quick run through history. The original web was open: you could fetch HTML with a simple curl. Around 2010 you could build apps on free social media APIs, and billions of apps were built that way. But around 2018 the platforms realized data itself is a product, started closing it off and charging for it. Maps got more expensive, Twitter likewise. Now the web is closed. Cloudflare introduced one-click AI crawl blockers last year; millions of sites enabled them, and in five months they blocked more than 400 billion bot requests. Billions.
So it’s getting tighter and tighter. On the other side, AI agent traffic keeps growing, getting nearly equal to humans. How are we going to manage this?
We built the web for human consumption for a long time. Now bots and humans share the space: machines acting like humans, clicking buttons, scrolling PDFs, doing OCR on the fly. That is what it will look like for years, until the transition is complete.
Why “just ask an LLM” doesn’t scale
Say I want data on millions of products from Amazon: pricing, ratings, descriptions. I’m building a competitive intelligence application, or I’m an e-commerce company that needs the data. For one page or a few tens of pages, fine: ask an LLM, or use solutions like Apify or Firecrawl. But what if I need millions or billions of pages? If you’re building a RAG or AI application, you want to train on fresh, real-time data, not something six months old, and not from ten websites.
One approach: ask your AI agent, “go get it for me.” It feels like a good idea, but there are limits. The agent works like a human: it opens a browser, clicks pages, navigates, one task at a time. For 10,000 pages that takes days. Then cost: per-task cost multiplied by millions of pages. And most important, working through a browser, the more requests it makes, the more blocking and CAPTCHAs it hits. You’d have to solve all of that too.
DIY scraping becomes anti-bot infrastructure
So what do we do as builders? We build our own. First an automation framework like Selenium or Playwright. It works up to maybe 50 pages, then my IP gets noticed and blocked. So I add a proxy solution to rotate IPs. Still blocked by CAPTCHAs, so I add CAPTCHA solvers. Then TLS fingerprinting recognizes my handshake before the request even lands. Then I’m mimicking mouse movements, simulating human behavior. My goal was to get simple data, and now I’m building anti-bot infrastructure. Is that what we want to build? No.
Think about a database: you just query it and get data. Or a CDN: configure it, request, get data. I want the same for web data: give a URL, get website data at scale.
And there’s another problem: even with the infrastructure built, the website changes. The DOM structure changes and you fix it manually. It can get worse: with a library like Beautiful Soup you don’t even get an error, you get a null value that flows into your AI models and data pipeline. Your downstream data quietly goes bad.
Fundamentally, this is not a bug. It’s the architecture. DIY scraping is brittle because selectors and DOM structures change constantly. Adding a new source takes time. And because the code keeps breaking, nobody wants to touch it.
Demo: from URL to API in minutes
The idea is simple: tell me what data you want, give me a URL, and I give you an API instantly that you can put in your data pipeline. Not one website: thousands, millions of pages.
For the demo I picked coolblue.nl, the most common e-commerce site in the Netherlands according to an attendee. I open a smartphone product page. My goal, as an e-commerce competitor: extract pricing, name, description. I give the prompt and the URL. The tool analyzes the website and proposes a schema I can approve or edit: product title, price, rating, review count, availability, description, brand. I trim it to what I need and approve. It builds a collector, effectively preparing an API I can call for any page of this site. I can add URLs one by one or upload thousands of product URLs. I run it, download the JSON, and there it is: the pricing, title, and ratings I asked for.
The more interesting part: I can drive all of this by API. That was the goal: prompt plus URL, get an API. Why an API? Full automation. From Postman I pass the URL, authorize with my API key, get the collection, request the data. What I passed was one URL, but the same call scales to thousands or millions of pages.
Self-healing scrapers
Now the hardest part: maintenance. What if the DOM changes or you want different data? Initially I skipped ratings, now I want them, or I want them removed to save tokens in my AI application. I go to the agent and say “remove ratings.” I’m not coding. It inspects the schema, makes the change, and asks me to confirm. If the scraper breaks, I ask it to fix the scraper and it patches itself. And it’s not a black box: it shows the generated code and the exact change. I accept, save to production, run again, and the ratings field is gone from the output.
You get everything you need. You own the code. If you prefer to code, edit it yourself. It’s up to you.
Where this goes
What I showed is one approach: AI writing your data pipelines for you. Sites change, broken scrapers are detected instantly and patched, that is self-healing capability. Want ten more sources? Add them and scale in days. And developers keep control: what’s collected, versioning, the code itself.
I believe this is one of the ways we get web data at scale. If it’s interesting, give it a try. Thank you very much.
FAQ
Why can’t I just use Playwright or Selenium for large-scale scraping?
They work up to roughly tens of pages. Beyond that your IP gets flagged, then you need rotating proxies, CAPTCHA solving, and TLS fingerprint evasion. Each layer is more anti-bot infrastructure to maintain, and a single DOM change still silently breaks your selectors.
Why not let an AI agent browse and collect the data?
An agent browses like a human: one page at a time, through a real browser. At 10,000 pages that means days of runtime, per-task cost multiplied by page count, and the same blocking and CAPTCHAs any scraper faces.
What is a self-healing scraper?
A scraper that detects when a site’s DOM structure changed, regenerates its own extraction code, and shows you the diff before you save it to production. You own the code and can edit it, but you are not the one watching for breakage.
Where can I watch the full talk?
The full recording is on YouTube, from devWorld Conference Amsterdam, May 2026.