Picture this:
You land on a travel site, type “best family-friendly spots in Barcelona,” and instead of clicking through menus, the site just… talks back with a personalised, conversational answer.
Not from Google. Not from ChatGPT, scraping it.
Directly from the site itself.
That’s the Agentic Web, and it’s already rolling out.
What’s Actually Happening?
Two big players are behind this shift: Microsoft and Cloudflare.
NLWeb (Microsoft): Think of it as a new framework that makes your site conversational. It sets up two magic endpoints:
/ask→ where humans can query your site in natural language./mcp→ where AI agents (using the Model Context Protocol) can pull structured answers directly.
AutoRAG (Cloudflare): Handles all the grunt work: crawling, embedding, indexing, and serving those answers in real time. Basically, you flip a switch and your site starts acting like a mini-AI app.
Together, these tools make websites queryable by humans and AI agents, without the guesswork of scraping.
Why This Matters
Let’s be honest: scraping has always been messy. Models guess, they misinterpret, they regurgitate outdated stuff.
Agentic Web changes that:
No more blind scraping: Agents can just ask your site.
You’re in control: Site owners decide what’s exposed and how.
Better UX: Users ask questions, get direct answers—no more link-hopping.
Freshness built-in: AutoRAG reindexes your site continuously, so the answers stay current.
This shift? It’s as big as the move to mobile-first design a decade ago.
How Do You Get Started?
If you’re on Cloudflare, it’s surprisingly plug-and-play:
Log into your dashboard.
Go to Compute & AI > AutoRAG.
Pick the NLWeb Website option.
AutoRAG crawls and embeds your content.
/askand/mcpendpoints go live.Test it in Cloudflare’s preview chat UI.
That’s it. Your site is now conversational, for both people and AI.
What It Means for Us
Designers: Time to start thinking about conversational entry points in your layouts.
Marketers: Stop chasing clicks, start chasing queries. What’s your audience asking?
Writers: Structure content in “surface-ready” formats: FAQs, lists, bullets, stats.
SEOs/GEO/LLMO pros: Visibility is no longer just about SERPs, it’s about being cited by AI agents.
Quick Note on Security
Of course, no shiny tech is without its bumps. Early NLWeb builds had a path traversal flaw (yikes!) that let attackers peek at sensitive files.
Microsoft patched it in July 2025, but only if you update.
The Verge has the full story
So yes, this is exciting. But if you’re going agentic, stay patched.
The Big Picture
This isn’t a “someday” thing. It’s live now. Early adopters already include Shopify, TripAdvisor, and Hearst.
For businesses, it’s both:
A challenge: rethinking content, design, and SEO fundamentals.
An opportunity: becoming the default answer AI assistants serve up.
Final Word
The Agentic Web is here, and it’s not waiting for anyone.
If your brand lives on the web (spoiler: it does), the question is simple:
Do you want to be scraped randomly or asked directly?