Something strange is happening to the web. Not the usual "everything is changing" hand-waving, but something more specific. We aren't just bolting AI things onto the existing internet, we're building a whole new one specifically for machines.
It started small, with file formats that are easier for agents to read, AI-generated summaries replacing search results, new protocols gaining traction. But zoom out and the scope of it becomes clear. There's a whole parallel web taking shape, with its own ways of discovering content, reading it, interacting with it, paying for it, and deciding who to trust. And it's being assembled remarkably fast.
SEO → AEO
For two decades, the game was SEO. You optimized for Google's crawlers, you played the keywords game, you built backlinks. The output was a ranked list of blue links, and humans clicked through to them. You know, you "Google it."
That model is cracking apart. Gartner predicts traditional search volume will drop 25% by 2026 as users shift to AI chatbots and agents. Zero-click Google searches went from 56% in 2024 to 69% in 2025. ChatGPT alone now serves over 800 million users weekly. People aren't scanning results pages anymore, they're getting synthesized answers.
The industry is calling the replacement Answer Engine Optimization, or AEO, and it works completely differently. With SEO, you were trying to show up high on the page. With AEO, you're trying to be the source that an AI cites when it answers someone's question. Your content doesn't need to rank anymore, it needs to be trusted enough that an AI pulls from it, and the way these systems decide what to trust looks nothing like how Google decides what to rank.
This shift matters for anyone who publishes anything online. If an AI assistant answers a question about your product and doesn't pull from your site, you're invisible. Not "page 2 of Google" invisible, but genuinely absent from the conversation entirely.
Which raises an obvious follow-up, what does your site actually look like to one of these agents?
HTML is too noisy
The web was built for browsers, not language models. This sounds obvious once you say it out loud, but it matters more than you'd think. Navigation menus, cookie banners, JavaScript, footers, sidebars, tracking scripts, all the stuff browsers need to render a visual experience, and all of it is noise if you're just trying to get to the content (this is the whole point of reader mode).
Serving a stripped-down, text-only version of a page to AI crawlers instead of the full HTML makes the gap concrete. A typical blog post shrinks by about 80% when you strip away all the visual cruft, and the major infrastructure providers that sit between websites and the internet are already building this conversion into their platforms so site owners can flip it on without rewriting anything.
The reaction from the SEO world was predictably split. Google's John Mueller called the practice of serving these simplified pages to bots "a stupid idea" on Bluesky, arguing that AI systems can already parse regular web pages just fine. Technical SEO consultant Jono Alderson raised a more nuanced concern, arguing that once you create a machine-specific representation of a page, you've created a second version of reality that can drift from the original.
Then there's llms.txt, proposed by Jeremy Howard of Answer.AI in September 2024. It's basically a cheat sheet you put on your website that tells AI systems what your most important content is and where to find it, like a table of contents written specifically for machines. Over 844,000 websites have implemented it, including Anthropic, Stripe, and Spotify. But the data on whether it actually impacts citations is thin. An SE Ranking study of 300,000 domains found no measurable effect on how often AI systems cited those sites. The honest answer is that nobody really knows if it matters yet, but the implementation cost is so low that most companies are treating it as an insurance policy.
The point isn't whether llms.txt specifically wins, it's that a whole layer of the web is forming that exists purely for machine consumption - with new file formats, structured data, and curated content summaries designed for contexts where no human will ever see them.
But making content readable is only half the problem.
Agents need to do things, not just read things
AI agents increasingly need to interact with the web, filling out forms, navigating workflows, taking actions on your behalf. The current web is almost comically bad at supporting this.
Think about it from an agent's perspective. You land on a page and encounter a form. You don't know what the fields are for, what format they expect, or what happens when you submit. A human can look at the labels, infer context from the layout, and figure it out. But an agent is mostly guessing, it's like handing someone a paper form in a language they don't speak and asking them to fill it out correctly.
MCP (Model Context Protocol) is the most interesting answer so far. Anthropic introduced it in November 2024 as a universal standard for connecting AI agents to outside tools and services, sort of a USB-C port for AI, one standard plug that works everywhere. In the year since, adoption has been steep. The official registry lists over 6,400 integrations. OpenAI, Google DeepMind, Microsoft, and Amazon have all adopted it. In December 2025, Anthropic handed governance of MCP over to the Linux Foundation, with OpenAI and Block as co-founders of the new body overseeing it.
The catch is that MCP only works when the service on the other end has been set up to support it, and most websites haven't been. That's where WebMCP gets interesting. The idea is to add a machine-readable layer on top of existing web pages, so agents can interact with forms and workflows.
The other approach is to just have the agent drive a web browser the way a human would, clicking buttons and filling in fields. It works, but it's slow, expensive, and brittle, more of a workaround than a real solution.
All of this raises a question that the tech industry is good at deferring but can't defer forever.
Who pays for this?
The web's content was created by people and organizations who expected to be compensated, whether through ad revenue, subscriptions, or the traffic that sustained both. AI agents that synthesize answers from that content and deliver them directly to users short-circuit that entire model. The value still flows, it just stops flowing to the people who created it.
RSL (Really Simple Licensing) is the most serious attempt to address this head-on. Launched in September 2025 by RSS co-creator Eckart Walther and former Ask.com CEO Doug Leeds, it lets publishers attach licensing and payment terms to their content in a way that AI systems can automatically read and respect. A site can say "this content is free to use," or "you need to pay us every time you crawl this," or "you owe us a fee every time an AI uses this to answer a question." The 1.0 spec shipped in December 2025, and the supporter list is substantial, with Reddit, Yahoo, Medium, Quora, Ziff Davis, O'Reilly Media, and over 1,500 other organizations signed on.
The big infrastructure companies that route internet traffic are getting behind it too, building gatekeeper technology that can admit or block AI bots depending on whether they've agreed to a publisher's terms.
The parallel to the music industry is intentional. The RSL Collective functions like ASCAP for web content, pooling publisher rights and negotiating collectively with AI companies. Whether it actually shifts the economics depends entirely on whether major AI providers play ball, and history isn't encouraging on that front. AI companies have been scraping content first and asking permission later for years, so RSL needs real enforcement behind it, not just a polite request.
Ben Thompson has been working this from another angle. In a recent Stratechery piece, he argued that agents represent the third major LLM inflection point, and that agentic commerce will reshape how products are discovered and purchased. If agents are doing the research and making buying decisions on behalf of users, the advertising model that funds most of the web starts looking very different. Thompson's framing of "perfect competition" through AI, where agents obsessively optimize every purchase decision, is exciting or threatening depending on where you sit in the value chain.
Even if the economics get sorted out, though, there's a deeper problem that nobody has a good answer for yet, which is whether you can actually trust these agents once they're loose on the web.
Agents in hostile territory
Every previous version of the web had a trust problem, and every time we underestimated how bad it would get. Spam, phishing, identity theft, each one arrived faster than the defenses against it. AI agents are about to repeat that cycle, except the stakes are higher because agents don't just read the web, they act on it.
Simon Willison has described what he calls the "lethal trifecta" for AI-enabled browsers, which is an agent that can access your private data, communicate with the outside world, and visit websites it's never seen before. The moment an agent can both read your email and browse the open web, a malicious website can potentially trick it into doing things you never asked it to do. Imagine a phishing attack, except instead of fooling you into clicking a link, it fools your AI assistant into forwarding sensitive documents to the wrong person.
Right now, there's no standardized way for a website to verify that an incoming agent is who it claims to be, that it has permission to act on behalf of a particular user, or that it should be trusted with sensitive information. We're in the pre-HTTPS era of agent security, except this time the agents aren't just reading pages, they're booking flights and sending emails.
What's actually happening here
Step back from any one of these threads and the picture gets clearer. How agents find content is being rebuilt with AEO and llms.txt. How they read it is being reimagined with cleaner formats designed for machines instead of browsers. How they interact with services is being standardized through MCP and WebMCP. How content creators get paid is being formalized with RSL. And how we verify that an agent is who it says it is and can be trusted to act on your behalf, the piece that ties it all together, barely exists yet.
That's an infrastructure stack, and we're building it while the existing web keeps running, which is a bit like renovating a plane mid-flight except nobody agrees on the blueprints and the passengers keep asking for snacks.
Every company on the web faces the same question, which is whether to invest in this now or wait. The companies that build for agents today, even clumsily, will be the ones agents can actually work with tomorrow. The ones that don't will become invisible to an entire class of users who never open a browser.
Jeremy Howard of Answer.AI said it bluntly in March 2025, "99.9% of attention is about to be LLM attention, not human attention." The specific ratio is debatable, but the direction isn't. The web that was built for people with browsers needs a companion built for machines. Ready or not, that companion is already here.