ACIS
    Back to Insights

    Field notes

    How GEO, WebMCP and Bot Paywalls are Building the AI-Native Internet

    The internet of the future will look much different than the one today. Businesses are already transitioning.

    By Rodrigo Almeida, GEO Specialist at ACIS · November 2026

    Person in a mustard sweater typing on a laptop at a wooden desk by a window
    AI agents now browse the web alongside people, and websites are adapting to serve both. Photo by Christin Hume via Unsplash

    With the rise of AI agents, the internet as we know it is changing in fundamental ways. Since its inception, the internet was built for humans. Our path was always the same. We searched on Google, went directly to websites, browsed, and found what we wanted, whether that was information or a product to purchase.

    When AI agents came along, those principles stayed in place, and the agents had to work around them. They crawl as bots, browse dozens of websites in seconds, collect the information they need, and bring it back to the user. The process works, but in many ways it is inefficient. In 2026, Cloudflare Radar data showed that bots now account for over 57.5% of web requests, surpassing human traffic. AI makes up a rapidly growing part of that share. Across TollBit's publisher network, AI bot visits grew from 1 for every 200 human visits in early 2025 to 1 for every 31 by the end of the year. Most internet traffic now runs on principles that were never designed for it, and the internet as we know it can no longer effectively support the volume of AI traffic.

    AI gathers information from the internet in two primary ways, model training and retrieval. Although the purposes differ, both involve collecting content from the internet. During model training, the information is collected ahead of time, while during retrieval the AI agent crawls for information after the user asks a question. Retrieval is now the larger of the two. In late 2025, retrieval bots made roughly ten page requests for every one made by a training bot.

    Over the last year especially, there has been a lot of work to better support how AI collects information, and I want to outline the main developments here.

    Generative Engine Optimization (GEO)

    Also referred to as Answer Engine Optimization (AEO), GEO is the practice of optimizing a website so AI can easily retrieve its information and is more likely to recommend it. GEO is particularly valuable for e-commerce stores, where the shopping experience is changing drastically.

    More and more, people are turning to AI for product recommendations. AI can sift through several brands and make real-time comparisons. According to Adobe, AI-referred traffic converted at a 60% higher rate than non-AI traffic in July 2026, since these shoppers arrive already referred to the brand. All this depends on how easily AI can find the information it needs, whether it trusts that information, and whether that information leads it to recommend a store. Every GEO project has three parts, the technical structure, the website content, and third-party citations.

    Technical structure

    The technical structure determines how AI navigates the website. It works like a map, giving AI an efficient path to the information it needs. The technical structure steps include:

    • Robots.txt. Confirm the file is not blocking the AI agents you want visiting your store.
    • Sitemap. Keep it accurate so agents can find every page that matters.
    • llms.txt and agents.md. These files give AI agents a guide to the website. Shopify now generates both automatically and treats agents.md as its primary discovery file for AI. The default versions are generic and worth customizing.
    • Product schema. Structured product data keeps prices, availability, and specifications accurate and easy for AI to read.
    • Rendering. Content that only loads through JavaScript may be invisible to many AI crawlers.

    It is a common misconception that the technical structure steps alone are enough for GEO. They guide AI to the right content, but they won't produce results if the content itself is not optimized. Which brings me to the second part of a successful GEO project.

    Website content

    Everything begins with knowing exactly which questions AI is trying to answer. The best sources for those questions are customer reviews, forum conversations, and keyword search data. From there, the work involves rewriting product copy to answer those questions directly, building a comprehensive FAQ, keeping product information consistent across every sales channel, and publishing informational content such as comparisons and guides.

    Third-party citations

    Technical structure and website content are both in the store owner's control. Third-party citations are not, and they matter most during product discovery, when a shopper has not yet decided on a brand. At that stage, AI leans on what others say about a brand in reviews, Reddit threads, YouTube videos, and industry publications. Building that presence takes longer than the on-site work, and it allows smaller brands to compete with industry leaders in AI recommendations.

    MCP and WebMCP

    When it comes to exchanging information and actionable tools between AI agents and websites, MCP is the established standard today.

    Model Context Protocol (MCP)

    Model Context Protocol (MCP) is an open standard that defines how AI systems connect to external tools and data. Anthropic introduced it in November 2024, OpenAI adopted it in March 2025, and most major AI platforms now support it. An MCP server exposes tools (actions an agent can take), resources (data it can read), and prompts (templates for common tasks). Instead of every AI company building a custom integration for every service, one MCP server works with any compatible agent.

    WebMCP

    In February 2026, Google's Chrome team announced an early preview of WebMCP, a proposed web standard developed in partnership with Microsoft. WebMCP brings the MCP concept to the browser. A website can publish structured actions, such as searching products, adding an item to the cart, or booking an appointment, and AI agents call them directly instead of guessing their way through screenshots and page code.

    The process outlined in the GEO section involves people using AI to browse and compare products, but it is still the user's job to go to the online store and make the purchase. This customer journey is slowly changing.

    AI agents are increasingly gaining wallet access, so they can now buy products directly after comparing them. WebMCP gives agents a reliable way to use the store, and a separate set of protocols handles the sale itself. OpenAI and Stripe launched the Agentic Commerce Protocol (ACP) in September 2025 to power checkout inside ChatGPT. Google followed with the Universal Commerce Protocol (UCP), built with Shopify, Walmart, Target, and others, to power checkout in Gemini and Google's AI Mode. Underneath both, Google's Agent Payments Protocol (AP2) and agent-specific card tokens from Visa and Mastercard confirm the shopper authorized the purchase without exposing card details to the agent.

    WebMCP creates a second version of your website built for AI agents, and I like to think of it as a house.

    Imagine your website is a house with doors, windows, and a fence around it. People get through the fence gate and the front door easily, but AI agents can't. Without GEO or WebMCP, agents have to jump the fence and crawl through a window to get in. GEO lowers the fence, which makes the trip easier but still far from smooth. WebMCP gives AI its own gate and its own door, so agents can walk in as easily as people do, through a path built for them.

    According to a 2026 NMI survey, while only 11% of U.S. adults have used AI to complete a transaction, 33% used AI for shopping research in the past month.

    Bot paywalls for content monetization

    The last shift I want to cover is one I find especially exciting. In the previous two sections, I assumed the website owner WANTS AI to visit the site. For e-commerce stores, AI is simply a new pathway to sales and revenue. Other online businesses see it differently, and many have been severely harmed by the rise in AI traffic.

    Content websites (news outlets, research publishers, recipe sites, reference sites, and review blogs) don't sell a product on their website. Their revenue model relies on advertising, so they make money from human attention. As people use AI to sift through and summarize information, that attention disappears, and revenue drops significantly.

    When Google shows an AI summary, users click a traditional search result 8% of the time, compared to 15% when no summary appears (Pew Research, 2025). The average click-through rate from AI tools to publishers fell by roughly two-thirds over the course of 2025 (TollBit).

    This entire time, AI has been using internet data for free. Businesses like TollBit, Cloudflare, and AWS are building bot paywall systems to change that.

    Bot paywall systems work by identifying each incoming bot and deciding whether to allow, block, or charge it. A bot that has not paid receives a price, often through an HTTP 402 "Payment Required" response, and gets the content once it pays.

    • TollBit routes AI bot traffic to a TollBit subdomain that serves an agent-optimized version of the site. Publishers set rates by license type (summarization or full display) and can add rules by bot, page, keyword, time window, or directory. The AI company pays a small transaction fee, and the publisher keeps the full rate.
    • Cloudflare launched Pay Per Crawl on July 1, 2025, the same day it began blocking AI crawlers by default. Publishers set a flat per-request price, and Cloudflare handles payment as the merchant of record.
    • AWS added AI traffic monetization to AWS WAF in June 2026 for sites on CloudFront. Publishers set a price per page, and agents pay in stablecoin through the x402 protocol. AWS has announced Stripe support as coming soon.

    You may be thinking, "AI companies already get this information for free, so why would they pay for it?" It is one of the main questions on the topic, and the answer has two sides. In the ideal scenario, the system works as an information marketplace where both sides benefit.

    What is in it for publishers

    On the publisher's side, the initial appeal is intuitive. Bot paywalls turn traffic that returns almost nothing today into revenue. In mid-2025, Cloudflare measured Anthropic crawling roughly 38,000 pages for every visitor it referred back to a site, and OpenAI roughly 1,100. Publishers also gain control over which bots access which content and for what purpose, along with an audit trail of every request.

    What is in it for AI companies

    It may be hard to believe that AI companies will pay for information they get for free today, but that information isn't truly free. Going back to the house analogy, AI agents today have to jump fences and climb through windows to get inside. An information marketplace comes with MCPs built for publishers' content, which gives the bot its own door. The inefficiency in how AI finds information already shows up on AI companies' balance sheets as computing costs.

    Second, AI companies face no shortage of lawsuits over their crawling practices. Perplexity alone faces around ten lawsuits, including cases from Dow Jones and the New York Post (2024), Encyclopedia Britannica, Reddit, and Amazon (2025), and The New York Times and CNN (2025 and 2026). In 2025, Cloudflare also publicly accused Perplexity of using undeclared crawlers to get around websites' blocking rules. Anthropic agreed to a $1.5 billion settlement with authors in 2025 over pirated books used for training, and Reddit sued Anthropic that same year over scraping. OpenAI is still fighting The New York Times in a case filed in 2023. Legal fees and settlements across the industry now reach into the billions of dollars.

    Third, more and more publishers are actively using security measures to block AI bots. Since July 2025, Cloudflare, which serves roughly one in five websites, blocks AI crawlers by default. With a bot paywall, that high-quality data becomes available again for model training and retrieval.

    Each transaction would cost a fraction of a dollar, and in the big picture, it will be worth it for AI companies to take the clean road and pay the toll. A toll highway is almost always better than the alternative.

    The last thing to consider is pricing, and it remains an open question across the industry. It is hard to know how much AI companies will be willing to pay for content, so dynamic pricing strategies will be crucial. Today's systems offer rule-based pricing. TollBit lets publishers price by bot, page, and time window, and AWS lets them price based on how well a bot's identity is verified. The market is still early, and paid volume remains small. Over time, prices will need to reflect how valuable a piece of content is, how fresh it is, and how much AI demand it attracts.

    What's next?

    The internet is going through its largest structural change since search engines arrived. GEO helps AI find and trust a website, WebMCP and the new commerce protocols let agents act and buy on a shopper's behalf, and bot paywalls give publishers a way to get paid for the content AI depends on. Each of these is still early, and most businesses have yet to adopt any of them. The businesses that move now will have their infrastructure in place by the time AI traffic becomes the norm, and they will have shaped how AI represents them instead of reacting to it.

    Want to know how your business is currently perceived by AI?

    Works cited