Search, Agent, and Training Traffic Need Different Rules
Cloudflare's new AI traffic controls expose a bigger shift for marketers: search, agent, and training traffic need separate rules, access, and KPIs.
Most marketers are still treating AI traffic like one bucket. That was barely workable six months ago. It is wrong now.
On July 1, Cloudflare rolled out new controls that split AI traffic into three categories: Search, Agent, and Training. That change matters because it matches what is happening in the market. Search systems are indexing pages to answer questions later. Agents are visiting sites in real time to do work for a user. Training crawlers are collecting content to improve models. Those are not the same behavior, so they should not get the same policy.
That is the myth worth killing this week: “AI bots” is no longer a useful operating category for marketers.
The Old AI Bot Setting Is Too Blunt
For a long time, the practical choice was simple: allow crawlers or block them.
Cloudflare is now arguing for a more useful taxonomy. In its July 1 update, the company defines Search as traffic that indexes your content to answer future queries, Agent as traffic acting on a person’s behalf in real time, and Training as traffic that absorbs your content into model improvement. It also says site owners should expect referral traffic or equitable compensation from search behavior, while agent visits are job-oriented and training visits are extractive by nature.
That is a meaningful strategic split for marketing teams.
If a search system is likely to cite your pages, link back, or help buyers discover you, the business case for allowing that traffic is different from a browser-use agent trying to interact with forms or a training crawler copying content into a model pipeline. Treating all three as one decision makes your website policy sloppy from the start.
What Changed This Week
The practical trigger is not just Cloudflare’s new dashboard setting. It is the broader direction of the market.
Cloudflare said the new controls are live now and that, starting on September 15, 2026, new domains on Cloudflare will default to blocking Training and Agent traffic on ad-monetized pages while leaving Search allowed by default. That is a strong signal that website infrastructure is moving beyond the one-size-fits-all “AI bot” label.
OpenAI is pushing from the product side. In its launch of ChatGPT agent, OpenAI says the agent can use a visual browser, text browser, terminal, and connectors to complete real tasks on the web. That is not passive crawling. That is a system visiting pages to get work done.
Google is pushing from the discovery side. In its I/O 2026 Search update, Google said Search is entering a new phase with AI features that let users “use agents just by asking a question.” That means search behavior and agent behavior are starting to overlap in the user’s experience even when they remain different from the website owner’s point of view.
The result is simple: marketers now have to separate discovery traffic from action traffic from model-training traffic.
Why Marketers Should Care
This is not a technical nuance for your IT team to handle quietly.
It changes how you think about discoverability, conversion, and content protection.
Search traffic is closest to classic AEO. You want the right pages to be crawlable, citable, and easy to interpret. If a buyer asks an AI system for options in your category, this is the lane most likely to influence whether your brand even enters the conversation.
Agent traffic is a workflow problem. If an agent is checking pricing, reading a product page, or submitting a request on behalf of a user, your page structure, form logic, and access rules matter more than ever. This is where our earlier post on AI agents needing website rules still applies, but Cloudflare’s new taxonomy gives teams a more practical way to manage it.
Training traffic is a content rights problem. The value exchange is different. If a crawler is not there to send you traffic or complete a user task, then you need a more deliberate position on whether it should have access at all.
Once those use cases get separated, your reporting should separate too. “AI traffic” is becoming too vague to guide smart decisions.
The Wrong Reaction
A lot of teams will respond in one of two bad ways.
The first bad response is to block everything and call it protection. The second is to allow everything and call it innovation.
Both shortcuts miss the point. Cloudflare’s taxonomy exists because the behavior is now different enough to deserve different treatment. The company even introduced a more granular content-use signal, ranging from immediate interaction to reference-level indexing to fuller reuse. That tells you where this is headed: not toward one universal AI policy, but toward finer rules about who can access what and why.
This matters commercially. If your site depends on discovery, you may want broad search access but tighter rules around agent actions on sensitive flows. If your business runs ads or monetized content, you may want to keep training and some agent traffic away from those pages while preserving search visibility. If you sell high-consideration services, you may want agents to interpret FAQ and pricing pages cleanly without giving them unrestricted access across the full site.
There is no serious strategy left in “allow AI” or “block AI.”
What To Change On Monday Morning
Start with a short audit, not a giant rebuild.
First, list the pages where AI discovery actually matters: service pages, comparison pages, FAQ pages, proof pages, and any page that should show up when buyers ask AI systems for recommendations.
Second, separate the pages where agent interaction could create value from the pages where it could create risk. Quote forms, appointment requests, login states, checkout flows, and gated resources should not all be handled the same way.
Third, review your infrastructure settings. If your team is already using Cloudflare or similar controls, this becomes a website development and governance conversation, not just a content conversation. The policy decisions need to match commercial intent.
Fourth, stop reporting AI influence as a single blob. Discovery, action, and content-use risk are different questions with different owners.
The bigger takeaway is that AI visibility is getting more operational. Search, agent, and training traffic are three different jobs touching the same website. The marketers who separate those jobs early will make better decisions about access, measurement, and content value. The ones who keep lumping them together will keep shipping policies that are too loose where they should be strict and too strict where they should be visible.