Verified bots face a behaviour check. One study found a 24% seller-identity gap.

18 September to 18 October 2026. The first edition of our monthly review of the files, rules and systems that decide how a website is found and used.

A website can publish a careful crawler policy and still be unable to answer three ordinary questions about the traffic it receives: who got through, whether they kept to the rules they were given, and whether the visit did anything for the business. Most of this month's developments sit somewhere along that chain. Cloudflare has begun assessing whether the bots it has verified keep respecting the preferences sites publish, a discovery pilot it runs with OpenAI has reported its first results, a gateway for charging software per request has opened as a closed beta, and a preprint on advertising transparency has measured how often the account an ads.txt file names can be found in the registry that is meant to confirm it. What connects them is follow-through, because a published rule only opens the relationship with a visitor, and it is the response the visitor received, what it did next and the evidence left behind that show whether the rule is working.

A verified bot still has to keep its word

On 16 October Cloudflare described a system that assesses its verified bots continuously, observing over time whether their traffic is consistent with the robots.txt preferences of the sites they visit and whether they slow down or pause after an HTTP 429 response, the status that tells a client it is sending too many requests. Isolated lapses are not what it is looking for; repeated patterns feed what Cloudflare calls a trust band, a range intended to show how consistently a bot behaves and which observations informed that judgement, rather than a permanent verdict on the bot or its operator.

The system is in internal use today. The two things that would put it in a site owner's hands, public scores on Cloudflare Radar and a customer setting for the trust threshold applied to verified bot traffic, are described as plans in the announcement. For now the useful change is in the question being asked: knowing who operates a crawler helps you decide whether to admit it, but only continued observation shows whether that decision still serves the site, since an identified visitor can stay within its permitted paths and still put more load on a server than its owner intended.

An illustrative reading guide rather than a test of any particular crawler; the four questions need different evidence, even when a dashboard places the answers side by side.

AI discovery is getting help beyond the link graph

On 15 October Cloudflare and OpenAI published early results from a research pilot begun in July, in which signals from Cloudflare's network help OpenAI's search crawler find pages and notice when they change, while OpenAI evaluates and ranks what it finds independently. The participating cohort covered about 1% of Cloudflare's network, and the companies report that the pilot brought hundreds of millions of previously unknown pages into OpenAI's search index. That is a discovery result, and the post treats it as one: measuring what happens after content is found, and whether better discovery turns into business outcomes, is described as the next phase, so more indexed pages do not yet establish more enquiries, sales or referrals. Cloudflare also states that the pilot uses metadata about the web for search and discovery rather than sharing page content for model training, and that a signal about a page does not grant permission to fetch it.

The practical lesson reaches beyond the pilot's cohort, because the waste it describes is the kind any site can produce: automated requests for files that do not exist, sites that redirect almost any address to their homepage, and different addresses that lead to copies of the same content. It is the same fault we recorded in our study of llms.txt on 100 brand websites, where several domains answered a request for a missing file with a homepage, a country picker or a sign-in page. A missing page that answers as missing, and a page that lives at one address rather than several, cost every crawler less and leave the useful pages easier to find; discovery now has more routes than links alone, and the destination still has to be worth reaching.

Paying per request moves into closed beta

On 30 September Cloudflare opened its Monetization Gateway as a closed beta for eligible US-based sellers and buyers, letting a domain owner charge agents per request for access to a website, an API, an MCP tool or a dataset. The exchange runs on HTTP 402, Payment Required, inside the request itself, with no redirect to a checkout page: the seller defines which requests need payment and at what price, the buyer signs an authorisation, and the resource is returned once payment settles, which in the beta happens in USDC on the Base blockchain through Coinbase's x402 facilitator. The four launch customers Cloudflare names, an inference service, a web search API, a market-data provider and a PDF-generation API, each sell something that is consumed whole in a single request.

Cloudflare draws the limit itself. It describes the gateway as built for resources where each request is the use, notes that a page of high-value content may be crawled once and used a thousand times, and points that kind of content to a separate product, Pay Per Use, in which verified buyers report and pay for each use. For a publisher weighing paid machine access, the first decision is therefore the unit being sold and the readers who still need to reach the work freely, and a clear boundary between public pages and paid resources will do more than a payment rule applied across a whole site. The beta is a development to evaluate, not evidence that a revenue stream will follow.

ads.txt names an account. Can the registry confirm it?

A preprint by Tommaso Grotto, Alessandro Cantelli-Forti and Alessio Merlo, first posted on 10 October and revised on 14 October, measures what the authors call the seller-identity gap: a seller ID that a publisher authorises in ads.txt or app-ads.txt but that the applicable sellers.json registry, when it can be read, does not list. The rule is deliberately conservative, counting a row only when the registry was available and omitted the ID, and excluding unavailable, malformed and other non-evaluable registries from the absent-only numerator. Confidential records count as account-present. Each percentage is therefore a lower-bound measure. In the reference population, 40,384,095 declaration rows from 78,686 hosts in the Tranco top million, excluding hosts with 100 or more Google copyright-removal requests, 24.25% named a seller ID missing from an available registry; among publishers with at least 100 removal requests the floor rises to 30.71%, and to 33.08% among those with 10,000 or more.

Sample sizes and uncertainty

Chart and table: Share of declaration rows whose seller ID is absent from an available sellers.json. Each figure is a floor, since rows whose registry could not be evaluated are left out of the numerator rather than counted as present. The figures describe rows in the authors' sample, not the share of websites affected, and a single cross-section, not a change during the month. Source: Table 3 of the preprint. The grey remainder includes all other evidence states, including non-evaluable records; it is not a verified share. The removal-request groups are cumulative and overlap. Their labels indicate enforcement attention, not culpability.

For a publisher the value is a sharper maintenance question, because a file can parse cleanly while an account it authorises has no public corroboration. Before changing a declaration, ask the advertising partner to confirm both the account and its sellers.json entry; an account that is missing and a registry that could not be read call for different explanations, and neither observation on its own establishes fraud.

A smaller change for the incident runbook

On 6 October Google restructured the emergency section of its guidance on reducing crawl rate and added examples of the Retry-After header, support its changelog describes as long-standing but previously documented only on its page about pausing a website. The guidance allows a 500, 503 or 429 response to stand in for a 200 for a short period, from a few hours to a day or two, notes that enough of these responses slow crawling across the whole hostname, including URLs that still return content, and lets a 503 or 429 carry a Retry-After value in seconds or as a UTC date. It also warns against keeping such responses in place for longer than one or two days, because a URL that returns them for several days can be dropped from Google's index. If excessive crawling becomes an incident, the person responding needs a temporary measure, a check that confirms recovery and an owner whose job is to remove it.

What good looks like

Keep the policy, the response and the behaviour together. A dated copy of the robots.txt you served, the response each visitor received and what it did afterwards are what separate a deliberate restriction from an access fault, and a reassuring label from behaviour you can observe.

Let missing pages answer as missing. A not-found response for an address that does not exist, and one address for each page, reduce wasted crawling for every visitor, whether it feeds a search index or an AI assistant.

Decide the unit before the paywall. Per-request payment suits what is consumed in a single request, so settle what is being sold, and who must still reach the rest freely, before any rule is applied.

Check ads.txt accounts against their registry. Ask partners to confirm the account and its sellers.json entry, and record whether a gap is a missing account or a registry that could not be read.

Give emergency responses an expiry. A 429 or 503 with Retry-After is a measure for hours or a day or two, with a named owner responsible for removing it.

More on each file and how Flowpane assesses it in robots.txt, ads.txt and cross-file coherence.

What we will watch for next month

For the next edition we will be watching whether Cloudflare's bot assessments reach its customers or Radar, whether the discovery pilot reports anything about what happens after a page is found, and how far paid machine access moves beyond its first beta. Each could change a decision a website owner makes, and none of them is a reason to add another file simply in order to look ready.

How we compiled this

This review covers 18 September to 18 October 2026 and draws only on dated primary sources, the companies' own announcements and documentation and the preprint itself, reviewed and linked where cited. Earlier capabilities appear only as background, and research findings are reported as their authors' results rather than ours; we did not repeat the preprint's measurements. Plans are described as plans: Cloudflare's 16 October announcement describes public bot scores and customer trust thresholds as future plans, and the Monetization Gateway is a closed beta limited to eligible US-based participants. We found no substantive change in the window to the specifications for sitemaps, security.txt or humans.txt, and nothing that called for a change to ai.txt or consent guidance in this edition.