# What Mastercard's and Visa's robots.txt files say about AI

> One lists 88 bots from the early 2000s. The other is empty. Neither names a single AI crawler, so both let every one of them in.

Spotted, 3 October 2026. The Flowpane team.

Two of the most trusted names in payments. Two very different robots.txt files. And on the question every publisher is now being asked, what AI crawlers may do with your pages, both files say exactly the same thing: nothing.

We checked both on 3 October 2026.

## Mastercard: a file from another era

`www.mastercard.com/robots.txt` starts sensibly. Four short groups for every crawler allow the site, keep out an admin area and two internal paths, and point to a sitemap index.

Then comes a section headed "Blocked Bot Requests": 88 named user agents, each sent away with `Disallow: /`. Most of them are email harvesters and offline-browsing tools from the early 2000s, including one that introduces itself as `Mozilla/4.0 (compatible; BullsEye; Windows 95)`. Lists like this were widely copied between sites twenty years ago.

A few things stand out:

- **The bots it names mostly no longer exist,** and the ones that misbehaved never read robots.txt anyway. It is an honour system.
- **Many of the entries cannot match.** Under RFC 9309, crawlers match on a short product token. Names with spaces, slashes or version numbers, like `Wget/1.5.3` or the Windows 95 string, are not valid tokens.
- **It blocks `ia_archiver`,** the crawler of Alexa, which closed in 2022.
- **It names no AI crawler.** GPTBot, ClaudeBot, Google-Extended and CCBot fall under the general rules, which allow almost everything.
- **Browsers download it.** The file is served with `Content-Disposition: attachment`, so opening it in a browser saves it instead of showing it. Crawlers don't mind. People checking the file do.

The server reports the file as last modified in June 2025, well into the era of AI crawlers.

## Visa: an empty file

`www.visa.com/robots.txt` answers `200 OK` with zero bytes. An empty robots.txt is valid. It means every crawler may fetch every path, and no sitemap is declared.

Visa's UK site, `www.visa.co.uk`, has a short, tidy file: one rule and one sitemap. It doesn't mention AI either.

## Silence is a policy

Neither file is broken. Search engines will index both sites happily. But a robots.txt that never mentions AI crawlers is still making a decision on its owner's behalf: yes to training, yes to AI search, yes to everything.

That may be exactly what these companies want. The question is whether anyone decided it, or whether the file simply hasn't been opened since the crawlers changed.

## What good looks like

- **Decide your AI stance, then write it down.** Training crawlers, AI search crawlers and fetches on a user's behalf are different agents with different tokens. Name the ones you mean.
- **Keep one group per crawler.** Merging is in the standard, but one clear group is easier to read and harder to get wrong.
- **Remove what no longer does anything.** Rules for crawlers that stopped existing years ago add noise and protect nothing.
- **Serve it as plain text.** `text/plain`, UTF-8, shown in the browser, so the people responsible for it can actually read it.
- **Give it an owner.** These files drift because they sit between teams. Someone should be told when they change.

More on the format in [robots.txt](/learn/robots-txt) and on stating AI preferences in [ai.txt](/learn/ai-txt).