Use case
Level 2 / ContextAI crawler policy
Whether models may crawl, train on, or quote your content is now a business decision. For most sites it is made by accident, in a robots.txt nobody has reviewed since AI crawlers arrived.
Without a check
What goes wrong
- Some AI crawlers are blocked and others are not, with no stated reason.
- ai.txt says no to training while robots.txt lets the training crawlers in.
- llms.txt points assistants at pages that robots.txt forbids them to read.
With Flowpane
What changes
- Assesses robots.txt against a directory of named AI crawlers and reads Content-Signal directives.
- Flags conflicts between robots.txt, ai.txt and llms.txt as cross-file findings.
- Grades AI readiness from A to F, with the checks behind the grade.
What it looks like in Flowpane
Illustrative- Warningrobots.txt11 known AI crawlers have no explicit rule.
- Warningai.txtAI-Training: no, but GPTBot may crawl everything.
- Issuellms.txt4 linked URLs are blocked by robots.txt.
Level 3 / Method
How each check works.
Each file, how it fails in public, what true looks like, and exactly how it is checked.
robots.txtrobots.txtrobots.txt tells crawlers which paths they may fetch. One stray line can hide a whole site, so Flowpane checks it against RFC 9309 on every run.Readai.txtai.txtai.txt declares how you want AI systems to use your content. It states preferences without enforcing them, and must agree with what robots.txt says.Readllms.txtllms.txtllms.txt gives language models a curated Markdown map of your best pages. It is only useful while every link is current and reachable by AI crawlers.Read
See what your sites are claiming.
Add an origin and get a score, the gaps, and the evidence behind each one. Then Flowpane keeps checking, so the answer stays current.
Free during the beta. Flowpane only reads your origin.