Free AI crawler and llms.txt tools
Each of these runs one check from the Cite Files scan on its own, with no account: read an llms.txt, write one, see whether an AI crawler is blocked, test a robots.txt, or build an ai.txt. The full check below does all of them at once, and more.
llms.txt validator: check your llms.txt file
Paste an llms.txt or enter a site address and get the file graded, with each problem listed. Free, no account.
llms.txt generator: build one from your own pages
Enter a site address and get an llms.txt built only from that site's own page titles and meta descriptions. Free, no account.
Is GPTBot blocked on my site? AI crawler access checker
Enter a site address and see whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are served or blocked, beside what robots.txt says. Free, no account.
robots.txt tester for AI crawlers
Paste a robots.txt or enter a site address and see, agent by agent, whether each AI crawler is allowed or disallowed. Free, no account.
ai.txt generator
Paste a robots.txt or enter a site address and get an ai.txt that states the same position on AI training. Free, no account.
Each tool states on its own page what it requests from your site. A tool follows a redirect only within your own site, never to another one. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored.
Or run every check at once
Free, and the summary needs no account. Read the rubric first if you would rather know how it is judged before it judges you.
Last reviewed . Maintained by Cite Files — corrections to [email protected].