Checking whether AI crawlers can actually read a site
Have you ever looked at a finished page in Chrome and wondered what a crawler that never runs JavaScript receives from the same URL? The browser builds what you see by running the bundle. The crawlers that train and search with large language models never run it, so they keep only the HTML the server sent.
I published isready.ai to answer one question about that gap: when a training or search crawler fetches the page, does it get the article, or an empty document? It answers with thirty-two checks. The scan and the written fix are free, while monitoring is what you pay for.
Two machines read the same page
The first machine is the browser. Chrome executes JavaScript, so a React or Vue app that hydrates in the browser shows its content there. The second machine is a crawler such as GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot, which fetches raw HTTP with no JavaScript and short timeouts, under an honest user-agent. The same app can therefore arrive at the crawler as a blank document, since nothing runs the code that would have filled it in. Ordinary search dashboards stay green while that happens, because they are measuring a different machine.
Running the audit
The full audit is one command, pointed at the domain you want to check:
npx isreadyai yourdomain.com --deep --md
That command and the Markdown report it writes are both free. --deep walks past the homepage, up to ten pages on the free web scan. You can also open the site and scan from the page, with no account.
If you want the score as a CI gate, isreadyai/audit-action@v1 puts the report in the job summary and fails the run when the score drops below a threshold you set. A second pass uses a real browser, for agents that can click. The CI exit code still follows the crawler score, so that pass does not blow up an existing gate.
What the thirty-two checks look at
The default fetch imitates the crawler, requesting raw HTML the way those bots actually arrive. The thirty-two checks that run on what comes back are scored from 0 to 100. They ask whether the known AI user-agents are allowed in and whether the response contained the article or only a bundle those crawlers will not run. They also check that the page gives a model facts it can quote and that the transport is HTTPS. The writing itself is checked for the kinds of facts the GEO literature measured. That literature is the GEO paper by Aggarwal et al. at KDD 2024 (arXiv:2311.09735), where quotations, statistics, and citations helped while keyword stuffing was not one of the winners.
Each finding carries an observed value and its consequence, followed by a concrete fix, which is the written fix the free scan gives you. The score is versioned, so a later change in method does not silently rewrite old reports.
What the score leaves out
One file appears in the report without ever moving the score. llms.txt is a community proposal for pointing language models at a site’s documentation, but it does not control crawlers. isready reports whether the file exists and never moves the score because of it. Server logs on the product side show AI crawlers fetching that file in about 0.1% of visits. Google says in its guide for Search that you do not need special AI files for Google Search, which does not use llms.txt. isready is the check for the bots that never execute the bundle.
Those bots come in more than one kind. OpenAI documents several bots; Anthropic splits theirs the same way. A user-triggered fetch often ignores robots.txt, while a training crawl usually does not. isready promises no ranking inside ChatGPT. A 92 on this scan means the crawler got a page it could parse. Citation share inside an answer is a later measurement.
What stays free
The audit itself is free: the scan, the CLI, the Markdown, the deeper pass, the browser pass, and the GitHub Action. The scanner is open source, while the hosted dashboard, from Smart Squad S.r.l., is source-available under PolyForm Shield. Pro is €19 a month and Team is €49. That money pays for monitoring, history, a live badge, Ask-your-site (300 chats a month on Pro, 1500 on Team), and quotas for automated fix pull requests (200 a month on Pro, 1000 on Team). The image below shows what the hosted weekly report looks like.
You can scan a site from isready.ai without an account, or run the npx command above from a terminal.
References
- Aggarwal et al., GEO: Generative Engine Optimization, KDD 2024. arXiv:2311.09735
- llms.txt, community proposal. llmstxt.org
- Google, AI optimization guide, Google Search Central. developers.google.com