Checking whether AI crawlers can actually read a site

I published isready.ai because a site can look finished in Chrome and still be empty to the crawlers that train and search with large language models. Chrome executes JavaScript. Googlebot can render it too. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot, as their vendors document the training and search crawls, fetch raw HTTP: no JavaScript, short timeouts, and an honest user-agent. A React or Vue app that hydrates in the browser can therefore arrive as a blank document. Classic SEO dashboards stay green while that happens, because they are measuring a different machine.

The homepage still puts it as a question: “The future is AI. Is your website ready for AI?” The npm package is isreadyai.

How to run it

The full audit is one command:

npx isreadyai yourdomain.com

--llm writes a resolution plan. --md writes markdown solutions. Both stay free. --json and --quiet are there for scripts. --deep walks more than the homepage (on the free web scan, sitemap plus links, up to ten pages). --smart-ai is a second pass with a real browser, using agent-browser from Vercel Labs, for agents that can click. The CI exit code still follows the crawler score, so turning on --smart-ai does not blow up an existing gate.

You can also open https://isready.ai and scan from the page, with no account. If you want the score as a CI gate, there is a GitHub Action at isreadyai/audit-action (isreadyai/audit-action@v1). The full Markdown report lands in the job summary. Set a threshold and the run fails when the score drops below it.

What the 32 checks look at

The default fetch imitates the crawler: raw HTML, the way those bots actually arrive. Thirty-two checks are scored from 0 to 100 across five dimensions. Crawler access asks whether robots.txt and challenges let the well-known AI user-agents in (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended), and whether Cloudflare-style challenges, redirects, TTFB, or noindex are in the way. Rendering asks whether the response contained the article, or only a JavaScript bundle a training crawler will not run. Structured data asks whether JSON-LD, meta, Open Graph, and author signals give a model facts it can quote without guessing. Trust and security covers HTTPS, TLS, HSTS, and mixed content. Content GEO looks at the kinds of facts the GEO literature measured: quotations, statistics, and citations. Keyword stuffing was not one of the winners. The paper is Aggarwal et al., GEO: Generative Engine Optimization, KDD 2024 (arXiv:2311.09735).

Each finding carries an observed value, a consequence, and a concrete fix. The score is versioned, so a later methodology change does not silently rewrite old reports.

llms.txt and the other bots

llms.txt is a community proposal for pointing language models at a site’s documentation. It does not control crawlers. isready reports whether the file exists and never moves the score because of it. Server logs on the product side show AI crawlers fetching that file in about 0.1% of visits. Google’s AI optimization guide is worth reading once on this point: you do not need special AI files for Google Search, and Google Search does not use llms.txt. isready sits next to that work. It is the check for the bots that never execute the bundle.

OpenAI documents several bots (GPTBot, OAI-SearchBot, ChatGPT-User), and Anthropic splits ClaudeBot, Claude-SearchBot, and Claude-User the same way. A user-triggered fetch often ignores robots.txt. A training crawl usually does not. Mixing those jobs is how a site ends up with a file that does not do what people think it does. isready does not promise ChatGPT rankings. A 92 on this scan means the crawler got a page it could parse. Citation share inside an answer is a later measurement, and a different product.

What is free and what is hosted

The scan, the CLI, --md, --llm, --deep, --smart-ai, and the GitHub Action are free. The scanner engine is MIT. Pro is €19 a month and Team is €49. That money is monitoring, history, a live badge, Ask-your-site (300 chats a month on Pro, 1500 on Team), and quotas for automated fix PRs (200 a month on Pro, 1000 on Team). The hosted dashboard is source-available under PolyForm Shield, Smart Squad S.r.l.

Run the CLI. Pay if you want it watched.