Why AI crawlers can't see your site: JavaScript, bot protection and how to test both

Most advice about AI crawlers stops at robots.txt. That is the first gate, and our study of 1,069 top sites shows plenty of sites configure it deliberately. But a crawler that is allowed in can still leave with nothing. Three problems cause most of the cases we see: content that only exists after JavaScript runs, security rules that refuse the crawler before robots.txt matters, and URLs that fail when the crawler arrives.

1. Content that only exists after JavaScript runs

In December 2024 Vercel and MERJ published a log study of AI crawler behaviour on nextjs.org and the Vercel network, validated on two job-board sites. Their conclusion was blunt: none of the major AI crawlers they tracked currently render JavaScript. ChatGPT's and Claude's crawlers did download JavaScript files (11.50% and 23.84% of their requests) but did not execute them. Googlebot is the exception, and the study notes that Gemini uses Googlebot's infrastructure, which renders JavaScript. AppleBot also renders.

The practical meaning is simple. If your pricing table, product description or article text is injected by a script, a crawler that does not run scripts sees an empty shell. Your page can rank in Google, which renders JavaScript, and still contribute nothing to a ChatGPT or Perplexity answer. The study is almost two years old and vendors change behaviour, so do not treat it as permanent. Treat it as a reason to check, which takes one command (below).

The fix is to put the content you want cited in the HTML the server sends: server-side rendering or static generation. Client-side rendering is fine for interactive extras. This site is built that way: its pages are generated ahead of time, so the text is present with no script running.

2. Bot protection that stops the crawler first

Robots.txt is a request. A firewall is a decision. In our study we requested robots.txt from 2,000 popular domains using an ordinary browser user agent, and 137 of them answered 403 Forbidden. Those sites may be perfectly happy to be crawled by named, verified bots, but it shows how common it is for a security layer to refuse automated traffic before any file is read.

Some of that is now the default. In July 2025 Cloudflare announced that new domains would block AI crawlers unless the owner allows them. Older zones are not affected unless the owner switched on an AI-bot block or a managed robots.txt. Cloudflare also reports that its Content Signals Policy was enabled through managed robots.txt on over 3.8 million domains at launch. If your site sits behind a CDN you did not configure yourself, such as one set up by an agency or a host, find out what it does to AI user agents.

• Open your CDN or firewall dashboard and look for an AI crawler, bot-fight or managed robots.txt setting.
• Check the bot analytics or firewall log for 403, 429 and challenge responses to OAI-SearchBot, PerplexityBot and Claude-SearchBot.
• Do not block by IP address as a way to opt out. Anthropic warns that blocking IPs may not work reliably, because it can stop the crawler reading your robots.txt at all.

3. URLs that fail when the crawler arrives

The same Vercel and MERJ study measured response codes. 34.82% of ChatGPT's crawler fetches and 34.16% of Claude's hit a 404, against 8.22% for Googlebot. The study does not explain why, but a third of fetches ending in a 404 suggests these crawlers often request URLs that no longer exist. That makes housekeeping matter more than it used to: keep a current sitemap, redirect URLs you have moved with a permanent redirect instead of letting them die, and avoid long redirect chains. A crawler that spends its visit on dead links has less budget for your best page.

A five-minute test of what a crawler sees

1. Read your own robots.txt at /robots.txt and confirm the search and user agents are not blocked. The free audit on the home page does this and parses the groups correctly.
2. Check the status code each crawler's user agent receives for an important page (commands below). You want 200, not 403 or 429.
3. Check that your key sentence is in the raw HTML (second command). If it is missing, a crawler that skips JavaScript cannot see it.
4. As a visual cross-check, turn JavaScript off in your browser's developer tools and reload. What you see is roughly what a non-rendering crawler gets.
5. Look in your server or CDN logs for the real crawlers over the past week. See how to watch crawlers in your logs.

# What status does the crawler's user agent get?
curl -sIL -A "OAI-SearchBot" https://example.com/pricing | head -n 1
curl -sIL -A "PerplexityBot" https://example.com/pricing | head -n 1

# Is the sentence you want cited in the raw HTML?
curl -sL https://example.com/pricing | grep -i "starts at"

A test with curl only changes the user agent string. Firewalls usually decide using the source address as well, so a 200 here does not prove the real crawler is let in. The log check in step 5 is the real confirmation.

Symptom, cause and fix

Frequently asked questions