Why AI search cannot read my site.

1. AI crawlers are blocked in robots.txt

A "Disallow: /" rule under "User-agent: *" or an explicit block for GPTBot, OAI-SearchBot, or PerplexityBot is the most common reason. Check your robots.txt and add explicit allow rules for each AI crawler.

# Check your robots.txt for these patterns:
# User-agent: GPTBot
# Disallow: /

# Fix:
User-agent: GPTBot
Allow: /

2. The page requires JavaScript to render

Most AI crawlers do not execute JavaScript. If your site content only appears after JavaScript runs (single-page apps, client-side rendering), AI crawlers see a blank or minimal page. Move to server-side rendering or static generation.

3. Missing or generic meta tags

A missing or vague meta description forces AI systems to interpret your page from scratch. A specific meta description acts as a quick summary that the model can trust.

4. No structured data (JSON-LD)

Without JSON-LD, AI models must guess your entity type, location, and contact information from prose. Add at minimum an Organization or LocalBusiness schema to give AI crawlers accurate entity facts.

5. Thin content (under 300 words)

AI systems prefer pages with substantive content. A page with under 300 words of readable text gives AI very little to work with. Add detailed descriptions, FAQ sections, or answer blocks to key pages.

6. Missing llms.txt

While not required, a missing llms.txt means AI agents have no machine-readable summary of your site. Adding /llms.txt is a low-effort signal that your site is AI-aware.

Related guides