Is this actually free?

Yes. Run unlimited audits, see your overall score and partial results — no card, no trial expiry. The full 18-module report with issue-level detail is behind a free account.

What data do you store?

The URL you submit, the audit results, and your account email if you sign up. We don't install trackers on your site, don't store your page content, and don't sell data.

Can I run it on client sites?

Absolutely. It only needs a public URL — no code access, no DNS changes, no JavaScript snippets. Run it on your clients' sites, competitors' sites, or that side project you haven't touched in months.

Why should I configure specific directives for AI/LLM crawlers?

Configuring separate rules for LLM crawlers like GPTBot, ClaudeBot, and PerplexityBot determines whether AI platforms are allowed to read your site context. Allowing access builds index maps that help LLM search engines reference and cite your pages directly in dynamic answers.

What is Google-Extended and how does it relate to SEO?

Google-Extended is a user-agent token that allows webmasters to control whether their website content is parsed to train Google's Gemini models and AI products. Setting Google-Extended to Allow ensures Google's generative models have complete access to cite your brand details.

What happens if I block GPTBot in my robots.txt?

If you block GPTBot, OpenAI's training and dynamic search agents will not crawl your page. Your site will not be used in future ChatGPT training runs, and the AI agent may fail to cite your site when answering relevant user queries.

How does search crawl budget change for large websites?

For domains with millions of pages (like dynamically routed sites), search spiders have strict bandwidth budgets. If Googlebot spends time requesting unindexed parameter paths (filters, sorting), it won't index your new content routes. Blocking unneeded URLs in robots.txt optimizes your crawl budget.

What is the difference between blocking in robots.txt and adding a noindex tag?

Disallow rules in robots.txt only instruct bots not to crawl or request that page. However, if external domains link to it, Google can still index the URL. To fully block indexation, you must allow crawling so Googlebot sees a noindex tag in the DOM page metadata.

Last updated: August 1, 2026

GEO & AI Visibility

AI & Search Robots.txt Generator

Configure search crawler directives optimized for organic indexers and generative AI scrapers (ChatGPT, Gemini, Claude, Perplexity) to maximize GEO visibility.

1. Base Crawl Settings

XML Sitemap URL

Crawl Delay

2. AI & LLM Bot Permissions (GEO Optimization)

Disallowed Paths (one per line)

Allowed Paths (one per line)

AI Search Bot Visibility

ChatGPT Search (OAI-Search)ALLOWED

GPTBot (OpenAI Training)ALLOWED

ClaudeBot (Anthropic)ALLOWED

Perplexity AI CrawlerALLOWED

Google Gemini (Google-Extended)ALLOWED

Apple Intelligence (Applebot)ALLOWED

Cohere AI IndexerALLOWED

Meta Llama (FacebookBot)ALLOWED

Generated robots.txt

Save this content exactly as a robots.txt file inside your website's root folder.

# AI & Search Robots.txt Builder
# Generated via siteauditscore.com

User-agent: *
Disallow: /api/
Disallow: /admin/
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Cohere-crawler
Allow: /

User-agent: FacebookBot
Allow: /

Sitemap: https://siteauditscore.com/sitemap.xml

18 Parallel Audits in Seconds

Want a Complete 18-Module SEO & AI Audit?

Run your site through our full parallel audit suite: Schema validation, Core Web Vitals, Content E-E-A-T, AI Crawler indexation, and Google AI Overview CTR simulator.

⚡ Technical & Vitals

LCP, CLS, INP, schema markup validation, and security headers checked in parallel.

🤖 AI & GEO Indexing

Perplexity, ChatGPT Search, RAG chunking, and direct answer factual density scoring.

📊 360° Actionable Plan

Prioritized issue list with code-level fix snippets and instant PDF executive reports.

Get Started Free Today Run Free Audit

100% Data Privacy GuaranteedNo Credit Card RequiredInstant Parallel Results

Robots.txt Optimization Guide: Maximize Crawl Budgets & AI Visibility

An actionable guide explaining how search engine bots crawl your website and how to configure robots.txt to drive SEO ranking value.

What is a Robots.txt File?

Robots.txt is a plain text file saved in the root directory of your website. Its job is to provide instructions to search engine bots (like Googlebot) and AI web crawlers (like GPTBot) regarding which pages and folders they are allowed to request.

Disallow Directive: Restricts crawlers from requesting specific paths (e.g. checkout forms or private dashboards).
Sitemap Declaration: Points bots to your sitemap XML link for fast discovery and indexation of new pages.

Why Crawl Budget Matters (Key Statistics)

Search engine spiders process over 8.5 billion search queries daily with a limited bandwidth allowance per domain, known as a crawl budget. Disallowing redundant query parameters can save up to 45% of your allocated crawl budget, ensuring search engines index core pages 3x faster. Over 68% of generative AI citations in ChatGPT Search rely on unblocked robots.txt permissions.

Block URL Parameters: Block filters in robots.txt (e.g., Disallow: /*?sort=) to optimize crawl frequency.
Optimize Server Bandwidth: Keeping crawlers out of staging routes prevents server slow-downs.

Configuring Robots.txt for AI Search & GEO Visibility

The emergence of AI search engines (like ChatGPT Search and Perplexity) has introduced new bots to the web ecosystem. Allowing these bots to crawl your marketing and documentation pages determines whether your brand is referenced and cited directly inside dynamic AI answers.

AI Crawler Guidelines

Allow GPTBot & PerplexityBot: Ensure these are not blocked in your robots.txt file to be cited in ChatGPT/Perplexity search results.
Configure Google-Extended: Decide whether your pages should be used to train Google Gemini models.

Checklist: Auditing your Robots.txt File

Step 1

Check for Site-wide Blocks

Ensure you do not accidentally block search engines entirely with a directive like Disallow: / on your live server.

Step 2

Protect Staging Sites

Do not rely on robots.txt to hide sensitive folders. Block indexing of private routes using server password shields.

Step 3

Reference the Sitemap

Always include a Sitemap: https://yourdomain.com/sitemap.xml line at the bottom of the file to guide search bots.

AI & Search Bots Quick Reference Table

User-Agent Token	Crawler Purpose	Primary Content Ingested	GEO Visibility Status
Googlebot	Google search engine crawler	All public HTML & media pages	Critical. Required for search engine indexing.
OAI-SearchBot	ChatGPT Search indexer	Dynamic text summaries & links	Highly Recommended. Required for ChatGPT citations.
PerplexityBot	Perplexity dynamic search agent	Real-time answer resources	Recommended. Determines citation inclusion.
ClaudeBot	Anthropic AI model agent	Technical documentation & guides	Recommended. Aids context training.