Robots.txt Generator

Create a customized robots.txt file to control web crawler indexing, block AI scrapers, and link your XML sitemap.

Methodology & standardsRuns on your device

Inputs

Results update live

Robots.txt Rules

Default Crawler Access (User-agent: *)

Disallowed Directories / Files

XML Sitemap URL

Crawl Delay (Seconds, optional)

Block AI Training Bots (GPTBot, ClaudeBot, CCBot)

robots.txt Preview

# robots.txt generated by Useful Tools Online

User-agent: *
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://usefultoolsonline.com/sitemap.xml

FAQ

Frequently asked questions

How the calculation works and where its limits are.

What is a robots.txt file?

A robots.txt file is a text file placed in the root directory of your website (e.g., yourdomain.com/robots.txt) that instructs search engine web crawlers and AI bots which pages or sections they are allowed or forbidden to crawl.

How do I block AI web scrapers in robots.txt?

You can add specific `User-agent: GPTBot`, `User-agent: ClaudeBot`, and `User-agent: CCBot` directives with `Disallow: /` to prevent AI training crawlers from scanning your content.

Where should I upload my robots.txt file?

Upload the generated file directly to your website's top-level root web server directory (e.g., `https://example.com/robots.txt`).