Build a robots.txt file with custom crawler rules and sitemap references.
robots.txt is a text file at the root of your domain (/robots.txt) that tells web crawlers which pages they may or may not access. Compliant bots (Googlebot, Bingbot) honor it; malicious crawlers ignore it. Use it for crawl efficiency, not security.
To prevent ChatGPT, Claude, and other LLMs from training on your content, block bots like GPTBot, CCBot, ClaudeBot, anthropic-ai, Google-Extended, and PerplexityBot.
Which crawlers can access which paths on your site, crawl-delay, and pointers to your sitemap — it's a set of crawling instructions, not an access-control mechanism.
Not reliably — Disallow only asks crawlers not to crawl a URL; a disallowed page can still get indexed if other pages link to it. Use a noindex tag or header to actually prevent indexing.
At the root of your domain, e.g. https://example.com/robots.txt — it won't be recognized anywhere else.
Want more detail? Read How to Use robots.txt Generator: Practical Guide and Best Practices.