Search and AI

robots.txt

robots.txt is a small text file at the root of your website that tells crawlers which pages they may visit.

It lives at yoursite.com/robots.txt. Each rule names a crawler, like Googlebot or OAI-SearchBot, and the paths it may or may not visit. Google says robots.txt is not a way to keep a page out of search; use a noindex tag for that.

One wrong line can block your whole site. Open your robots.txt today and check it does not say "Disallow: /" for the crawlers you want. Add your sitemap address to it. Put it on your technical SEO checklistsoon.

Go deeper

  • Technical SEO checklist for a new websiteComing soon
  • How to add llms.txt and schema markupComing soon

Related terms

What links here

Sources

Last checked 24 September 2026.

All glossary terms, A to Z