Search and AI
robots.txt
robots.txt is a small text file at the root of your website that tells crawlers which pages they may visit.
It lives at yoursite.com/robots.txt. Each rule names a crawler, like Googlebot or OAI-SearchBot, and the paths it may or may not visit. Google says robots.txt is not a way to keep a page out of search; use a noindex tag for that.
One wrong line can block your whole site. Open your robots.txt today and check it does not say "Disallow: /" for the crawlers you want. Add your sitemap address to it. Put it on your technical SEO checklistsoon.
Go deeper
- Technical SEO checklist for a new websiteComing soon
- How to add llms.txt and schema markupComing soon
What links here
Sources
- Introduction to robots.txt, Google Search Central, checked 24 September 2026
Last checked 24 September 2026.