Search and AI
robots.txt
robots.txt is a small text file at the root of your website that tells crawlers which pages they may visit.
It lives at yoursite.com/robots.txt. Each rule names a crawler, like Googlebot or OAI-SearchBot, and the paths it may or may not visit. Google says robots.txt is not a way to keep a page out of search; use a noindex tag for that.
One wrong line can block your whole site. Open your robots.txt today and check it does not say "Disallow: /" for the crawlers you want. Add your sitemap address to it. Put it on your technical SEO checklist.
Go deeper
- Technical SEO checklist for a new websiteRun the technical SEO checks that matter for a new website in one afternoon: turn off every setting that hides it, check Google sees your words, one secure address, your sitemap in Google and Bing and a free crawl with 0 errors, then test each new page the day you publish it and the whole site every Sunday.
- How to add llms.txt and schema markupAdd llms.txt and schema markup in about an hour: check what your site builder already made, put Organization markup on your home page and the right markup on each kind of page, publish a short llms.txt, test each new page's markup the day you publish it, and fix errors every Sunday, knowing Google ignores llms.txt for its AI answers.
What links here
- How to add llms.txt and schema markup
- How to get your brand mentioned by ChatGPT
- How to get your company recommended by ChatGPT as a supplierB2B
- How to get your products recommended by ChatGPTD2C
- How to show up in Google AI Overviews
- How to track AI traffic and ChatGPT mentions
- How to get traffic to a new website for free
- How to market a SaaS productSaaS
- How long does SEO take to work?
- How to do SEO for a Shopify storeD2C
- How to get your website indexed on Google
- How to use Google Search Console
- Technical SEO checklist for a new website
Sources
- Introduction to robots.txt, Google Search Central, checked 24 September 2026
Last checked 24 September 2026.