Search and AI
Crawler
Also called Bot, Spider.
A crawler is a program that visits web pages and follows their links, so a search engine or AI tool can use them.
Google's crawler is called Googlebot. OpenAI runs OAI-SearchBot for ChatGPT search. Crawlers find pages through links and sitemaps, and Googlebot follows the rules in your robots.txt file about which pages it may visit.
If a crawler cannot reach a page, that page cannot be indexed or quoted. Check that your robots.txt does not block Googlebot or OAI-SearchBot by mistake, and that each page is linked from another page. Then work through a technical SEO checklist.
Go deeper
- Technical SEO checklist for a new websiteRun the technical SEO checks that matter for a new website in one afternoon: turn off every setting that hides it, check Google sees your words, one secure address, your sitemap in Google and Bing and a free crawl with 0 errors, then test each new page the day you publish it and the whole site every Sunday.
- How to get your website indexed on GoogleGet your website indexed on Google: add it to Search Console, turn off anything that hides it, submit your sitemap and request indexing for your 5 main pages today, then link and request every new page each day and fix the pages that are not indexed every Sunday.
What links here
- How to add llms.txt and schema markup
- How to get your brand mentioned by ChatGPT
- How to get your company recommended by ChatGPT as a supplierB2B
- How to track AI traffic and ChatGPT mentions
- How to market a SaaS productSaaS
- How to design a pricing page
- How to get your website indexed on Google
- Technical SEO checklist for a new website
Sources
- Googlebot, Google Search Central, checked 24 September 2026
- Overview of OpenAI crawlers, OpenAI, checked 24 September 2026
Last checked 24 September 2026.