Search and AI
Crawler
Also called Bot, Spider.
A crawler is a program that visits web pages and follows their links, so a search engine or AI tool can use them.
Google's crawler is called Googlebot. OpenAI runs OAI-SearchBot for ChatGPT search. Crawlers find pages through links and sitemaps, and Googlebot follows the rules in your robots.txt file about which pages it may visit.
If a crawler cannot reach a page, that page cannot be indexed or quoted. Check that your robots.txt does not block Googlebot or OAI-SearchBot by mistake, and that each page is linked from another page. Then work through a technical SEO checklistsoon.
Go deeper
- Technical SEO checklist for a new websiteComing soon
- How to get your website indexed on GoogleComing soon
What links here
Sources
- Googlebot, Google Search Central, checked 24 September 2026
- Overview of OpenAI crawlers, OpenAI, checked 24 September 2026
Last checked 24 September 2026.