Search and AI

Crawler

Also called Bot, Spider.

A crawler is a program that visits web pages and follows their links, so a search engine or AI tool can use them.

Google's crawler is called Googlebot. OpenAI runs OAI-SearchBot for ChatGPT search. Crawlers find pages through links and sitemaps, and Googlebot follows the rules in your robots.txt file about which pages it may visit.

If a crawler cannot reach a page, that page cannot be indexed or quoted. Check that your robots.txt does not block Googlebot or OAI-SearchBot by mistake, and that each page is linked from another page. Then work through a technical SEO checklistsoon.

Go deeper

  • Technical SEO checklist for a new websiteComing soon
  • How to get your website indexed on GoogleComing soon

Related terms

What links here

Sources

Last checked 24 September 2026.

All glossary terms, A to Z