Crawler
A crawler is an automated program that browses web pages by following links, to analyze and index them.
Googlebot is the best known crawler, but every search engine and generative AI has its own. It reads HTML code, follows internal and external links, and passes what it finds to the index.
If a crawler cannot reach a page (server error, robots.txt block, broken link), that page stays invisible in search. A well structured site makes the crawler's job easier, and indexing follows.