All > Technology > Search > SEO / Search Marketing
Googlebot is a search bot used by Google. It collects documents from the web to build a searchable index for the Google search engine. If a webmaster wishes to restrict the information on their site available to a Googlebot, or other well-behaved spider, they can do so by with the appropriate directives in a robots.txt file.
Google's search engine spider. Google has a shared crawl cache between their various spiders, including vertical search spiders and spiders associated with ad targeting. See also: Google's crawl caching proxy Google AdSense (see AdSense) Google AdWords (see AdWords)
- Browse Related Terms: Agent Name, applet, Bot, Client, Crawler, GoogleBot, Hallway page, Politeness Window, Pop-up, Robot, Spider, Standards compliant, user agent, Web Crawler