Spidering

The process of surfing the web, storing URLs and indexing keywords, links and text.

Typically, even the largest search engines cannot spider all of the pages on the net. This is due to the huge amount of data available, the speed at which the new data appears, the use of politeness windows and practical limits on the number of pages that can be visited in a given time . The search engines have to make compromises in order to visit as many sites as possible, and they do this in different ways. For example, some only index the home pages of each site, some only visit sites they're explicitly told about, and some make judgements about the importance of sites (from number and quality of inbound links) before "digging deeper" into the subpages of a site.