Web crawler

A class of robot software that explores the Web by retrieving a Web document and following the links within that document.

Also known as a 'web robot' or 'web spider', it is a program or automated script which browses the World Wide Web in a methodical, automated manner.

Synonym spider.