What is a Crawler?

What is a Crawler?

A crawler is a piece of code ( a programme) sent out by a search engine to gather data from the websites on the Internet. It may also be called a ‘spider’ or ‘bot’.

Crawlers were initially called crawlers or spiders because of the way they moved around the web, by ‘crawling’ through web site code and moving from one page to another or from site to site using hyperlinks.

Crawlers are essential to websites because without them the search engines would not know that a site exists. We talk about making a website ‘crawler friendly’ because this is a very important part of SEO.

How a Crawler works

  • The crawler travels from one hyperlinked page to another and crawls through all the Html code collecting data. (Html is the mark-up language used to create web pages).
  • Once a site has been visited by a crawler we say it has been ‘crawled’ or ‘spidered’.
  • The data it collects is then returned to the search engine database. When information on a site has been entered in to a search engine database we say the site has been ‘indexed’.
  • The crawler keeps moving from site to site collecting information.
  •  When any changes are made to a site it may be some time before those changes are recognised by the crawler and added to the database.

This is how Google is able to keep records of what your site is about.