Spiders and Other Not-So-Creepy Crawlers

Okay, this is not an early Halloween post, rather a quick overview into terms like spiders, crawlers, and robots. What do these three terms have to do with web design and marketing? Alot actually!

Basically the terms spiders, crawlers, and robots are pretty interchangeable in the world of the web. They are all non-human, automated programs that are used to connect the dots and weave the big web called the Internet. Search engines launch their spider programs which run 24/7/365 to crawl through the millions of web pages, log key content words and phrases and follow links to other pages in the site and away from the site. In doing so, spiders basically catalog the web. Search engines then use this data to help users find what they are looking for.

Search engines take the data collected by the spiders and use it to create an index of sites that match the words being used in searches. They also use the information on pages the site links to and from to and percentage match in code (meta tags and content), to determine how high in the search the page will rank. Okay, there is more to the exact formula than that, you get the jist!

If you look at your web statistics closely, you can tell if and which spiders have visited. There also seems to be growing controversy over the need to submit your site to search engines based on the fact that the spiders are always going. They will eventually find your site, but manually submitting your site can help sometimes and it certainly won't hurt.

On a final note, not all spidering programs are good or being put to good use. The same type of program that helps Google build its great search engine, can also be used by spammers to harvest email addresses.

For more articles like this, don't forget to sign up to receive email updates from us (not too often, and we won't share your email address with anyone!).