Spider

Mention a spider and the first thing you probably think of is those crawling, 8 legged, terrors that you would not come within 20 metres of. However an Internet spider is a bot that any webmaster is glad to have crawl right over their website! To be more precise spiders are used by search engines to crawl your page. This point may seem trival at first, but understanding how spiders crawl and index your pages, can be invaluable in improving your search engine standing.

There are different kinds of spiders that search engines use; deep, shallow and verification are the three main types.

A Deep spider is one that crawls through all levels of your web site, doesn't matter how deep it is. Altavista is an example of a search engine using deep spider. This type of spider, will study your site and bring the related information back to the search engine to store in the database, the information they are looking are like title, meta tag, relevance and what ever else search engines deem important for indexing. It also serves the secondary functions of verifying links, and remove any dead links in the main database.

A Shallow spider, is one that crawls the first level, or a few levels after that, an example of search engine using a shallow spider is Excite. It carries a similar process as the deep spider in terms of information gathered, but it stops at a pre-definded depth. So if you page is in 3 or 4 level deep with a website, there is a good chance, this type of spider will not crawl it.

A Verification spider, is a spider that is sent out to the site to verify that the URL actually exists. It does not bring back any information or particular about the web site. This is mostly used by the directory type of search engine such as Yahoo. This type of spider does not care what sort of title or what sort of content you have, they just come and check whether the site is active or not, then report back.

Above is the spider family and what they do. Even if two different engines use a shallow spider, they can differ again in terms how they collect and organize the data. Each engine has their own ranking algorithm based on what they think is important. Commonly they all favour content rich and theme based websites. This is the new direction search engine has move towards in order to return a better search result.

Generally there are a couple things the spider will look for; title, meta tags, texts in the page related to the keyword, checking whether you are spamming or not, and such. If you write whole page of nothing but the keyword, the spider is smart enough to pick that up and remove your listing, or worse, ban the domain. So a well designed page with rich content is often the best way to go.

For more information on web promotion via search engines, please visit the Dynamic Submission 2000 Homepage.

Act now and submit your web sites to hundreds of search engines by downloading the Dynamic Submission 2000.

Also check our related article, web spider...

Meta Tag Generator, an excellent read!