How Do Search Engines Work – Web Crawlers

Monday, June 7, 2010

Profit Instruments is an Internet Marketing training course by Ritoban Chakrabarti. This course teaches you how to earn money using SEO to get your small websites rank well in search engines. Searh engine and web crawlers will be discuss on the remaining part of this article.

The search engine will be a big factor for your website to get notice by a potential clients. Hence it is better to know how these search engines actually work and how they present information to the customer initiating a search.   

There are basically two types of search engines. The first is by robots named crawlers or spiders.

A spider is used by the search engines to index websites. The search engine’s spider will index the entire website when you submit your website pages to a certain search engine by completing their necessary submission web page. Search engine runs an automated program called ’spider’. Spider visits a web site, read the content on the actual site, the site’s Meta tags and also follow the links that the site connects. The spider then returns all that information back to a central depository, where the data is indexed. All the links on your website will be visited and those visited site will also be indexed. Some spiders will only index a certain number of pages on your site, so don’t create a site with 500 pages!

An occasional visit to the websites will be done by the spider to find out for any changes has been made. The moderators of the search engine determines the frequency for this to happen.

A spider may index up to a million web pages a day and acts like a library that index a book where it contains the table of contents, the actual content and the links and references to other websites during its search.

AltaVista, Lycos, Excite and Google are some examples.

When you ask a search engine to locate information, it is actually searching through the index which it has created and not actually searching the Web. Every search engine has it’s own algorithm that’s why they produce different rankings.

One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing or spamdexing. Algorithms analyze on how they are link to another pages in the World Wide Web. By checking how pages link to each other, an engine can both determine what a page is about, if the keywords of the linked pages are similar to the keywords on the original page.

You can read a full Profit Instruments Review by visiting: http://www.profitinstruments.net. Here you will learn more about this program and what it has to offer.

Tags: , ,


Leave a Reply