Basic concepts ==>> Search engines Before we start talking about search engine optimization we need to understand how search engines work. Basically, each search engine consists of 3 parts: 1. The Crawler (or the spider). This part of a search engine is a simple robot that downloads pages of a website and crawls them for links. Then, it opens and downloads each of those links to crawl (spider) them too. The crawler visits websites periodically to find the changes in their content and modify their rankings accordingly. Depending on the quality of a website and the frequency of its content updates this may happen from say once per month up to several times a day for a high popularity news sites. The crawler does not rank websites itself. Instead, it simply passes all crawled websites to another search engine module called the indexer. 2. The Indexer. This module stores all the pages crawled by the spider in a large database called the index. Think of it as the index in a paper book: you find a word and see which pages mention this word. The index is not static, it updates every time the crawler finds a new page or re-crawls the one already presented in the index. Since the volume of the index is very large it often takes time to commit all the changes into the database. So one may say that a website has been crawled, but not yet indexed. Once the website with all its content is added to the index, the third part of the search engine begins to work. 3. The ranker (or search engine software). This part interacts with user and asks for a search query. Then it sifts millions of indexed pages and finds all of them that are relevant to that query. The results get sorted by relevance and finally are shown to a user. What is relevance and how would one determine if a page is more or less relevant to a query? Here comes the tricky part - the ranking factors...
It is well said. To get informed about search engine is as same important as to get informed about search engine optimization.
Hi, Very nice write up. I always noticed that most of the people are confused by the terms crawling and indexing, you have explained both of them in very simple language. Keep it up the good work.
I am little bit confuse with your explanation. You have mentioned that Ranker (Search Engine Software) will modify & decide ranking of web page then how crawler can modify their ranking?
You have provided such a important fact on paper in the simplest of words. Hands off to you. really glad to read it and to know it also in simple way. It brushed my existing knowledge.
Hi, The Ranking modified by the crawler is the Page rank (or ranking) decides the relevancy of content on the website. And the Ranking modified by the ranker is the SERP ranking which consults both the crawler and indexer to show the results for the submitted query. I think you get the point now.
