Advanced Crawler Optimization Methods

rsq0522's Avatar, Join Date: Jan 2012
Light Poster
I am looking to construct web pages in a way that search engines will crawl it best.

I understand the basic tagging and on page SEO keyword related material.

I am looking for something more in the way the code is constructed or possible commands that can be given from the server or from more advanced programming so that pages can be crawled the best.

Any links, references, guidelines, articles, and subjects on what I am looking for would be very helpful and very much appreciated.
rsq0522's Avatar, Join Date: Jan 2012
Light Poster
Do you believe their is any server side or HTML bound commands I could give so that the most important content is crawled first or is this not possible?
jhonden's Avatar, Join Date: Sep 2011
Ambitious contributor
Content should be better, if you can gather a good page rank through link exchange then there is a chance of getting crawled easily and quite often.
ozsubasi's Avatar, Join Date: Jan 2012
Invasive contributor
Quote:
Originally Posted by rsq0522 View Post
Do you believe their is any server side or HTML bound commands I could give so that the most important content is crawled first or is this not possible?
I don't believe that there is. I know that you can restrict what is being crawled, but am not not aware of any way to tell the crawlers in what order you want them to crawl.

Last edited by ozsubasi; 6Mar2012 at 19:07..
Apoorva's Avatar, Join Date: Nov 2011
Ambitious contributor
Hi,
In my knowledge, there is no such code to drive/force crawler to crawl your site. You can do it only through submission in active high PR sites. Active sites are those sites, which are updating and crawled frequently.
For example. Directory sites, Social bookmarking sites and forum sites. These sites updates very frequently and therefore, crawled fast.
ozsubasi's Avatar, Join Date: Jan 2012
Invasive contributor
Quote:
Originally Posted by jhonden View Post
Content should be better, if you can gather a good page rank through link exchange then there is a chance of getting crawled easily and quite often.
It is apparently true that sites with higher PR get crawled more often, but that isn't what the OP is looking to achieve.
ozsubasi's Avatar, Join Date: Jan 2012
Invasive contributor
Quote:
Originally Posted by Apoorva View Post
Hi,
In my knowledge, there is no such code to drive/force crawler to crawl your site. You can do it only through submission in active high PR sites. Active sites are those sites, which are updating and crawled frequently.
For example. Directory sites, Social bookmarking sites and forum sites. These sites updates very frequently and therefore, crawled fast.
Again, the OP isn't looking to get the site crawled more often but to have it crawled in a certain order.
William9's Avatar, Join Date: Feb 2010
Go4Expert Member
Quote:
Originally Posted by rsq0522 View Post
Do you believe their is any server side or HTML bound commands I could give so that the most important content is crawled first or is this not possible?
+1 to this, as you can only make your web page search engine friendly and you can not do anything to get it crawled faster than search engine does it in its normal way.
h2so4 like this
ozsubasi's Avatar, Join Date: Jan 2012
Invasive contributor
[
Quote:
Originally Posted by William9 View Post
+1 to this, as you can only make your web page search engine friendly and you can not do anything to get it crawled faster than search engine does it in its normal way.
You have used a quote in your comment but you have answered something different. The question was whether important content could be crawled first, not faster.
h2so4 like this
ozsubasi's Avatar, Join Date: Jan 2012
Invasive contributor
The OP asked if there was an HTML command to make crawlers crawl their most important content first.
However, no-one has been able to offer such a solution though we have had answers concerning crawling faster, which is not the subject of the thread.

Therefore, if anyone can suggest a way to do what the OP asks, please post it.
Anyone who has any thoughts on getting crawled faster, please keep them to yourself as far as this thread is concerned.