Go4Expert

Go4Expert (http://www.go4expert.com/)
-   Internet Marketing (http://www.go4expert.com/forums/internet-marketing-forum/)
-   -   Advanced Crawler Optimization Methods (http://www.go4expert.com/forums/advanced-crawler-optimization-methods-t27657/)

rsq0522 25Jan2012 00:04

Advanced Crawler Optimization Methods
 
I am looking to construct web pages in a way that search engines will crawl it best.

I understand the basic tagging and on page SEO keyword related material.

I am looking for something more in the way the code is constructed or possible commands that can be given from the server or from more advanced programming so that pages can be crawled the best.

Any links, references, guidelines, articles, and subjects on what I am looking for would be very helpful and very much appreciated.

rsq0522 25Jan2012 20:54

Re: Advanced Crawler Optimization Methods
 
Do you believe their is any server side or HTML bound commands I could give so that the most important content is crawled first or is this not possible?

jhonden 1Mar2012 17:11

Re: Advanced Crawler Optimization Methods
 
Content should be better, if you can gather a good page rank through link exchange then there is a chance of getting crawled easily and quite often.

ozsubasi 1Mar2012 18:16

Re: Advanced Crawler Optimization Methods
 
Quote:

Originally Posted by rsq0522 (Post 91663)
Do you believe their is any server side or HTML bound commands I could give so that the most important content is crawled first or is this not possible?

I don't believe that there is. I know that you can restrict what is being crawled, but am not not aware of any way to tell the crawlers in what order you want them to crawl.

Apoorva 15Mar2012 14:28

Re: Advanced Crawler Optimization Methods
 
Hi,
In my knowledge, there is no such code to drive/force crawler to crawl your site. You can do it only through submission in active high PR sites. Active sites are those sites, which are updating and crawled frequently.
For example. Directory sites, Social bookmarking sites and forum sites. These sites updates very frequently and therefore, crawled fast.

ozsubasi 15Mar2012 14:45

Re: Advanced Crawler Optimization Methods
 
Quote:

Originally Posted by jhonden (Post 92927)
Content should be better, if you can gather a good page rank through link exchange then there is a chance of getting crawled easily and quite often.

It is apparently true that sites with higher PR get crawled more often, but that isn't what the OP is looking to achieve.

ozsubasi 15Mar2012 14:47

Re: Advanced Crawler Optimization Methods
 
Quote:

Originally Posted by Apoorva (Post 93412)
Hi,
In my knowledge, there is no such code to drive/force crawler to crawl your site. You can do it only through submission in active high PR sites. Active sites are those sites, which are updating and crawled frequently.
For example. Directory sites, Social bookmarking sites and forum sites. These sites updates very frequently and therefore, crawled fast.

Again, the OP isn't looking to get the site crawled more often but to have it crawled in a certain order.

William9 11Apr2012 15:26

Re: Advanced Crawler Optimization Methods
 
Quote:

Originally Posted by rsq0522 (Post 91663)
Do you believe their is any server side or HTML bound commands I could give so that the most important content is crawled first or is this not possible?

+1 to this, as you can only make your web page search engine friendly and you can not do anything to get it crawled faster than search engine does it in its normal way.

ozsubasi 11Apr2012 17:12

Re: Advanced Crawler Optimization Methods
 
[
Quote:

Originally Posted by William9 (Post 94148)
+1 to this, as you can only make your web page search engine friendly and you can not do anything to get it crawled faster than search engine does it in its normal way.

You have used a quote in your comment but you have answered something different. The question was whether important content could be crawled first, not faster.

ozsubasi 11Apr2012 17:17

Re: Advanced Crawler Optimization Methods
 
The OP asked if there was an HTML command to make crawlers crawl their most important content first.
However, no-one has been able to offer such a solution though we have had answers concerning crawling faster, which is not the subject of the thread.

Therefore, if anyone can suggest a way to do what the OP asks, please post it.
Anyone who has any thoughts on getting crawled faster, please keep them to yourself as far as this thread is concerned.


All times are GMT +5.5. The time now is 21:02.