1. We have moved from vBulletin to XenForo and you are viewing the site in the middle of the move. Though the functional aspect of everything is working fine, we are still working on other changes including the new design on Xenforo.
    Dismiss Notice

Advanced Crawler Optimization Methods

Discussion in 'Internet Marketing' started by rsq0522, Jan 24, 2012.

Thread Status:
Not open for further replies.
  1. rsq0522

    rsq0522 New Member

    Joined:
    Jan 11, 2012
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    0
    I am looking to construct web pages in a way that search engines will crawl it best.

    I understand the basic tagging and on page SEO keyword related material.

    I am looking for something more in the way the code is constructed or possible commands that can be given from the server or from more advanced programming so that pages can be crawled the best.

    Any links, references, guidelines, articles, and subjects on what I am looking for would be very helpful and very much appreciated.
     
  2. rsq0522

    rsq0522 New Member

    Joined:
    Jan 11, 2012
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    0
    Do you believe their is any server side or HTML bound commands I could give so that the most important content is crawled first or is this not possible?
     
  3. jhonden

    jhonden New Member

    Joined:
    Sep 13, 2011
    Messages:
    190
    Likes Received:
    4
    Trophy Points:
    0
    Content should be better, if you can gather a good page rank through link exchange then there is a chance of getting crawled easily and quite often.
     
  4. ozsubasi

    ozsubasi New Member

    Joined:
    Jan 26, 2012
    Messages:
    505
    Likes Received:
    29
    Trophy Points:
    0
    Occupation:
    Real Estate
    Location:
    Altinkum, Didim, Turkey
    Home Page:
    I don't believe that there is. I know that you can restrict what is being crawled, but am not not aware of any way to tell the crawlers in what order you want them to crawl.
     
    Last edited: Mar 6, 2012
  5. Apoorva

    Apoorva New Member

    Joined:
    Nov 15, 2011
    Messages:
    111
    Likes Received:
    0
    Trophy Points:
    0
    Hi,
    In my knowledge, there is no such code to drive/force crawler to crawl your site. You can do it only through submission in active high PR sites. Active sites are those sites, which are updating and crawled frequently.
    For example. Directory sites, Social bookmarking sites and forum sites. These sites updates very frequently and therefore, crawled fast.
     
  6. ozsubasi

    ozsubasi New Member

    Joined:
    Jan 26, 2012
    Messages:
    505
    Likes Received:
    29
    Trophy Points:
    0
    Occupation:
    Real Estate
    Location:
    Altinkum, Didim, Turkey
    Home Page:
    It is apparently true that sites with higher PR get crawled more often, but that isn't what the OP is looking to achieve.
     
  7. ozsubasi

    ozsubasi New Member

    Joined:
    Jan 26, 2012
    Messages:
    505
    Likes Received:
    29
    Trophy Points:
    0
    Occupation:
    Real Estate
    Location:
    Altinkum, Didim, Turkey
    Home Page:
    Again, the OP isn't looking to get the site crawled more often but to have it crawled in a certain order.
     
  8. William9

    William9 New Member

    Joined:
    Feb 9, 2010
    Messages:
    40
    Likes Received:
    2
    Trophy Points:
    0
    Location:
    India
    Home Page:
    +1 to this, as you can only make your web page search engine friendly and you can not do anything to get it crawled faster than search engine does it in its normal way.
     
    h2so4 likes this.
  9. ozsubasi

    ozsubasi New Member

    Joined:
    Jan 26, 2012
    Messages:
    505
    Likes Received:
    29
    Trophy Points:
    0
    Occupation:
    Real Estate
    Location:
    Altinkum, Didim, Turkey
    Home Page:
    [
    You have used a quote in your comment but you have answered something different. The question was whether important content could be crawled first, not faster.
     
    h2so4 likes this.
  10. ozsubasi

    ozsubasi New Member

    Joined:
    Jan 26, 2012
    Messages:
    505
    Likes Received:
    29
    Trophy Points:
    0
    Occupation:
    Real Estate
    Location:
    Altinkum, Didim, Turkey
    Home Page:
    The OP asked if there was an HTML command to make crawlers crawl their most important content first.
    However, no-one has been able to offer such a solution though we have had answers concerning crawling faster, which is not the subject of the thread.

    Therefore, if anyone can suggest a way to do what the OP asks, please post it.
    Anyone who has any thoughts on getting crawled faster, please keep them to yourself as far as this thread is concerned.
     
  11. ozsubasi

    ozsubasi New Member

    Joined:
    Jan 26, 2012
    Messages:
    505
    Likes Received:
    29
    Trophy Points:
    0
    Occupation:
    Real Estate
    Location:
    Altinkum, Didim, Turkey
    Home Page:
    No more suggestions - thread closed.
     
Thread Status:
Not open for further replies.

Share This Page