1. We have moved from vBulletin to XenForo and you are viewing the site in the middle of the move. Though the functional aspect of everything is working fine, we are still working on other changes including the new design on Xenforo.
    Dismiss Notice

Is this impossible or just difficult to do?

Discussion in 'MySQL' started by zaindhanani, Aug 12, 2009.

  1. zaindhanani

    zaindhanani New Member

    Joined:
    Aug 12, 2009
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    0
    Hello,

    I am trying to integrate the databases for 5 websites into my site and allow my users to search through them and see the results side-by-side so they can compare them.


    I've made a blue print of what I am trying to do:

    ShopperGeek. com/demo


    Now these 5 sites have APIs and data-feed programs but I am not a part of them (nor can I be).


    Is it possible to integrate or access the databases of these 5 sites without being an affiliate and having access to their affiliate/API/data resources.


    I had a programmer who claimed he could do it but then told me he couldn't.




    Is this impossible or simply difficult?

    I would appreciate some answers and advice.


    Thanks,
    Zain Dhanani
     
  2. xpi0t0s

    xpi0t0s Mentor

    Joined:
    Aug 6, 2004
    Messages:
    3,012
    Likes Received:
    203
    Trophy Points:
    0
    Occupation:
    Senior Support Engineer
    Location:
    England
    Sounds like the project is a non-starter if you can't get hold of the APIs and data feeds for the sites you wish to compare. If this is a site you intend to make money from then you MUST comply with their licencing otherwise you could find yourself up in court.

    If you can't access their databases then the only way to get the data will be via what's called screen scraping, where you get the page with the formatted data, then strip out the formatting leaving just the data. But this is usually against sites' TCs.

    Did your programmer explain why he couldn't do the job, having previously claimed he could? Had he previously assumed access to the data feeds and APIs?
     
  3. zaindhanani

    zaindhanani New Member

    Joined:
    Aug 12, 2009
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    0
    xpi0t0s,

    Thanks for your response.

    I don't understand why this goes against the websites rules when I will be sending them free traffic, how does this harm them? My previous programmer said they can permanently ban me from their servers.


    My programmer knew from the beginning that there wouldn't be access to the data feeds and APIs.

    He said he could do it and got started on the work. About a week later he said he couldn't do it for two reasons.

    He couldn't scrape the sites HTMLs because they are too complex.
    He also said he couldn't develop a program in which the search results always correspond perfectly with the searched term(s), because the search strings and data vary from each site.
     
  4. xpi0t0s

    xpi0t0s Mentor

    Joined:
    Aug 6, 2004
    Messages:
    3,012
    Likes Received:
    203
    Trophy Points:
    0
    Occupation:
    Senior Support Engineer
    Location:
    England
    You would need to discuss that with the website concerned. Explain how your proposal benefits them and they then have the choice to accept or reject your proposal.

    Scraping HTML is easy in theory but can get very complex depending on how much variety goes into it. Each site will vary the output from time to time and you could have a full time job decoding the HTML each time it changes, and you'd have a LOT of downtime for your own users while the changes were incorporated into your own website.

    So I suggest you start negotiating with the sites concerned to get access to their data feeds. If after your proposal they still say no then I suggest you look someplace else. Or you could learn to program, then you can find out first hand the difficulties involved in screen scraping.
     

Share This Page