Hello, I am trying to integrate the databases for 5 websites into my site and allow my users to search through them and see the results side-by-side so they can compare them. I've made a blue print of what I am trying to do: ShopperGeek. com/demo Now these 5 sites have APIs and data-feed programs but I am not a part of them (nor can I be). Is it possible to integrate or access the databases of these 5 sites without being an affiliate and having access to their affiliate/API/data resources. I had a programmer who claimed he could do it but then told me he couldn't. Is this impossible or simply difficult? I would appreciate some answers and advice. Thanks, Zain Dhanani
Sounds like the project is a non-starter if you can't get hold of the APIs and data feeds for the sites you wish to compare. If this is a site you intend to make money from then you MUST comply with their licencing otherwise you could find yourself up in court. If you can't access their databases then the only way to get the data will be via what's called screen scraping, where you get the page with the formatted data, then strip out the formatting leaving just the data. But this is usually against sites' TCs. Did your programmer explain why he couldn't do the job, having previously claimed he could? Had he previously assumed access to the data feeds and APIs?
xpi0t0s, Thanks for your response. I don't understand why this goes against the websites rules when I will be sending them free traffic, how does this harm them? My previous programmer said they can permanently ban me from their servers. My programmer knew from the beginning that there wouldn't be access to the data feeds and APIs. He said he could do it and got started on the work. About a week later he said he couldn't do it for two reasons. He couldn't scrape the sites HTMLs because they are too complex. He also said he couldn't develop a program in which the search results always correspond perfectly with the searched term(s), because the search strings and data vary from each site.
You would need to discuss that with the website concerned. Explain how your proposal benefits them and they then have the choice to accept or reject your proposal. Scraping HTML is easy in theory but can get very complex depending on how much variety goes into it. Each site will vary the output from time to time and you could have a full time job decoding the HTML each time it changes, and you'd have a LOT of downtime for your own users while the changes were incorporated into your own website. So I suggest you start negotiating with the sites concerned to get access to their data feeds. If after your proposal they still say no then I suggest you look someplace else. Or you could learn to program, then you can find out first hand the difficulties involved in screen scraping.
