Search engine marketers have always had – and will always have, a need to extract and analyze search result data of various kinds. If there is an API we will use that. If not we will scrape. One way or another we will get that data. It would be better for all if there was good APIs available!
We could off course all just be doing manual searches on the search engines to gather the data we need. But really, who wants to do this? The engines of all should understand – they themselves are all about automating the editorial tasks that to a large degree used to be managed manually by librarians and the like.
Realistically the engines will never get rid of all automated web search scraping – but I think they could reduce it dramatically by providing reliable and good web search APIs.
I believe that most professional search engine marketers and companies spending serious resources on SEO would be happy to pay a fair fee for the use a good web API. I definitely would and I would recommend anyone to go for that option.
There have been, and there are still some, web search APIs available. But the problem has always been that the APIs where not very reliable, too restricted and often did not show the same results as a web searcher would get. That makes it unusable for us marketers for research and we are forced back to scraping.
I have on several occasions talked directly with Google about getting a good web search API. As I understand it it’s a question of resource – and maybe about politics.
I don’t understand the “political” issues here. As I said, we – the marketers, will get the data we need, some way or another. Not giving us an API will not stop us – it will just make it harder and not the least put a greater load on Google. Let’s stop that stupid game of cat and mouse now!
Google, what happened to your policy on giving the world better access to all data in the universe?
From a resources point of view I do realize and accept that even large companies such as Google have to prioritize. That’s fair enough. But let me assure you, dear Google, that there is already a very large market for a good web search API. We are many that are already ready to pay for it and I am sure more will come when the API becomes available.
So Google, please give us a good web search API – the sooner the better.