Dear Google. Please give us a web search API

google-babeSearch engine marketers have always had – and will always have, a need to extract and analyze search result data of various kinds. If there is an API we will use that. If not we will scrape. One way or another we will get that data. It would be better for all if there was good APIs available!

We could off course all just be doing manual searches on the search engines to gather the data we need. But really, who wants to do this? The engines of all should understand – they themselves are all about automating the editorial tasks that to a large degree used to be managed manually by librarians and the like.

Realistically the engines will never get rid of all automated web search scraping – but I think they could reduce it dramatically by providing reliable and good web search APIs.

I believe that most professional search engine marketers and companies spending serious resources on SEO would be happy to pay a fair fee for the use a good web API. I definitely would and I would recommend anyone to go for that option.

There have been, and there are still some, web search APIs available. But the problem has always been that the APIs where not very reliable, too restricted and often did not show the same results as a web searcher would get. That makes it unusable for us marketers for research and we are forced back to scraping.

I have on several occasions talked directly with Google about getting a good web search API. As I understand it it’s a question of resource – and maybe about politics.

I don’t understand the “political” issues here. As I said, we – the marketers, will get the data we need, some way or another. Not giving us an API will not stop us – it will just make it harder and not the least put a greater load on Google. Let’s stop that stupid game of cat and mouse now!

Google, what happened to your policy on giving the world better access to all data in the universe? :)

From a resources point of view I do realize and accept that even large companies such as Google have to prioritize. That’s fair enough. But let me assure you, dear Google, that there is already a very large market for a good web search API. We are many that are already ready to pay for it and I am sure more will come when the API becomes available.

So Google, please give us a good web search API – the sooner the better.

Skrevet af:

Mikkel deMib Svendsen er grundlægger af og fungerer i det daglige som kreativ direktør.

Kommentarer (6)

Trackback URL | Comments RSS Feed

  1. monchito siger:

    If it were any good, i would happily pay for it. Maybe if we just say this message often enough, googleguy will listen :)

  2. Kim Bach siger:

    They actually had one, SOAP based, but it has been discontinued in favour of their JavaScript API, which isn’t open. I suppose that it has something to do with the fact that you could use the SOAP API without having ads served.

    • Mikkel deMib Svendsen siger:

      The API Google used to have never worked the way it should. What we need is something far better – and we are ready to pay for it (partly for the service and partly for bypassing the ads) :)

  3. Adam siger:

    Google obviously realise rankings are part of a webmasters metrics as they show them in Webmaster Tools (innacurately).

    I agree, Google could easily reduce the amount of bandwidth used by scrapers by providing an API, this would cut out most of the code scrapers dont take notice of anyway. Should’ve brought this out years ago IMO.

  4. Kim Bach siger:

    Yes, the SOAP API was indeed very limited, I belive it had a cap that was something like 100 requests/hour (or was it a day).

    • Mikkel deMib Svendsen siger:

      The API from Google was also very slow, unreliable and did not always show the same results as regular web users would get. We need something better than that :)

Leave a Reply

If you want a picture to show with your comment, go get a Gravatar.