Google Hacks

4.1 Hacks #41-49

As you've seen so far in this book, amazing things can be done with Google data access without ever using the Google API. This section of the book deals with Google applications that scrape Google's HTML to access its data rather than use the sanctioned Google API.

4.1.1 Scraping Versus the API

What is scraping and how is it different from using the Google API? Scraping is the act of using a program to pull information from an HTML page.

The Google API, on the other hand, allows you to query Google's search data directly, instead of pulling information from saved HTML as the scrapers in this section do.

4.1.2 Why Scrape Google Data?

Why have Google scrapers in this book anyway? Can't you do everything with the Google API? Alas, you can't. The Google API is a great way to search Google's main body of web pages, but it doesn't go much further than that. It's even limited in what it can pull from Google's main web search. For example, the Google API can't do a phonebook search. And it can't access the data from Google News, Google Catalogs, or most of Google's other specialty search properties.

That's too bad, because cool things can be done with the data from those searches. Need to track news stories for a certain topic? It's a lot easier to access several compiled searches from a spreadsheet than to manually scan HTML pages. Plus, once the information is loaded into a spreadsheet, you can resort and manipulate the data just about any way you please.

4.1.3 Things to Keep in Mind While Scraping

Though the programs provided in this section will provide you with hours of fun Google scraping, there are a few things you'll need to keep in mind.

• Scrapers break These scrapers are built based on the format of the Google results at this writing. If the format of results changes significantly, scrapers can, and will, break.

• Don't automate your scraping It might be tempting to go one step further and create programs that automate retrieving and scraping of Google pages.

  Don't do it. Retrieving Google pages by any automated methods other than Google's API is against Google's Terms of Service (TOS). So what, you might think. After all, they can't find you, right? They might not be able to find you specifically, but they can ban access from an entire block of IP addresses based on your IP address, affecting you and others around you. Would they do this? They would and they have. See http://news.com.com/2100-1023-883558.html for information about a Comcast block that took place in early 2002.

• Search results have limits Remember that even though you're scraping saved result pages, you're still subject to the limitations of Google's search—you can't get more than 1,000 results for a web search, for example. That said, make sure that you've set up your web search to get the most out of each scraped page. Make sure you're getting 100 results per page.

4.1.4 Using Scraped Data Files

Once you've got some results, scraped them, and saved them to a comma-delimited file, what use are they?

Most of the time, you think of comma-delimited files as spreadsheet files, but they're more versatile than that. Comma-delimited files can be loaded into databases and different data-handling programs. You could even write a Perl program that did something interesting with comma-delimited files (or get a geek friend to write it for you).

But the best thing about comma-delimited files is that they age well. They're just text files. You can use them with several different programs, you don't have to worry about compatibility issues, and they're small.

4.1.5 Not Fancy but Handy

Google scraping applications aren't as complicated as Google API applications, nor are they anywhere near as glamorous. But if you're trying to save search results from a variety of Google's properties, they really do come in handy.