Google Hacks

Hack 44 Scraping Google Results

Scraping the results of a Google search into a comma-delimited file.

Because you can use the Google API to get results and put them in any format you like, why in the world would you bother to do a manual search result on Google, save the results, and then scrape them with a Perl program? You might not want, or indeed be able, to do anything as fancy as the Google API allows; you might just want to grab some results, drop them into a spreadsheet, and go.

Just like we did in Peeling Phone Numbers [Hack #49], you can save Google web search results to a file, and then process them into a comma-delimited text file with a short Perl script.

Be sure to set your preferences [Hack #1] to 100 results per page to get the most out of this hack.

44.1 The Code

#!/usr/bin/perl
# google2csv.pl # Google Web Search Results exported to CSV suitable
# for import into Excel 
# Usage: perl google2csv.pl < results.html > results.csv

print qq{"title","url","size","domain suffix"\n};

my($results) = (join '', <>) =~
  m!<div>(.*?)</div>!mis;

while ( $results =~ 
m!<p><a href="?(.+?)"?>(.+?)</a>.+?\s+-\s+(\d+k)?!mgis
) { 
  my($url,$title, $size) = ($1||'',$2||'',$3||''); 
  my($suffix) = $url =~ m!\.(\w+)/!; 
  $title =~ s!"!""!g;         # double escape " marks 
  $title =~ s!<.+?>!!g; # drop all HTML tags print
  qq{"$title","$url","$size","$suffix"\n};
}

44.2 Running the Script

Run the script from the command line, specifying the result's HTML filename and name of the CSV file you wish to create or to which you wish to append additional results. For example, using results.html as input and results.csv as output:

$ perl google2csv.pl < results.html > results.csv

Leaving off the > and CSV filename sends the results to the screen for your perusal:

$ perl google2csv.pl < results.html

44.3 The Results

Here's a sample run on the results of a search for Mac OS X:

$ perl google2csv.pl < results.html
"title","url","size","domain suffix"
"Apple - Mac OS X","http://www.apple.com/macosx/","","com"
"Apple - Software - Mac OS X Server","http://www.apple.com/server/",
"29k","com"
"Mac OS X Development","http://developer.apple.com/macosx/","28k","com"
"Mac OS X Hints - Get the most from X!","http://www.macosxhints.com/",
"","com"
"Mac OS X Apps - The Source For Mac OS X Software",
"http://www.macosxapps.com/","39k","com"
"VersionTracker.com - free Macintosh software downloads for Mac 
OS ... ","http://www.versiontracker.com/macosx/","101k","com"
"O'Reilly Mac OS X Conference",
"http://conferences.oreillynet.com/macosx2002/","25k","com"
"MacNN | OS X","http://osx.macnn.com/","94k","com"
"???? - Mac OS X","http://www.apple.co.jp/macosx/","43k","jp"
"Apple - Support - Mac OS X",
"http://www.info.apple.com/usen/macosx/","36k","com"

You'll see that the program records four attributes to the CSV file: title, URL, size (when available), and top-level domain. The "snippet" of web page usually included with a Google result was omitted, because it's difficult to read in a spreadsheet format.

So why include the page size and domain? Research. If you're generating a set of results to be referred to later, it's handy to be able to sort them by suffix. "edu" results tend to be different from "org" results, which tend to be different from "com" results, and so on. Not to mention differing result sets by country, .uk versus .jp, for instance. And if you're generating links to contact later (to ask for a reciprocal link, for example), it's handy to be able to set apart the less-commercial suffixes such as .edu and .org.