Google Hacks

Hack 48 Scraping Google Catalogs

Scrape and save Google catalog search results to a comma-delimited file.

The December-Holiday-of-Your-Choice comes but once a year, but catalog shopping is a year-round joy. And Google Catalogs [Hack #33] makes it easier than ever.

Of course, just because you spend a rainy winter afternoon finding the perfect shawl for Aunt Prunella doesn't mean that you'll be able to replicate the search when you need it. Or maybe you want to do a search for several things and browse them at your leisure later.

Because Google Catalogs aren't supported by the Google API, this hack scrapes finds from the HTML of a Google Catalogs results page. It saves the catalog title, date or season, page number, and even a link to an image of the page itself at Google. Results are saved in CSV format, ready for import into a database or spreadsheet application.

Because Google's Terms of Service prohibits the automated access of their search engines except through the Google API, this hack does not actually connect to Google. Instead, it works on a page of results that you've saved from a Google Catalogs search you've run yourself. Simply save the results page as HTML source using your browser's File Save As... command.

As with the Google News hack [Hack #47], you can optimize the effectiveness of this hack by changing the results URL ever so slightly to tweak the order of and data displayed. By adding a &num=100 to the end of the catalog search results URL, you'll get up to 100 results instead of only the first.

For example, Figure 4-3 shows results of a query for the perfect shawl for that aunt.

48.1 The Code

#!/usr/bin/perl
# catalogs2csv.pl
# Google Catalogs Results exported to CSV suitable for import into Excel
# Usage: perl catalogs2csv.pl < catalogs.html > catalogs.csv

print qq{"title","link","date","page"\n};

my($results) = join '', <>;

while ( $results =~ m!<td>(.+?)&nbsp;&nbsp;&nbsp;<font size=-1>(.+?) - Pa
ge (\w+?) -.+?<a href="(/catalogs.+?)"><img src=.+?></a>&nbsp;&nbsp;!migs ) {
    my($title, $date, $page, $url) = ($1||'',$2||'',$3||'',$4||'');
    $title =~ s!"!""!g;  # double escape " marks
    my $output = qq{"$title","$url","$date","$page"\n};
    $output =~ s!&nbsp;! !g;  # clean spaces
    print $output;
} 

Figure 4-3. Google Catalogs results for "perfect shawl"

48.2 Running the Script

Run the script from the command line, specifying the Google Catalogs results HTML filename and name of the CSV file you wish to create or to which you wish to append additional results. For example, using catalogs.html as our input and catalogs.csv as our output:

$ perl catalogs2csv.pl < catalogs.html > catalogs.csv

Leaving off the > and CSV filename sends the results to the screen for your perusal.

48.3 The Results

You (and your aunt) appear to be in luck; there's an abundance of perfect shawls to be found in the Google Catalogs:

"title","link","date","page"
"Isabella","http://catalogs.google.com/catalogs?num=100
&hl=en&lr=&ie=UTF-8&issue=13655&catpage=cover","Fall 2002","3"
"Sovietski Collection","http://catalogs.google.com/catalogs?num=100
&hl=en&lr=&ie=UTF-8&issue=9447&catpage=cover","Summer 2002","37"
"Rego","http://catalogs.google.com/catalogs?num=100&hl=en
&lr=&ie=UTF-8&issue=12484&catpage=cover","2002","21"
"Crazy Crow Trading Post","http://catalogs.google.com/catalogs?num=100
&hl=en&lr=&ie=UTF-8&issue=12346&catpage=cover","2002","39"
"Winter Silks - Sale","http://catalogs.google.com/catalogs?num=100
&hl=en&lr=&ie=UTF-8&issue=10002&catpage=cover","Summer 2002","11"
...
"Previews","http://catalogs.google.com/catalogs?num=100
&hl=en&lr=&ie=UTF-8&issue=14468&catpage=cover","Oct 2002","381"

(Each listing actually occurs on its own line; lines are broken and occasionally shortened for the purposes of publication.)

48.4 Hacking the Hack

The output format may be altered to suit your fancy; see the Google News Scraper [Hack #47] for details on hacking the hack.