Google Hacks

Hack 51 Looping Around the 10-Result Limit

If you want more than 10 results, you'll have to loop.

The Google API returns only 10 results per query. Ten results is plenty for some queries, but for most applications, 10 results barely scratches the surface. If you want more than 10 results, you're going to have to loop, querying for the next set of 10 each time. The first query returns the top ten. The next, 11 through 20. And so forth.

This hack builds on the basic query shown in [Hack #50]. To get at more than the top 10 results, no matter the programming language you're using, you'll have to create a loop. The example is in Perl, because that's what most of the hacks in this book are written in. Alterations to support looping are shown in bold.

51.1 The Code

#!/usr/local/bin/perl
# looply.pl
# A typical Google Web API Perl script
# Usage: perl looply.pl <query>

# Your Google API developer's key
my $google_key='insert key here';

# Location of the GoogleSearch WSDL file
my $google_wdsl = "./GoogleSearch.wsdl";

# Number of times to loop, retrieving 10 results at a time
my $loops = 3; # 3 loops x 10 results per loop = top 30 results

use strict;

# Use the SOAP::Lite Perl module
use SOAP::Lite;

# Take the query from the command-line
my $query = shift @ARGV or die "Usage: perl looply.pl <query>\n";

# Create a new SOAP::Lite instance, feeding it GoogleSearch.wsdl

my $google_search = SOAP::Lite->service("file:$google_wdsl");

# Keep track of result number
my $number = 0;

for (my $offset = 0; $offset <= ($loops-1)*10; $offset += 10) {
 # Query Google
 my $results = $google_search -> 
  doGoogleSearch(
    $google_key, $query, $offset, 10, "false", "", "false",
    "", "latin1", "latin1"
  );
 
 # No sense continuing unless there are more results
 last unless @{$results->{resultElements}};
 
 # Loop through the results
 foreach my $result (@{$results->{'resultElements'}}) {
 
  # Print out the main bits of each result
  print
   join "\n",  
   ++$number,
   $result->{title} || "no title",
   $result->{URL},
   $result->{snippet} || 'no snippet',
   "\n";
 
 }
}

Notice that the script tells Google which set of 10 results it's after by passing an offset ($offset). The offset is increased by 10 each time ($offset += 10).

51.2 Running the Script

Run this script from the command line, passing it your preferred query:

$ perl looply.pl "query"

51.3 The Results

% perl looply.pl
Usage: perl looply.pl <query>
% perl looply.pl "learning perl"
1
oreilly.com -- Online Catalog: Learning Perl, 3rd Edition
http://www.oreilly.com/catalog/lperl3/
... Learning Perl, 3rd Edition Making Easy Things 
Easy and Hard Things Possible By Randal<br> L. Schwartz, Tom Phoenix 
3rd Edition July 2001 0-596-00132-0, Order Number ... 
 
...
29
Intro to Perl for CGI
http://hotwired.lycos.com/webmonkey/98/47/index2a.html
... Some people feel that the benefits of learning 
Perl scripting are few.<br> But ... part. That's right. 
Learning Perl is just like being a cop. ...  
30
WebDeveloper.com ®: Where Web Developers and Designers Learn How ... 
http://www.webdeveloper.com/reviews/book6.html
... Registration CreditCard Processing Compare Prices. 
Learning Perl. Learning<br> Perl, 2nd Edition. 
Publisher: O'Reilly Author: Randal Schwartz ...    

51.4 See Also

• [Hack #50]