Google Hacks

Hack 57 Programming the Google Web API with Python

Programming the Google Web API with Python is simple and clean, as these scripts and interactive examples demonstrate.

Programming to the Google Web API from Python is a piece of cake, thanks to Mark Pilgrim's PyGoogle wrapper module (http://diveintomark.org/projects/pygoogle/). PyGoogle abstracts away much of the underlying SOAP, XML, and request/response layers, leaving you free to spend your time with the data itself.

57.1 PyGoogle Installation

Download a copy of PyGoogle and follow the installation instructions (http://diveintomark.org/projects/pygoogle/readme.txt). Assuming all goes to plan, this should be nothing more complex than:

% python setup.py install

Alternately, if you want to give this a whirl without installing PyGoogle or don't have permissions to install it globally on your system, simply put the included SOAP.py and google.py files into the same directory as the googly.py script itself.

57.2 The Code

#!/usr/bin/python
# googly.py
# A typical Google Web API Python script using Mark Pilgrim's
# PyGoogle Google Web API wrapper 
# [http://diveintomark.org/projects/pygoogle/]
# Usage: python googly.py <query>

import sys, string, codecs

# Use the PyGoogle module
import google

# Grab the query from the command-line
if sys.argv[1:]:
  query = sys.argv[1]
else:
  sys.exit('Usage: python googly.py <query>')

# Your Google API developer's key
google.LICENSE_KEY = 'insert key here'

# Query Google
data = google.doGoogleSearch(query)

# Teach standard output to deal with utf-8 encoding in the results
sys.stdout = codecs.lookup('utf-8')[-1](sys.stdout)

# Output
for result in data.results:
  print string.join( (result.title, result.URL, result.snippet), "\n"), "\n"

57.3 Running the Hack

Invoke the script on the command line as follows:

% python googly.py "query words"

57.4 The Results

% python googly.py "learning python"
oreilly.com -- Online Catalog: <b>Learning</b> 
<b>Python</b>
http://www.oreilly.com/catalog/lpython/
<b>Learning</b> <b>Python</b> is an 
introduction to the increasingly popular interpreted programming
language that's portable, powerful, and remarkably easy to use in both 
<b>...</b>   
...
Book Review: <b>Learning</b> <b>Python</b>
http://www2.linuxjournal.com/lj-issues/issue66/3541.html
<b>...</b> Issue 66: Book Review: <b>Learning</b> 
<b>Python</b> <b>...</b> Enter 
<b>Learning</b> <b>Python</b>. My executive summary 
is that this is the right book for me and probably for many others 
as well. <b>...</b>   

57.5 Hacking the Hack

Python has a marvelous interface for working interactively with the interpreter. It's a good place to experiment with modules such as PyGoogle, querying the Google API on the fly and digging through the data structures it returns.

Here's a sample interactive PyGoogle session demonstrating use of the doGoogleSearch, doGetCachedPage, and doSpellingSuggestion functions.

% python
Python 2.2 (#1, 07/14/02, 23:25:09) 
[GCC Apple cpp-precomp 6.14] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import google
>>> google.LICENSE_KEY = 'insert key here'
>>> data = google.doGoogleSearch("Learning Python")
>>> dir(data.meta)
['_  _doc_  _', '_  _init_  _', '_  _module_  _', 'directoryCategories', 
'documentFiltering', 'endIndex', 'estimateIsExact', 
'estimatedTotalResultsCount', 'searchComments', 'searchQuery', 
'searchTime', 'searchTips', 'startIndex']
>>> data.meta.estimatedTotalResultsCount
115000
>>> data.meta.directoryCategories
[{u'specialEncoding': '', u'fullViewableName': "Top/Business/Industries/
Publishing/Publishers/Nonfiction/Business/O'Reilly_and_Associates/
Technical_Books/Python"}]
>>> dir(data.results[5])
['URL', '_  _doc_  _', '_  _init_  _', '_  _module_  _', 'cachedSize', 
'directoryCategory', 'directoryTitle', 'hostName', 
'relatedInformationPresent', 'snippet', 'summary', 'title']
>>> data.results[0].title
'oreilly.com -- Online Catalog: <b>Learning</b> <b>Python'
>>> data.results[0].URL
'http://www.oreilly.com/catalog/lpython/'
>>> google.doGetCachedPage(data.results[0].URL)
'<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">\n
<BASE HREF="http://www.oreilly.com/catalog/lpython/"><table border=1
...
>>> google.doSpellingSuggestion('lurn piethon')
'learn python'