PHP CookBook

Recipe 11.15 Program: Finding Stale Links

The stale-links.php program in Example 11-5 produces a list of links in a page and their status. It tells you if the links are okay, if they've been moved somewhere else, or if they're bad. Run the program by passing it a URL to scan for links:

% stale-links.php http://www.oreilly.com/
http://www.oreilly.com/index.html: OK
http://www.oreillynet.com: OK
http://conferences.oreilly.com: OK
http://international.oreilly.com: OK
http://safari.oreilly.com: MOVED: mainhom.asp?home
...

The stale-links.php program uses the cURL extension to retrieve web pages. First, it retrieves the URL specified on the command line. Once a page has been retrieved, the program uses the pc_link_extractor( ) function from Recipe 11.9 to get a list of links in the page. Then, after prepending a base URL to each link if necessary, the link is retrieved. Because we need just the headers of these responses, we use the HEAD method instead of GET by setting the CURLOPT_NOBODY option. Setting CURLOPT_HEADER tells curl_exec( ) to include the response headers in the string it returns. Based on the response code, the status of the link is printed, along with its new location if it's been moved.

Example 11-5. stale-links.php

function_exists('curl_exec') or die('CURL extension required');

function pc_link_extractor($s) {
    $a = array();
    if (preg_match_all('/<A\s+.*?HREF=[\"\']?([^\"\' >]*)[\"\']?[^>]*>(.*?)<\/A>/i',
                       $s,$matches,PREG_SET_ORDER)) {
        foreach($matches as $match) {
            array_push($a,array($match[1],$match[2]));
        }
    }
    return $a;
}

$url = $_SERVER['argv'][1];

// retrieve URL
$c = curl_init($url);
curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($c, CURLOPT_FOLLOWLOCATION,1);
$page = curl_exec($c);
$info = curl_getinfo($c);
curl_close($c);

// compute base url from url
// this doesn't pay attention to a <base> tag in the page
$url_parts = parse_url($info['url']);
if ('' == $url_parts['path']) { $url_parts['path'] = '/'; }
$base_path = preg_replace('<^(.*/)([^/]*)$>','\\1',$url_parts['path']);
$base_url = sprintf('%s://%s%s%s',
                    $url_parts['scheme'],
                    ($url_parts['username'] || $url_parts['password']) ?
                    "$url_parts[username]:$url_parts[password]@" : '',
                    $url_parts['host'],
                    $url_parts['path']);

// keep track of the links we visit so we don't visit each more than once
$seen_links = array();

if ($page) {
    $links = pc_link_extractor($page);
    foreach ($links as $link) {
        // resolve relative links
        if (! (preg_match('{^(http|https|mailto):}',$link[0]))) {
            $link[0] = $base_url.$link[0];
        }
        // skip this link if we've seen it already
        if ($seen_links[$link[0]]) {
            continue;
        } 
        
        // mark this link as seen
        $seen_links[$link[0]] = true;

        // print the link we're visiting
        print $link[0].': ';
        flush();
        
        // visit the link
        $c = curl_init($link[0]);
        curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($c, CURLOPT_NOBODY, 1);
        curl_setopt($c, CURLOPT_HEADER, 1);
        $link_headers = curl_exec($c);
        $curl_info = curl_getinfo($c);
        curl_close($c);

        switch (intval($curl_info['http_code']/100)) {
        case 2:
            // 2xx response codes mean the page is OK
            $status = 'OK';
            break;
        case 3:
            // 3xx response codes mean redirection
            $status = 'MOVED';
            if (preg_match('/^Location: (.*)$/m',$link_headers,$matches)) {
                $location = trim($matches[1]);
                $status .= ": $location";
            }
            break;
        default:
            // other response codes mean errors
            $status = "ERROR: $curl_info[http_code]";
            break;
        }

        print "$status\n";
    }
}