PHP CookBook

Recipe 11.14 Parsing a Web Server Log File

11.14.1 Problem

You want to do calculations based on the information in your web server's access log file.

11.14.2 Solution

Open the file and parse each line with a regular expression that matches the log file format. This regular expression matches the NCSA Combined Log Format:

$pattern = '/^([^ ]+) ([^ ]+) ([^ ]+) (\[[^\]]+\]) "(.*) (.*) (.*)" ([0-9\-]+)
    ([0-9\-]+) "(.*)" "(.*)"$/';

11.14.3 Discussion

This program parses the NCSA Combined Log Format lines and displays a list of pages sorted by the number of requests for each page:

$log_file = '/usr/local/apache/logs/access.log';
$pattern = '/^([^ ]+) ([^ ]+) ([^ ]+) (\[[^\]]+\]) "(.*) (.*) (.*)" ([0-9\-]+)
    ([0-9\-]+) "(.*)" "(.*)"$/';

$fh = fopen($log_file,'r') or die($php_errormsg);
$i = 1;
$requests = array();
while (! feof($fh)) {
    // read each line and trim off leading/trailing whitespace
    if ($s = trim(fgets($fh,16384))) {
        // match the line to the pattern
        if (preg_match($pattern,$s,$matches)) {
            /* put each part of the match in an appropriately-named
             * variable */
            list($whole_match,$remote_host,$logname,$user,$time,
                 $method,$request,$protocol,$status,$bytes,$referer,
                 $user_agent) = $matches;
             // keep track of the count of each request 
            $requests[$request]++;
        } else {
            // complain if the line didn't match the pattern 
            error_log("Can't parse line $i: $s");
        }
    }
    $i++;
}
fclose($fh) or die($php_errormsg);

// sort the array (in reverse) by number of requests 
arsort($requests);

// print formatted results
foreach ($requests as $request => $accesses) {
    printf("%6d   %s\n",$accesses,$request);
}

The pattern used in preg_match( ) matches Combined Log Format lines such as:

10.1.1.162 - david [20/Jul/2001:13:05:02 -0400] "GET /sklar.css HTTP/1.0" 200 
278 "-" "Mozilla/4.77 [en] (WinNT; U)"
10.1.1.248 - - [14/Mar/2002:13:31:37 -0500] "GET /php-cookbook/colors.html 
HTTP/1.1" 200 460 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)"

In the first line, 10.1.1.162 is the IP address that the request came from. Depending on the server configuration, this could be a hostname instead. When the $matches array is assigned to the list of separate variables, the hostname is stored in $remote_host. The next hyphen (-) means that the remote host didn't supply a username via identd,^[1] so $logname is set to -.

^[1] identd, defined in RFC 1413, is supposed to be a good way to identify users remotely. However, it's not very secure or reliable. A good explanation of why is at http://www.clock.org/~fair/opinion/identd.html.

The string david is a username provided by the browser using HTTP Basic Authentication and is put in $user. The date and time of the request, stored in $time, is in brackets. This date and time format isn't understood by strtotime( ), so if you wanted to do calculations based on request date and time, you have to do some further processing to extract each piece of the formatted time string. Next, in quotes, is the first line of the request. This is composed of the method (GET, POST, HEAD, etc.) which is stored in $method; the requested URI, which is stored in $request, and the protocol, which is stored in $protocol. For GET requests, the query string is part of the URI. For POST requests, the request body that contains the variables isn't logged.

After the request comes the request status, stored in $status. Status 200 means the request was successful. After the status is the size in bytes of the response, stored in $bytes. The last two elements of the line, each in quotes, are the referring page if any, stored in $referer^[2] and the user agent string identifying the browser that made the request, stored in $user_agent.

^[2] The correct way to spell this word is "referrer." However, since the original HTTP specification (RFC 1945) misspelled it as "referer," the three-R spelling is frequently used in context.

Once the log file line has been parsed into distinct variables, you can do the needed calculations. In this case, just keep a counter in the $requests array of how many times each URI is requested. After looping through all lines in the file, print out a sorted, formatted list of requests and counts.

Calculating statistics this way from web server access logs is easy, but it's not very flexible. The program needs to be modified for different kinds of reports, restricted date ranges, report formatting, and many other features. A better solution for comprehensive web site statistics is to use a program such as analog, available for free at http://www.analog.cx. It has many types of reports and configuration options that should satisfy just about every need you may have.

11.14.4 See Also

Documentation on preg_match( ) at http://www.php.net/preg-match; information about common log file formats is available at http://httpd.apache.org/docs/logs.html.