Processing Xml With Java - A Guide To Sax, Dom, Jdom, Jaxp, And Trax

Input

By far the hardest part of this or any similar problem is parsing the non-XML input data. Everything else pales by comparison. Unlike parsing XML, you generally cannot rely on a library to do the hard work for you. You have to do it yourself. And also unlike XML, there's little guarantee that the data is well-formed. More likely than not, you will encounter incorrectly formatted data.

In this case, because the records are separated into lines, I'll read each line, one at a time, using the readLine() method of java.io.BufferedReader. This method works well enough as long as the data is in a file, although it's potentially buggy when the data is served over a network socket.

Each line is dissected into its component fields inside the splitLine() method. Each record is stored in its own map. The keys for the map are read from a constant array, because the fields are always in the same position in each record.

Example 4.1 shows the input code. To use this, open an input stream to the file containing the budget data and pass that stream as an argument to the parse() method. You'll get back a List containing the parsed data. Each object in this list is a Map containing the data for one line item. Both keys and values for this map are strings. Because the keys are constant, they're stored in a final static array named keys. At various times I plan to use the keys as XML element names, XML attribute names, or SQL field names. Therefore, it's necessary to begin each of them with letters. Thus the keys for the fiscal year fields will be named FY1976, FY1977, FY1978, and so forth instead of just 1976, 1977, 1978, and so forth. This means we won't trivially be able to store the keys as ints. However, this turns out not to have been the case anyway because one of the year fields turns out to be the transitional quarter in 1976—which does not represent a full year and does not have a numeric name.

The code to parse each line of input is hidden inside the private splitLine() method. This code is relatively complex. It iterates through the record looking for comma delimiter characters but has to ignore commas that appear inside quoted strings. Furthermore, it must recognize that the end of the string delimits the last token. Even so, this method is not very robust. It will throw an uncaught exception if any quotes are omitted, or if there are too few fields. It will not notice and report the error if a record contains too many fields.

Example 4.1 A Class That Parses Comma-Separated Values into a List of HashMaps

import java.io.*;
import java.util.*;


public class BudgetData {

  public static List parse(InputStream src) throws IOException {

    // The document as published by the OMB is encoded in Latin-1
    InputStreamReader isr = new InputStreamReader(src, "8859_1");
    BufferedReader in = new BufferedReader(isr);
    List records = new ArrayList();
    String lineItem;
    while ((lineItem = in.readLine()) != null) {
      records.add(splitLine(lineItem));
    }
    return records;

  }

  // the field names in order
  public final static String[] keys = {
    "AgencyCode",
    "AgencyName",
    "BureauCode",
    "BureauName",
    "AccountCode",
    "AccountName",
    "TreasuryAgencyCode",
    "SubfunctionCode",
    "SubfunctionTitle",
    "BEACategory",
    "On-Off-BudgetIndicator",
    "FY1976", "TransitionQuarter", "FY1977", "FY1978", "FY1979",
    "FY1980", "FY1981", "FY1982", "FY1983", "FY1984", "FY1985",
    "FY1986", "FY1987", "FY1988", "FY1989", "FY1990", "FY1991",
    "FY1992", "FY1993", "FY1994", "FY1995", "FY1996", "FY1997",
    "FY1998", "FY1999", "FY2000", "FY2001", "FY2002", "FY2003",
    "FY2004", "FY2005", "FY2006"
   };

  private static Map splitLine(String record) {

    record = record.trim();

    int index = 0;
    Map result = new HashMap();
    for (int i = 0; i < keys.length; i++) {
      //find the next comma
      StringBuffer sb = new StringBuffer();
      char c;
      boolean inString = false;
      while (true) {
        c = record.charAt(index);
        if (!inString && c == '"') inString = true;
        else if (inString && c == '"') inString = false;
        else if (!inString && c == ',') break;
        else sb.append(c);
        index++;
        if (index == record.length()) break;
      }
      String s = sb.toString().trim();
      result.put(keys[i], s);
      index++;
    }

    return result;

  }

}