Opening Electronic Doors   

As database editor at the News & Observer Þn Raleigh, North Carolina, I've initiated more than my share of public records requests that began as legal questions and ended up as technical ones. Here's how we overcame initial reluctance from several state agencies:

The records are going to cost you. In May 1992, the News & Observer asked for a copy of a database containing information about more than 300,000 prison sentences handed out over a 20-year period.

The state Department of Corrections' initial response was disbelief. We could have the database, but it would require a significant effort – at least 20 hours at $50 an hour – to separate data that were not public records (such as the inmates' psychological and medical histories).

Our response was an offer to write the program for them (I thought their time estimate was overcooked). The agency said it would reconsider; the data arrived a month later at roughly a third of the estimated cost.

That's a "proprietary" database. One technique frequently used to deny access is to claim that the software the agency uses is "proprietary" – a trade secret controlled by the software manufacturer. We got this response when the state attorney general's office denied our request for its collection of consumer complaints.

But we didn't want the software, just the data. Any database system that can input data has a way to output it, either on paper or as a computer file (in computerese, a "print image") that can be read by other database systems. So we overcame the trade secrets problem by asking for the print image – the file – of the data. Getting the file, on disk or magnetic tape, is one of the simplest and most powerful methods around the proprietary defense.

We won't help. In North Carolina, the law is not clear about who must pay for separating information for release if a database contains some information that is not public. When we asked for state pay records, the Office of State Personnel conceded that the information was public but said that we couldn't have electronic copies because no program existed to separate the public data. For $3,000, however, it could write one.

We then asked for a print image of a report that contained only the categories that were public information. The personnel office said none of its 300 reports fit the bill and declined to write a special one for us. A month later, however, we found a paper copy of the report that included all but one of the public categories. We then asked for a print image of that report. Remember, if a paper version of the records exists, you can obtain an electronic copy.

You don't know what you're asking for. When we started requesting electronic records, most departments we contacted had never received such a request and didn't know what to do. "You must be kidding" was a common response, followed frequently by, "You have no idea what you just asked for."

We encountered just such an attitude when we asked the state controller's office for a copy of the government checkbook. We knew that it would be a massive database, but we were armed for the task. We have 6.5 billion bytes of storage on our network, and triple that available on systems used by our production department.

One hostile administrator insisted the checkbook database would require nearly four times the storage we had. He was way off. A year's worth of the register for two of the larger of 58 departments took about 60 million bytes of data, which barely taxed our largest PC. All 58 departments required only about 3.5 billion bytes.

You can have it raw. At the Department of Motor Vehicles, the registration database is a masterpiece of supercomplex, early 1970s programming – and was inaccessible without modifications. We couldn't argue that the DMV could easily write a program to make the data usable. So we spent weeks writing a program to read it. The result has been a great reference database. We can now find who owns any car based on its plate number, as well as the address of people who don't show up in the phone book. We used the same approach to decode a database of court records. We've used both of these databases to write stories ranging from a feature on vanity plates to an investigation of the use of the court system as a check collection service for local merchants.

Let us read the manual for you. At the local police department, one officer had responsibility for the incident database as well as a number of other systems. When we asked for some records, he tried to help but didn't have the expertise to reproduce them on a computer tape. We visited his computer room, grabbed the manuals off his shelves and searched until we found the right set of commands. He learned something new, and we got our data. l