BY BILL MANNING
On January 31, 2001, the USFA released the newest changes to its NFIRS 5.0 spec. There were 12, for a grand total, according to the USFA, of 205. The other 100 or more alleged unpublished changes-the changes that, as one vendor put it, were “hidden between the spreadsheets”-were critical, to Herb Clark and others who went out of business over them, Clark, it seems, with a little push from the USFA.
The USFA denies making unpublished code changes. No vendors have offered tangible proof. But finding proof is a tall order, considering the USFA controls the codes and so controls their “history.” Red is red until it is changed to blue, at which time it has always been blue, never red. It’s all very convenient, and very Orwellian.
On the other hand, credible sources have raised the probability that the National Fire Data Center itself lacks control over the NFIRS project. Underfunded and underresourced, it is run by FEMA’s Information and Technology Services Directorate (FEMA IT), which has given considerable control of NFIRS day-to-day operations to its contracted service provider, Verizon. So it is possible and even likely that the USFA itself may not be aware of facets of its own operation, including unpublished changes to the design spec. But that doesn’t absolve the USFA’s complicity by omission or negligence, whether forced upon it or not.
NFIC defends the USFA against charges of phantom changes. Paul Cooke, Colorado state fire marshal and president of NFIC, says, “It’s an issue of software development, not the national spec. [I know of] one software vendor [that] had 24 iterations of their product. It’s not unusual. The technical people at NFIC [were asked to look into alleged changes not being reported] and they said that every change that has been made was documented and publicized.”
Some, like Andy Fritz of NFIC, contend NFIRS 5.0 “complexity” is what drives vendors away, or crazy. But fire software vendors disagree. “I put my least experienced programmer on the NFIRS 5.0 software development project,” said one vendor. “If he couldn’t do that, well…. It’s a very tedious job and there’s an art to it, but for a good programmer it’s not hard.”
Either a whole group of fire software programmers got stupid all of a sudden, or the USFA (or Verizon) is hiding something.
Many long-time fire software vendors, who once thought of themselves as a “partner” in the process, are a disenfranchised lot. The USFA last met with them as a group in September 1999, at the vendors’ insistence, and hasn’t met with them since. Joe Zeigler of FirePrograms says that at that time, Alex Furr, head of the National Fire Data Center, made three promises to the vendors: First, the USFA would not go into the software business. Second, the USFA would allow vendors to review any spec changes prior to release. And third, the USFA would make no spec changes other than at six-month intervals (this was one vendor’s sarcastic recommendation that the USFA took seriously) and the changes would be announced in advance.
“All three promises were broken,” says Zeigler.
Even the promise about not getting into “competition” with the vendors?
“NFIC has a ‘marketing’ committee,” Zeigler responds. “The USFA has a 1-888 help line. They’re distributing software. They’re a software company!”
(Furr, in defending the USFA’s plentiful software changes, says, “Any user of commercial software distributed by major national corporations has had similar experiences”-an odd comparison to make, indeed, for an outfit that’s “not in the software business.”)
“Somehow, the USFA got involved in developing software. It has taken center stage,” says Michael Fay of FirePoint software. “They started viewing vendors as competition.”
And that, he says, represents a conflict with what he perceives the USFA’s role to be. “I think the USFA’s focus should be to define the specifications and assist vendors in implementing them. Instead, they’re defining and implementing, simultaneously.” As for assistance, Fay notes, “After that meeting in September 1999, I have never received any follow-up, never received a phone call from the USFA. When we complain, they say they’ll look into it, but they never do.”
And there are signs the six-month rule may be violated again. A recent advisory from a member of the NFIC Systems Committee (which serves as a technology quality control group) suggests that edit changes to some of the data fields are forthcoming, as early as March. The USFA denies that any changes to the product affecting vendors will be released prior to July, the next scheduled changes cycle. That remains to be seen.
Some vendors have used the words “monopolistic” and “predatory” to describe the USFA’s NFIRS implementation strategy. Whether or not the USFA’s actions go that far, it is certainly true that the USFA’s National Fire Data Center-under the FEMA Information Technology umbrella-has operated with a bureaucratic arrogance and a sense of invincibility that only those with no accountability could enjoy.
The USFA has vested itself with the power, through an arbitrary and apparently changeable certification “standard,” to determine which companies can and cannot do business with the fire service. In a competitive market, no certification, or a lesser certification, is a death sentence.
In late 1998, the USFA, in a message to a state data manager, confirmed its NFIRS 5.0 policy that “Departments should NOT buy software from any vendor unless they have a guarantee from the vendor (in writing) that they have no cost obligation until the software receives final certification from USFA.” But for two years after the 5.0 spec was released to vendors (from which software could be written), not a single vendor received “final certification” from the USFA-all vendors either fell into “registered” or “conditionally certified” categories.
The USFA attempted to clarify for end users in March 2000 its position on certifications but offered loose, hazy criteria: “In order for vendors to reach full certification status, their data must be monitored by the states and by USFA for problems for a fairly extended period of time before they can be confidently certified as being in full compliance …. If the vendor then does not address the problem in a reasonable way and in a reasonable amount of time, USFA will withdraw their conditional certification status.”
In mid-February 2001, shortly after Part 1 of this investigation was published, the USFA issued its new criteria for final certification: “Vendors with Certified Conditional status will receive Fully Certified status after they have a minimum of 5,000 incidents in the National Database and they successfully have corrected problems identified during the grace period in a timely fashion.” It then fully certified at least four third-party vendors, in rapid succession.
But even that was bungled, as Furr leaked to the press that two vendors had been singled out for full certification at least two days before the new criteria was made public to all vendors, widening the now growing rift between the feds and its vendor “partners.” More important, the USFA’s capriciousness and certification contingencies make “full certification” status a tenuous proposition for any vendor on the wrong side of the USFA.
Of course, the USFA has no legal or moral authority to certify vendors or otherwise manipulate the free market system. When confronted with this issue, Furr said the USFA was just honoring NFIC’s request. But then, what gives NFIC the authority? The USFA, which pays NFIC’s freight?
This presumption of certification power is sheer bureaucratic chutzpah. Some will say certification is necessary to protect fire departments from unscrupulous or incompetent vendors. Others will say it is necessary to ensure uniformity of data into the system. Others will say it is necessary for security. All miss the mark.
The USFA overengineered a system far beyond its technological capabilities to manage and implement. It created a cumbersome, ineffective, overinflated, monolithic system, attempting to pass off 1970s data management theory as state-of-the-art. It has spent millions of dollars in its preoccupation with front end tools and rules (data entry software, validation software, conversion software, upload software, single-source uploads, certification procedures) instead of focusing on what should be its only critical task: backend data collection and storage.
Asked if he thought the USFA’s NFIRS 5.0 system were overengineered, a programmer from outside the USFA, at one time closely connected with the NFIRS 5.0 project, said “I wouldn’t go that far but you’re on the right track. What the USFA ought to be doing [in addition to creating a set of specifications] is just providing backend validation and storage”-that is, creating a data base and the means to accept only what data it wants to accept.
After developing the design specifications-a mammoth job for which most give the USFA and NFIC a lot of credit-the USFA was more than halfway home. Instead, it set about to prove you can’t deliver front-end solutions through a bureaucracy. So the entire system is out of its orbit.
Vendor certifications are just a symptom of the USFA’s disease. They’re in the way of the USFA’s design agenda. They vendors are casualties, not partners, just as the fire service is a casualty, NFIC’s representation of it notwithstanding.
Vendors, fire departments, and state fire agencies are at the mercy of the USFA system design because, first, it has ignored the possibility that modern data management practices, as used in corporate industry, can be applied to a government system; second, it has adopted the position that the fires on American Main Streets and other fire department response data are a matter of national security; third, it has opted for a federalized system rather than encouraging a much less costly integrated state data network; fourth, it supports a philosophy that all fire departments have a right to software free or below the market value; fifth, it has insisted that the federal government should be creating software, under the nauseating name “Federal Client Tool” (“The USFA did not develop software with third party vendors in mind,” says Cooke. “It was NFIC that recognized we needed third-party vendors”); and sixth, it operates under the virtually unchallenged assumption that big government can do it better.
Even the USFA’s role as national fire data repository, held by most as the natural order of things, is suspect. Recently the USFA server crashed for a couple of days. An unspecified quantity of fire incident data was lost. Because state data systems, in the USFA scheme, are not necessarily built-in redundancy-that is to say, some states do not store their fire department data at that level, but, rather, simply comply with the notion “let the feds do it for you”-the USFA’s only answer is to increase the hardware components on the central system.
Michigan is one such state that uploads directly to the feds without storing its own data. Mary Nemeth, who manages the state’s fire data, reports that “because of all the coding problems because of all the changes,” only 62 percent of Michigan’s 1999 data was validated by the USFA. That means more than 100,000 Michigan incidents will not show up in the federal data for 1999.
In late 2000, the USFA circulated a memo announcing that it had shut down its main frame computers permanently and no longer had access to the past 20 years worth of fire data. When I asked if the data were lost, Furr laughed. “Anyone” could order it, she said, “on disk or tape” through the National Technology Information Center. I called the Center and requested the past 20 years of NFIRS data. The representative informed me that the records were available only on computer tape used on main frame computers. Each tape costs $454, so for only about $9,000 and a main frame computer I could purchase and use the whole set. These are the folks who are storing your data?
At last count, about eight states have implemented NFIRS 5.0 and about 18 states are in some stage of implementation. Nothing is perfect, and all have had problems to some degree. But the NFIRS 5.0 Implementation Nightmare Award probably should go to Massachusetts, which for two years has been trying to get up and running, without success. It is the only state thus far working off the Oracle database, a high-end, high-storage setup. After initial delays, state officials expected to receive the federal software in July 2000 but received it in December for a slated January 2001 implementation, which has yet to occur.
State Fire Marshal Mark Coan says, “It’s been a harrowing experience. For whatever reason, the USFA boxed in the states in the transition to Version 5.0. They made certain deadlines, so we made commitments to our fire chiefs as to when we’d transition. Information was due to Massachusetts that would let us act, train local fire departments and so forth, but that information just hasn’t been there. We’ve set up training schedules and we’re out there trying to do it, but we’re at a loss to explain the components of the federal system ellipse.
“It’s disturbing. There’s been a lack of follow-through by the USFA. I was left holding the bag with the fire departments in this state.
“If they [the USFA] were in this much trouble, they should not have made the commitment to the states.”
“The delay puts us in a terrible position,” says Jennifer Meith, manager of fire data and public education unit for Massachusetts. “[The federal system] is not ready for prime time.
“There were a lot of changes, but there were no notices to state managers about changes to the system. None of our people ever came back from a NFIC meeting saying ‘it’s not ready.’ We just weren’t told.
“Bell Atlantic [Verizon] said, ‘We’ll help you get up and running. We have error checking. We have a data analysis package. We have a converter package….’ When we started working with them two years ago, they told us we’d be up by January 2000….
“There’s no error checking. I get a huge file from a fire department and it might have mistakes in it, so you have to check for errors. But the utility tool the USFA provided doesn’t work for error checking-reports come out that aren’t readable, they’re not intelligible even to an experienced data manager-so we can’t figure out what’s wrong and we sure can’t send the data back to the fire chief.
“I just spent tens of thousands of dollars on printing manuals for local fire departments to train them on how to use the system. Are they going to change it on me again? I don’t want to bash the USFA, but….
“How does the Federal Client Tool work with Oracle uploads? That’s the great unknown.”
The USFA itself recognizes some implementation difficulties, but it sees this as normal growing pains. It points to the fact that many thousands of uploads of fire department data have been successful. That is true. But how “seamless” is the process, and what will happen as more and more states upload more and more data, as many states have not even begun their 2000 and 2001 uploads?
The process is not as seamless as the USFA thinks it is. Fire Engineering has learned that a third-party entity, unknown to the USFA and at a cost of thousands of dollars of its own (private) money, has been receiving and “cleaning up” data for thousands of incident reports from hundreds of fire departments such that they can be uploaded to the USFA. The entity requested anonymity because, technically, that’s a violation of USFA “security” policy and it fears retribution from the USFA.
Why is the entity taking such a costly risk? “This is supposed to be about fire departments. About their data. What good is it if the fire department’s data can’t be uploaded because of the USFA’s mess?”
NFIC, which walks a fine line between responsiveness to the feds who fund it and responsiveness to the fire service, itself recognizes implementation difficulties and problems with the system itself. “There’s no doubt the implementation process has not been smooth,” says Cooke. “The [USFA’s] focus on the Federal Client Tool did detract from implementation efforts.
“For me [as Colorado state fire marshal], the FCT is not a good way to go. Software vendors will provide the best options available for fire departments. The FCT does what it is supposed to do, but there are things the system should be doing that currently are not being done.
“There’s no full range of reporting capabilities-it was promised but not yet delivered. We don’t have the ability to do third-party connectivity-I was told it is a security issue but that they’re working on it. And the system is slow-they’re working on that, too.
“One of the problems is the contract the USFA has to develop [NFIRS 5.0]. The Bell Atlantic contract is not an effective use of resources. The FEMA umbrella is not optimum; it’s managed by FEMA IT, not USFA, so the implementation is catch-as-catch-can. It’s a ‘time and materials’-based contract. What compels the contractor to perform? The USFA needs to have its own separate contract with an applications service provider under set performance criteria.
“I think a lot of issues can be addressed if they went outside of FEMA resources to do it. Performance could be improved if the system were housed on something other than FEMA hardware.”
But even if performance issues were magically resolved, the fire service is left with the number one question. What is the USFA going to do with this data once it gets it? What will be its value?
If the past 20 years are any indication, the USFA has not shown any wherewithal in this regard. In fact, contrary to popular belief, the annual National Fire Protection Association fire department incident survey serves as the foundation for the USFA’s national statistics. And because the USFA has at least a two-year data turnaround time, its data will not have an impact at the local level or, more importantly, for states that have put all their eggs in the federal basket. It is not set up to deliver time-sensitive data at these levels.
This gives the lie to NFIC/USFA’s “chicken in every pot” software philosophy. Undoubtedly, small departments count and states should take steps to capture their incident data. But while the USFA was busy developing the “everyman” software-yes, acting, in effect, as a software company-precious time was taken from the smooth, seamless collection of data, and ensuring the participation of, the 20 percent of fire departments that do 80 percent of incident runs in this country.
There is, of course, great value in national data statistics, if the data were properly manipulated for desired outcomes. In fact, receiving additional federal appropriations through Congress in 2002 and beyond rely directly on our ability to frame national incident statistics in a usable way. But fire data analysts were not consulted in the NFIRS 5.0 process to an appreciable degree. “We’re not using NFIRS to fullest advantage,” said one data analyst, requesting anonymity. “[The USFA] gave away the store to the people filling out the forms and ignored the people who have analyzed data for years.” Another data analyst said, “I had minor involvement [in the NFIRS 5.0 process]. The people leading the meetings weren’t pleased with my observations. The data codes were all over the place. It was not well organized. And tracking down data equivalencies between 4.1 and 5.0 is going to be very difficult. But they chose to ignore me.”
It’s clear the people in control are thinking “big” but not “big picture.” Their “big thinking” has put up some very high walls, and it seems the fire service’s best interests fall outside that narrow channel.
To put it bluntly, the fire service deserves everything it gets. In many states, fire data collection resources are scant. For many state officials, as well as many fire departments, fire data is not a priority-it’s more a bother. The fire service in general has been a data collection dinosaur for so long, relying on paper or other antique systems, that anything is seen as an improvement. The fact that the federal government is holding the states’ hands is a relief for many. So what if the FCT is a waste of your tax dollars? In an informal survey, few state representatives would go on record saying that there were any problems with the federal NFIRS system. The emperor wears no clothes, and the fire service, frankly, deserves to pay to watch the show.
Can we stop this speeding train?
“I think we should work collectively,” says Cooke. “I think the USFA should seek an independent contract [that is, independent of FEMA IT] managed by the National Fire Data Center with an advisory committee providing advice and guidance to take a look at deliverables. I think we’d go a long way if we were to go to such a system.”
But, outside of the FEMA IT issue, isn’t that what we were doing (or supposed to be doing) all along? Isn’t that part of NFIC’s mission? Who is looking at “deliverables”? Verizon? What the hell is happening here?
Throw away half the problems raised in Part 1 and Part 2 of this editorial, and you’ve got good cause for great concern.
It didn’t have to be this way.
There’s no mass conspiracy, but it seems clear that the USFA is throwing up a wall of deception to mask its motives and its incompetence. It has wasted money and time and energy in goals it clearly can’t achieve and are beyond the scope of what the fire service needs it to do. For about one-fifth the price of what we think it has paid Bell Atlantic/Verizon (no one is really sure of the price because FEMA has successfully avoided Freedom of Information Act requests-why, if there’s nothing to hide?), we could have set up each state with a first-rate data collection system that would have provided the USFA a built-in redundancy factor and given all the states the means to customize their data.
There’s no reason to think the USFA couldn’t have served you well if it had remained focused on what was achievable-creating the spec on the information it wanted to capture, developing a storage data base, creating the limitations on what data it will accept, and collecting the data. But it didn’t. The USFA created a monster.
The monster sees your data as “proprietary”-as its own. It roars that it is “protecting” you, when it seems you may need protection from it. Heaven knows what will happen as more and more fire organizations upload data into the monster’s mouth. Will you ever get it back again? Will it do us any good?
We have a USFA that operates with only about $40 million a year, about half of which goes to the National Fire Academy. It answers to FEMA, which is more interested in hurricane evacuation plans and dollar losses to homes destroyed by natural disasters than it is interested in more than 4,000 civilian deaths, 100,000 firefighter injuries, and 100 line-of-duty firefighter deaths to fire per year. We know that partly because the monster built to control the data that could help minimize our fire/incident losses otherwise would not have been let out of its cage.
Firefighters, this is your data, not the federal government’s. You need results, not good intentions. And it would be a damn shame if you let the NFIRS 5.0 debacle continue for one byte longer.
