The FBI's VCF (Virtual Case Files) IT project is one of the most highly publicized software failure in U.S history. The main goal of the FBI’s VCF project was to automate the FBI's paper driven work environment, allow agents and intelligence analysts to share vital investigative information, and replace the obsolete Automated Case Support
(ACS) system. The FBI outsourced the code writing for the VCF project in the year 2001 to contractor Science Applications International Corporation (SAIC) based out of San Diego,CA. SAIC delivered 700,000 lines of code that was overly bug ridden, and functionally off target from what the FBI expected, that the FBI had to terminate the $170 million project which included $105 million worth of unusable code. Various government audits and independent reports show that the FBI did not have the proper IT management and technical expertise, and should share the blame for the VCF project failure. At the termination of the project in 2005, Glenn A. Fine, the U.S. Department of
Justice's inspector general, described the main factors that contributed to the VCF's failure.
His list included the following factors: poorly defined and slowly evolving design requirements; overly ambitious schedules; and the lack of a plan to guide hardware purchases, network deployments, and software development for the bureau. The FBI announced that it would buy off-the-shelf software to be installed in phases over the next four years from 2005-2009. The FBI will have to rely on the same combination of paper records and outdated software that the failed VCF project was supposed to replace. How could this happen to the FBI?

To understand the VCF project you have to first understand the FBI organization and the FBI’s agents. The FBI is an intricate network of commands and agents. The FBI is headquartered in the J. Edgar Hoover Building in Washington, D. C., and currently has 23 divisions. Some of these divisions include counterintelligence, criminal investigation, and cybercrime. The divisions then fall under the control of five executive assistant directors responsible for intelligence, counterterrorism and counterintelligence, criminal investigations, law enforcement services, and administration. Each division had its own
IT budget and system. Each division had the freedom and money to develop thier own software. The FBI has 40 to 50 different investigative databases and applications, and these systems have duplicating functions and information that is found in other systems in the FBI.

The FBI has 12,400 agents who work out of 56 field offices and 400 satellite offices, and 51 legal offices across the globe in U.S. embassies and consulates. A field agent works as part of a squad and each squad has a supervisor, who reports to the assistant special agent in charge, who in turn reports to the special agent in charge of the field office.FBI agents investigate everything from counterterrorism leads to bankruptcy fraud, cyber crimes, corrupt public officials, to kidnappings. The agents interview witnesses, develop informants, conduct surveillance, hunt for clues, and work with local law enforcement to find and aprehend criminals. The FBI agents document every step and build case files. They spend a large amount of time processing paperwork, faxing and mailing standardized memos and requisition forms through the approval chain to their squad supervisor or special agent in charge. This system of forms and approvals goes back to the 1920’s when J. Edgar Hoover, director from 1924 to 1972, standardized all of the bureau's investigative reports on forms. He put all reports on forms so an agent could walk into any FBI office and find the same system.
Today, the bureau has hundreds of standard forms. To record contact with an informant the agent would fill out Form FD-209 or to report information gained from an interview that may later become testimony, the agent would use Form FD-302, and so on.

These forms related to investigations go up and down the approval chain. Once the appropriate supervisors sign off on the form, the form goes back to the agent who gives it to a processing clerk to enter into the ACS system. Next, the paper form is filed as part of the official record of the particular case. The FBI’s policy is for all official records to be entered into the
ACS system. Introducing an electronic record-keeping system did raise legal policy questions from the agents. The agents were concerned on how records could be kept discoverable and undiscoverable. They were concerned that in an electronic world information that was inputed could not be destroyed and would always be somewhere.\

FBI and it’s agents were reluctant to embrace the digital age. In 2000 the FBI finally began to analyze it’s outdated IT systems. During that time, the FBI was under the direction of Louis J. Freeh and the FBI did not have a CIO or documentation detailing its IT systems. These tasks of creating a plan fell to former IBM executive
Bob E. Dies. Dies became assistant director in charge of the FBI Information Resources
Division on 17 July 2000. He was the first of five officials who would struggle to convert the FBI's intricate and outdated information systems , and get the
VCF project under way.
In a 2002 report from the Department of Justice Inpector General, when Dies arrived and was appointed to lead the IT organization, 13,000 of the computers could not run modern software. The majority of the 400 resident agency offices were connected to the FBI intranet with links about the speed of a 56- kilobit-per-second modem. Many of the bureau's network components were no longer manufactured or supported. And agents couldn't e-mail U.S. Attorney offices, federal agencies, local law enforcement, or each other.Instead the agents typically faxed case-related information. In September 2000, Congress approved $379.8 million over three years for what was then called the FBI Information Technology Upgrade Project. Eventually the upgrade project was divided into three parts. The program became known as Trilogy. The Information Presentation Component would provide all 56 FBI field offices, 22,000 agents and support staff with new Dell
Pentium PCs running Microsoft Office, new scanners, printers, and servers. The
Transportation Network Component would provide secure local area and wide area networks, allowing agents to share information with their supervisors and each other.
But the User Applications Component, which would become the VCF project, had the most unrealistic goals. First of all, the VCF project was to take the five most used investigative
Applications: the Automated Case Support system, IntelPlus, the Criminal Law
Enforcement Application, the Integrated Intelligence Information Application, and the
Telephone Application , and make them all accessible via a point and click web interface. Next, it would rebuild the FBI's intranet. Finally, it was supposed to identify a way to replace the FBI's
40 different investigative software applications. ACS was based on the 1970’s era database called Adabas and written in a programming language called
Natural. The ACS system was rolled out in 1995, and was already outdated when it was deployed. Originally the agents and clerks accessed the program through the vintage IBM green-screen terminals connected to a mainframe over dedicated lines. Eventually, the terminals were converted on to standard desktop PCs. Agents could do basic Boolean and keyword searches for things like an informant's name or the dates of a wiretap information related to cases they were working. But according to the appointed project manager Special Agent Larry Depew, an agent would have to be very dedicated and have the skills to learn the new yet outdated system. He also stated that most agents did not understand why the bureau would even use the ACS system other than an index.

In May and June 2001, the FBI awarded Trilogy contracts to two major U.S. government contractors: DynCorp, of Reston, Va., for the hardware and network projects, and SAIC for software. All three Trilogy components were to be delivered by the middle of 2004. The FBI used cost plus award fee contracts. These would pay the cost of all labor and materials plus additional money if the contractor managed costs honestly. If the scope of the project expanded or if the contractor incurred other unforeseen costs, the FBI would have to pay for those concessions.

On September 4th 2001 Robert S. Mueller III became the tenth director in FBI history.
One week afterwards, terrorists attacked New York City's World Trade Center and a piece of the Pentagon. The inability of FBI agents to share the most basic information about Al
Qaeda's U.S. activities turned into a front page news scandal. Within days, the FBI's outdated technology infrastructure went on to be scrutinized daily by politicians and newspaper columnists. The 9/11 Commission Report would conclude in 2004, "the FBI's information systems were woefully inadequate. The FBI lacked the ability to know what it knew; there was no effective mechanism for capturing or sharing its institutional knowledge."
Faced with pressure, Mueller put the Trilogy project into high gear. In October 2001, Mueller pulled Robert J. Chiaradio up from his position as special agent in charge of the field office in Tampa, Fla., to Hoover Building headquarters in Washington to advise him on the all new and important software components of Trilogy. Chiaradio was an accountant by training, and would become the FBI's executive assistant director for administration in December 2001. Chiaradio determined that the FBI's basic plan for the software portion of Trilogy, which included applying a Web interface onto the ACS system and the four other programs wasn't going to make agents more effective. So Chiaradio brought in Special Agent Larry Depew in on the project to help him figure out what would work best. Depew embraced technology. Depew was known for codding his own case management database using FoxPro in the early 1990’s, and made copies on floppy disk and distributed the program to any agents that requested it. Depew was joining a team of seven that assessed the Web interface SAIC was designing for the ACS system. When the project was completed, the interface would let agents point and click through the process of filling out official forms. Chiaradio and Depew recognized the limitations of the interface and ACS and decided to meet with Dies. They convinced Dies and the director , that the bureau needed an entirely new database, graphical user interface, and applications, which would let agents search across various investigations to find relationships to their own cases. The new case management system would host millions of records containing information on everything from witnesses, suspects, and informants to evidence such as documents, photos, and audio recordings.
These records would be accessible to both the FBI's agents and its intelligence analysts. Chiaradio named the new system the Virtual Case File.
Dies wanted to provide agents with this software as soon as possible. Chiaradio and Depew did not have a formal plan and decided the system would develop and grow along the way. Today, many organizations rely on a blueprint known as enterprise architecture This blueprint guides hardware and software investment decisions. It also describes an organization's mission and operations and how it organizes and uses technology to accomplish its tasks. Also the blueprint describes how the IT system is structured and designed to achieve those objective for the future. The FBI didn't have such a blueprint. Without this blueprint the FBI could not make coherent and logical operational or technical decisions about linking databases, creating policies ,sharing data and confirming measures for user access and security. The project team had no detailed description of the FBI's processes and IT infrastructure as a guideline.
Depew's group defined how agents worked , gathered information, and how that information was fed into ACS system. Depew worked with engineers from SAIC, and together they drew up diagrams and flowcharts of how the case management system operated then and how they wanted the new case management system VCF to operate in the future. Mueller himself attended one of these meetings to tell the agents to design a system that would work best for them.Depew's team also called in people from across the FBI as subject matter experts to explain how their divisions or units functioned internally and with the rest of the bureau.
In December 2001 the FBI asked SAIC develop a new application, database, and graphical user interface to completely replace the FBI’s ACS. To define what users needed the VCF to do for them, SAIC held a series of Joint Application Development (JAD) sessions. In these sessions Depew's team of agents and experts got together with a group of SAIC engineers to work out what functions the VCF would perform. Ideas captured in these sessions formed the basis of the requirements document that guided SAIC's application designers and programmers.
In January 2002, the FBI requested an additional $70 million to Trilogy;
Congress went further, approving $78 million. DynCorp committed to delivering its two components by July 2002. SAIC agreed to deliver the initial version of the VCF in
December 2003 instead of June 2004.
SAIC and the FBI were now committed to creating an entirely new case management system in 22 months, which would replace ACS . The conversion was known in the IT business as a flash cutover. Employees and agents would log off from ACS on a
Friday afternoon and log on to the new system on Monday morning. Once the cutover happened, there was no going back, even if it turned out that the VCF didn't work. The FBI did not have a backup plan. The projects contracts were changed to reflect the aggressive new deadline. Neither the original software contract nor the modified contract specified any formal criteria for the FBI to use to accept or reject the finished VCF software. No formal schedules or milestones were outlined. SAIC broke its VCF development group into eight teams.
They worked in parallel on different functional pieces of the program in order to finish the job faster. But the eight threads would prove too difficult for SAIC to combine into a single system. Meanwhile, Depew's team was describing the FBI's investigative and administrative processes on how agents built case files, how case files were used, and what additional functions they wanted the Virtual Case File to perform. Depew and his team prepared to communicate the processes that define the FBI to the SAIC engineers, but Mueller, Dies, and Chiaradio recruited a veteran IT program manager
C.Z. Higgins, a 29-year veteran of AT&T and Lucent. As project management executive for the Office of the Director, Higgins was brought in to create the Office of Program Management. Higgins new office would centralize IT management and oversee, develop, and deploy the FBI’s most expensive, complex, and risky projects. Her most important assignment was to manage the Trilogy project.Higgins was responsible for appointing Depew, who had no IT project management experience as the VCF project manager.////////////////////// Through the years of the VCF project schedules and budgets were blown, IT management changed with the departure of the CIO Bob Dies and many others

In the Summer of 2002, turmoil roiled the FBI's IT management. In May, Bob Dies, the
CIO who had launched Trilogy, left the bureau, turning over his duties to Mark Tanner, who held the position of acting CIO for just three months, until July 2002. He stepped aside for Darwin John, former CIO for the Mormon Church. Chiaradio, who declined to be interviewed for this article, left for a lucrative job in the private sector with BearingPoint
Inc., a global consultancy in McLean, Va., and was replaced by W. Wilson Lowery Jr.
Within a year, Lowery would replace John.
At the same time, SAIC was staffing up. By August 2002, it had around 200 programmers on the job. It was still looking for help, particularly for its security team, which was reviewing design documents that described the VCF software's overall structure, algorithms, and user interface, along with the ways data would be defined and handled.
Matthew Patton answered an ad on SAIC's Web site for security engineers. A 1995
Carnegie Mellon University graduate with a B.S. in information and decision systems,
Patton had financed college through service as a cadet in the U.S. Air Force Reserve
Officers' Training Corps. After college, he spent his four-year tour of military duty at the
Pentagon in the Office of the Secretary of Defense. There he designed and helped program the database and security components for a Web-based application used to plan the
Department of Defense's $400 billion budget.
Patton's still-valid top-secret DOD clearance qualified him to start work as part of the VCF security team. His clearance was provisional—the FBI would have to conduct its own background investigation (as it does for all contract employees) and grant him FBI topsecret clearance. So he was not allowed to see the data the FBI was sending to SAIC, which included information on all of the cases the bureau had digitized to that point, from the 1995 Oklahoma City bombing to 9/11. Instead, he spent a lot of time going through the requirements in his cubicle, segregated from his five colleagues and his boss. He left
SAIC in November 2002, after only three months on the job.
Patton regards himself as a straight shooter. "I'm not much of a culture guy," he admits.
"I say my piece, and if they don't like it, that's too damn bad."
But he quickly realized that SAIC didn't hire him for his opinions. When he began expressing concerns that security was not a top priority on the project, even in the post-
Hanssen era, he was told not to rock the boat.
"My refrain to my boss was, 'Why aren't we more involved? We should be in the thick of things.' But it was more that we weren't really invited and [SAIC teams working on the
VCF] aren't actively seeking our involvement," Patton said in an interview in Chicago earlier this year. "So his take on it was basically, once the designers come up with something, we say good, bad, or indifferent, and if it's not too bad, then we let it go."
Patton recounted his experience purely from memory. Unlike Higgins, who meticulously inserted internal FBI e-mails about Trilogy into her scrapbooks alongside photos of her kids visiting her in D.C., Patton said that he discarded the notebook he kept while he was at SAIC. The only existing artifact of his experience is a copy of the 26 October 2002
Internet posting that essentially got him kicked off the VCF project. The posting, archived at http://archives.neohapsis.com/archives/isn/2002-q4/0090.html, expressed specific security-related concerns and depicts SAIC as giving a clueless FBI exactly what it was asking for, no matter how impractical.
Patton's descriptions of the 800-plus pages of requirements show the project careening off the rails right from the beginning. For starters, this bloated document violated the first rule of software planning: keep it simple. According to experts, a requirements document should describe at a high level what functions the program should perform. The developers then decide how those functions should be implemented. Requirements documents tend to consist of direct, general phrases: "The user shall be able to search the database by keyword," for instance.
"In a requirements document, you want to dictate the whats, not the hows," Patton said.
"We need an e-mail system that can do x, and there's 12 bullets. Instead, we had things like 'there will be a page with a button that says e-mail on it.' We want our button here on the page or we want it that color. We want a logo on the front page that looks like x.
We want certain things on the left-hand side of the page." He shook his head. "They were trying to design the system layout and then the whole application logic before they had actually even figured out what they wanted the system to do."
Recalling the Web pages the agents would bring into the JAD sessions to demonstrate how they wanted the VCF to look, Higgins blamed both SAIC and the agents for creating the overstuffed requirements document. "The customer should be saying, 'This is what we need.' And the contractor should be saying, 'Here's how we're going to deliver it.' And those lines were never clear," Higgins said. "The culture within the FBI was, 'We're going to tell you how to do it.'"
Zalmai Azmi, the FBI's current CIO, has been in that job since December 2003. Originally brought on as a consultant to Mueller that November, Azmi had worked with the director when Mueller was U.S. Attorney in San Francisco and Azmi was CIO of the Executive
Office for United States Attorneys. Azmi saw the Virtual Case File through its final death throes. In an hour-long interview in his office at the Hoover Building, Azmi also traced the
VCF's demise to flawed requirements and emphasized that his office is taking pains to make sure it doesn't happen again.
Azmi insisted that SAIC should have clarified user needs in the JAD sessions rather than working with requirements that were not "clear, precise, and complete." On the other hand, the FBI's lax project management didn't stop the requirements from snowballing.
"There was no discipline to say enough is enough," Azmi said.
The overly specific nature of the requirements focused developers on their tiny piece of the puzzle. They were writing code, Patton said, with no idea of how their piece fit with the others. This presaged the integration problems that would later plague the project.
"The whole working procedure [SAIC project managers] had was very much, 'We'll give you your marching orders and you go,' without too much consideration of how in the world do you glue this sucker back together when all these different divergent pieces come back," Patton said.
Patton also claimed that SAIC was determined to write much of the VCF from scratch.
This included an e-mail-like system that at least one team, to his knowledge, was writing, even though the FBI was already using an off-the-shelf software package, Novell's
GroupWise, for e-mail. "Every time you write a line of code, you introduce bugs," noted
Patton. "And they had a bunch of people slinging code. I'm not saying that the guys were technically incompetent. But bugs happen, and not all programmers are great."
"Every time you write a line of code, you introduce bugs," noted Patton. "And they had a bunch of people slinging code." After several weeks of asking his boss questions and being repeatedly told that he needed to calm down and be "a team player," Patton posted a message to InfoSec News, an email forum which distributes information security news articles and comments from its subscribers. Without naming the VCF specifically, he mentioned that he was working on
Trilogy's case management system and complained that no one was taking security issues seriously. He pointed to some security measures the FBI already had in place that might make the case management system more secure. These included PKI, or public-key infrastructure, a system of digital certificates and independent authorities that verify and authenticate the validity of each party involved in an Internet transaction. He also mentioned Bedford, Mass.based RSA Security Inc.'s SecurID, which uses a combination of passwords and physical authenticators that function like ATM cards to protect various kinds of electronic transactions.
He asked for help in getting in touch "with some heavy-hitting clued-in people over at the
FBI," who would "demand some real accountability from the contractors involved.
"They [the FBI] don't know enough to even comment on a bad idea, let alone tear it apart," he went on. "As a two-bit journeyman I can't seem to get anyone to pay the slightest attention, nor do they apparently (want to) understand just how flawed the whole design is from the get-go."
He ended by asking, "Shouldn't somebody care?"
Somebody did. Sherry Higgins saw the message and promptly reported Patton to the FBI's
Security Division. "He had posted information that was not true and was sensitive," she told me in an e-mail. "He was pretty much a disgruntled employee. Instead of bringing his concerns up the ladder, he chose to post them on the Internet. He blasted the team both at SAIC and the FBI."
"Be careful of him," she warned. "In hindsite [sic], I guess it looks like he is saying now,
'I told you so.' However, at the time, he was disruptive instead of constructive."
In response to Higgins's concerns, FBI agents questioned Patton about whether he had disclosed national security information and breached his top-secret DOD clearance.
"There was nothing in there that was sensitive material," Patton maintained. "It was just not flattering of the FBI and the project itself."
After the interview, the FBI decided not to grant Patton top-secret clearance, making it impossible for him to continue working on the VCF. SAIC did invite him to find another position within the company, but it didn't have anything for him in Chicago, to which he was relocating for personal reasons. So at the end of November 2002, Patton left SAIC and the VCF.
That same month the FBI and SAIC agreed to a basic set of requirements, the baseline that SAIC would start from to build the VCF.
In December 2002, Higgins asked lawmakers to invest an additional $137.9 million in
Trilogy and the inspector general issued a report on the FBI's management of information technology that included a case study of the program. It found that "the lack of critical IT investment management processes for Trilogy contributed to missed milestones and led to uncertainties about cost, schedule, and technical goals." Apparently unperturbed by the findings, Congress approved another $123.2 million for a project whose total cost had now ballooned to $581 million.
Meanwhile, SAIC programmers were cranking out code. The company had settled on a spiral development methodology, an iterative approach to writing software. Basically, SAIC programmers would write and compile a block of code that performed a particular function, then run it to show Depew's agents what it would do. The agents—some of whom were working at SAIC's data center in Vienna, Va.—gave the programmers feedback, and the programmers tried to incorporate the suggested changes. If there was some dispute as to whether the change could or should be made, the agents sent an official request to the change control board, composed of SAIC engineers and FBI personnel, for review.
It wasn't long before the change requests started rolling in—roughly 400 from December
2002 to December 2003, according to SAIC.
"Once they saw the product of the code we wrote, then they would say, 'Oh, we've got to change this. That isn't what I meant,'" said SAIC's Reynolds. "And that's when we started logging change request after change request after change request." Reynolds added that
SAIC's bid on the original contract, and each subsequently revised cost estimate, was based on there being "minimal, minor changes" to the program once a baseline set of requirements had been agreed on. Instead, SAIC engineers were like a construction crew working from a set of constantly changing blueprints.
Some of the changes were cosmetic—move a button from one part of the screen to another, for instance. Others required the programmers to add a new function to a part of the program, such as the graphical user interface, common to all eight development threads. For example, according to SAIC engineers, after the eight teams had completed about 25 percent of the VCF, the FBI wanted a "page crumb" capability added to all the screens.
Also known as "bread crumbs," a name inspired by the Hansel and Gretel fairy tale, this navigation device gives users a list of URLs identifying the path taken through the VCF to arrive at the current screen. This new capability not only added more complexity, the
SAIC engineers said, but delayed development because completed threads had to be retrofitted with the new feature. Once SAIC engineers agreed on how the page crumbs would work, one of the development teams created a set of page-crumb-equipped screens for the other seven teams to use as a model. The design model and supporting documentation were updated, the teams made the change—and the schedule slipped again. When asked how SAIC programmers reacted to agents' change requests, Depew replied,
"Let's just say that we gave them feedback on what they were developing, where it met the requirements and where it didn't. And there was a lot of inconsistency between their development teams."
Higgins was aware that tensions were mounting inside the VCF project over the course of the winter and spring of 2003. Sometimes Depew's team had only two days to review a batch of code. Agents would pull all-nighters to get the evaluation finished, "and in the next iteration their comments wouldn't be taken into account," she said. Sometimes, she acknowledged, these evaluations would include changes to the requirements—functions that the agents had decided that they needed once they saw what they were going to get. Other times the FBI team would find bugs that needed to be fixed.
In March 2003, Computer Sciences Corp., in El Segundo, Calif., which had acquired
DynCorp that month, told Higgins that the final deployment of the computers and networks would be delayed until October. In August, October became December. And in
October, December became April 2004. The problem wasn't the PCs, which had been trickling in since 2001, but changing the e-mail system from Novell's GroupWise to
Microsoft Outlook and, according to the inspector general's 2005 audit, obtaining the components needed to connect the field offices to the wide area network. Higgins added that the delays were compounded by the FBI's own sloppy inventories of existing networks and its underestimation of how taxing the network traffic would be once all 22
000 users came online using their new PCs.
While the FBI and SAIC waited for the networks to go live so they could test the VCF on a real system, changes and fixes continued to strangle the VCF in the crib. Many of the changes had to be to made by all eight of SAIC's development teams. Arnold Punaro,
SAIC executive vice president and general manager, admitted in a posting on the company's Web site that in the rush to get the program finished by December, SAIC didn't ensure that all of its programmers were making the changes the same way. That inconsistency occasionally meant that different modules of the VCF handled data in different ways. Consequently, when one module needed to communicate with another, errors sometimes occurred.
"This, however, did not compromise the system," according to Punaro. The real killers, he said, were "significant management turbulence" at the FBI, "the ever-shifting nature of the requirements," and the agents' "trial-and-error, 'We will know it when we see it' approach to development."
Through the summer of 2003, frustration between the agents and the engineers mounted.
To quell tensions and discuss design flaws the agents believed were creeping into the VCF,
Depew's team asked for a sit-down, what one agent called the "emperor has no clothes" meeting. One Sunday in late September, the agents and the engineers gathered to hash out their differences. Higgins listened in by phone to the first part of the day-long meeting. "There was an awful lot of anger on both sides and a lot of finger-pointing," she recalled. "Nobody's hands were clean." Depew, on the other hand, characterized the meeting as a frank exchange of views. "There was never any animosity shown by my team to the SAIC team," Depew said.
Also in September, the U.S. General Accounting Office (renamed the Government
Accountability Office on 7 July 2004) released a report titled "FBI Needs an Enterprise
Architecture to Guide Its Modernization Activities." The GAO warned that without a blueprint that provides, in essence, the mother of all requirements documents, the bureau was exposing its modernization efforts, including the VCF, to unnecessary risk.
"I suspect what happened with the VCF is that in the rush to put in place a system, you think you got your requirements nailed, but you really don't," said GAO's Randolph C.
Hite, who worked on the report. "It was a classic case of not getting the requirements sufficiently defined in terms of completeness and correctness from the beginning. And so it required a continuous redefinition of requirements that had a cascading effect on what had already been designed and produced."
While stressing that there are no guarantees, Hite believes that "had there been an architecture, the likelihood of these requirements problems would have been vastly diminished." But the abundantly funded VCF juggernaut was already hurtling toward delivery. SAIC began testing the program in the fall of 2003, and according to Higgins, problems started cropping up, some of which the agents had warned SAIC about over the previous summer.
SAIC officials complained to Higgins that Computer Sciences Corp. didn't have its hardware and network in place, so SAIC couldn't adequately test the VCF, crucial for a successful flash cutover. They informed her that they would deliver a version of the VCF to be in technical compliance with the terms of the contract and that the FBI should feel free to make changes to it afterward.
"The feeling was, they knew that they weren't going to make it in December of '03," but they were not forthright about the fact, Higgins said.
On 13 December 2003, SAIC delivered the VCF to the FBI, only to have it declared
DOA.
Under Azmi's direction, the FBI rejected SAIC's delivery of the VCF. The bureau found 17
"functional deficiencies" it wanted SAIC to fix before the system was deployed. As an April
2005 report from a U.S. House of Representatives committee pointed out, there were big deficiencies and small ones. One of the big ones was not providing the ability to search for individuals by specialty and job title. Among the small ones was a button on the graphical user interface that was labeled "State" that should have read
"State/Province/Territory." SAIC argued that at least some of these deficiencies were changes in requirements. An arbitrator was called in. The arbitrator's findings, released on
12 March 2004, found fault with both SAIC and the FBI. Of the 59 issues and subissues derived from the original 17 deficiencies, the arbitrator found that 19 were requirements changes—the FBI's fault; the other 40 were SAIC's errors.
While SAIC fixed bugs, Azmi, with the help of Depew's team, created investigation scenarios that would take different cases from opening to closing and tested them on the
VCF. Those tests revealed an additional 400 deficiencies. "We have requirements that are not in the final product, yet we have capabilities in the final product that we don't have requirements for," Azmi said in an interview.
On 24 March, days after the arbitrator's findings were released, Director Mueller testified to the Senate Committee on Appropriation's Subcommittee on Commerce, Justice, State, and the Judiciary that the VCF would be "on board"—and presumably operational—by the summer of 2004. The director had scant reason to be so optimistic. True, Computer
Sciences Corp. was then delivering the final pieces of equipment to the FBI. By April, 22
251 computer workstations, 3408 printers, 1463 scanners, 475 servers, and new local and wide area networks would all be up and running, 22 months later than the accelerated schedule called for. But Azmi and SAIC had yet to agree on the VCF's ultimate fate, much less when it would be deployed. And when SAIC finally offered to take one more year to make all the changes the FBI wanted at the cost of an additional $56 million, Azmi rejected the proposal.
Azmi was promoted from interim to permanent CIO on 6 May 2004. Four days later, the
Computer Science and Telecommunications Board of the National Research Council delivered a report on Trilogy that the FBI had commissioned. The "graybeards," as Mueller dubbed them, were led by James C. McGroddy, who had headed IBM Research from 1989 to 1995. The report made two major recommendations. The flash cutover that would start up the VCF and shut down ACS all at once must not happen, as a potential failure would be catastrophic for the bureau. And the FBI should create an enterprise architecture to guide the development of its IT systems. The same committee had made both of these recommendations in September 2002, and according to McGroddy, both suggestions had been ignored until Azmi took charge.
Azmi invited the graybeards to talk with him, Mueller, Higgins, and a few other FBI officials on 20 May 2004. Azmi told the gathering that he had already contracted
BearingPoint, where Robert Chiaradio was a managing director and lead advisor on homeland security, to construct the current and future versions of the enterprise architecture by September 2005. And he abandoned the flash cutover idea.
In June, the FBI contracted an independent reviewer, Aerospace Corp., in El Segundo,
Calif., to review the December 2003 delivery of the VCF to determine, among other things, whether the system requirements were correct and complete and to recommend what the FBI should do with the VCF. At the same time, Azmi asked SAIC to take the electronic workflow portion of the VCF, code that was in relatively good shape, and turn that into what was eventually called the Initial Operating Capability (IOC), at an additional fixed price to the FBI of $16.4 million. SAIC and the FBI project team had six months to deliver a software package that would be deployed to between 250 and 500 field personnel in the New Orleans field office, the Baton Rouge, La., resident agency, and a drug enforcement unit at the Hoover Building.
The objectives for the new project were clear: test-drive the VCF's electronic workflow; see how people reacted to the graphical user interface; create a way to translate the output from the VCF forms, which was in the eXtensible Markup Language, into the ACS system; check out network performance; and develop a training program. The IOC was the perfect guinea pig for Azmi's rigorous approach to software development and project management, which he called the Life Cycle Management Directive.
The project also needed different managers. On SAIC's side, Rick Reynolds assumed executive oversight on the project from Brice Zimmerman. Reynolds replaced VCF project manager Pat Boyle with Charlie Kanewske. (SAIC declined repeated requests to interview them.) Depew, like other FBI officials, had only good things to say about Kanewske. He had been Kanewske's project manager counterpart for a portion of the Investigative Data
Warehouse project, the newest, shiniest tool at the disposal of FBI agents and intelligence analysts. Successfully deployed in January 2004, the warehouse translates and stores data from several FBI databases, including parts of ACS, into a common form and structure for analysis. But Depew would not be Kanewske's counterpart for the IOC project. He moved back to
New Jersey, where he became director of the FBI's New Jersey Regional Computer
Forensic Laboratory. When interviewed this past spring, he was overseeing the lab's daily operations and construction of a new wing. He was also anticipating retirement after 31 years of public service and thinking of pursuing job opportunities in the private sector. His final take on the VCF was to the point: "We wanted it really bad, and at the end it was really bad."
As for Sherry Higgins, she went back home to Georgia before the IOC project launched.
She now consults and teaches project management courses for the International Institute for Learning Inc., in New York City.
"When it's not fun anymore, Sherry's not a happy girl," Higgins said of her mood just prior to her departure. "The writing was on the wall that IOC was going to be Zal's project. And I just felt like it would be better for me and for Zal for me to leave."
Azmi handpicked his IOC project manager. He chose the bureau's gadget guru (think of
"Q" from the James Bond movies)—a man with 20 years of experience delivering surveillance technologies on tight schedules. At a meeting this past May at the Hoover
Building, the IOC project manager, whom the bureau made available on condition of anonymity, let me read through an internal FBI report on the IOC and explained the development process in detail. He stressed that the IOC was never meant to be deployed to all 28 000 FBI employees but was intended to test Azmi's methodology. "We followed all of this [process], even in this aggressive timeline, to prove he's got a good framework for managing these projects," he said.
With new management in place, about 120 SAIC engineers began work on the IOC project in June 2004. The FBI and SAIC agreed to keep to a strict development schedule, define acceptance criteria, and institute a series of control gates—milestones SAIC would have to meet before the project could continue.
Azmi, unlike the previous three CIOs, inserted himself into the day-to-day operations of the IOC project. All through the second half of 2004, he met with his project manager every morning at 8:15. Every night before 10 p.m., the project manager would issue a status report indicating what milestones had been hit, identifying risks, and suggesting actions to be taken to avoid mistakes and delays. Azmi's project manager worked closely with Kanewske to adhere to the baseline requirements SAIC and the FBI had agreed on for the IOC in July, thus avoiding a death spiral of change requests. In January, the IOC was rolled out as a pilot right on schedule, and just before the inspector general's stinging critique of the VCF was released.
The report on the VCF from Aerospace Corp., the $2 million study of the December 2003 delivery commissioned by the FBI, began circulating on Capitol Hill at the same time.
[Spectrum's attempt to obtain a copy of the report under the Freedom of Information Act was still being litigated at press time.]
But during a hearing this past 3 February, Senator Judd Gregg (R-N.H.) disclosed that the report said that "the [VCF] architecture was developed without adequate assessment of alternatives and conformance to various architectural standards, and in a way that precluded the incorporation of significant commercial off-the-shelf software." Furthermore,
"high-level documents, including the concept of operations, systems architecture, and system requirements were neither complete nor consistent, and did not map to user needs." Finally, "the requirements and design documentation were incomplete, imprecise, requirements and design tracings have gaps, and the software cannot be maintained without difficulty. And it is therefore unfit for use."
The IOC pilot, meanwhile, ended in March. The verdict: "Although the IOC application was an aid to task management, its use did not improve the productivity of most users," according to an internal FBI assessment.
When asked why the IOC did not improve productivity, the FBI project manager emphasized, "The goal was not to achieve improved productivity. What we learned through this is that when they deploy the work flow, there's a need to roll out an electronic records management capability simultaneously."
In other words, FBI employees, particularly agents, found that the IOC actually increased their workload. Why? Agents filled out forms electronically and routed them to superiors for approval, after which the electronic form was uploaded to the ACS, still in use, to be shared with the rest of the FBI. But to comply with the FBI's paper-based records management system, the form had to be printed out, routed, signed, and filed.
So what did the FBI get out of the VCF's last gasp? "We harvested some of the good work from the past," the FBI project manager told me. "We focused that into a pilot. We tested that life-cycle development model of Zal's, and that is a valid, repeatable process. And now we're in a good position to move on."
FBI officials say they are taking what they learned from the VCF and charging ahead with new IT projects on two major fronts. Last September, the White House's Office of
Management and Budget tapped the bureau to spearhead the development of a framework for a Federal Investigative Case Management System, an effort involving the
National Institutes of Health and the departments of Justice and Homeland Security. The goal here is to provide a guide for any agency in the federal government to use when creating a case-management system.
Then, late last May, Mueller announced Sentinel, a four-phase, four-year project intended to do the VCF's job and provide the bureau with a Web-based case- and recordsmanagement system that incorporates commercial off-the-shelf software. Sentinel's estimated cost remains a secret. The bureau expects to award the contract for phase one by the end of this year for delivery by December 2006. SAIC is one of only a handful of
PHOTO: CHAD DOWLING preapproved government contractors eligible to bid on the project.
The FBI's Azmi seems confident that the bureau is ready to handle a project as complex as Sentinel. He said that the FBI has been planning the program for a year, evaluating commercial off-the-shelf software, creating an enterprise architecture, and establishing a number of IT management oversight boards. The bureau has also provided project management training to 80 IT staff members over the last year.
Even so, Ken Orr, an IT systems architect and one of Mueller's graybeards, remains skeptical. He rated Sentinel's chances of success as very low. "The sheer fact that they made that kind of announcement about Sentinel shows that they really haven't learned anything," Orr said, from his office in Topeka, Kan. "To say that you're going to go out and buy something and have it installed within a year, based on their track record," isn't credible. "They need to sit down and really plan this out, because if they had working software today, they'd have only 25 percent of the problem solved," Orr estimated. The major questions the FBI needs to answer, he contended, include how to bring these new software programs online incrementally and train more than 30 000 people to use them.
Then they could focus on converting millions of paper records as well as all of the audio, video, photographic, and physical evidence that has piled up over the years, which will continue to grow at an increasing rate to support the bureau's counterterrorism mission.
"I would guess that it would be closer to 2010 or 2011 before they have the complete system up and running," Orr said. "That's assuming that you have a match between the software and the underlying requirements, which we know are subject to change."
PHOTO: CLAUDIO VAZQUEZ
PHOTO: DENNIS BRACK, BLOOMBERG, LANDOV
PHOTO: CHAD DOWLING
PHOTO: DAVID STUART
Countdown To Catastrophe
*September 2000FBI IT Upgrade Project, later called Trilogy, funded for US $379.8 million. *September 2001Robert S. Mueller III replaces Louis J. Freeh as FBI director a week before the terrorist attacks of 9/11.
*October 2001Robert J. Chiaradio advises Mueller on software he dubs the Virtual Case
File and brings Larry Depew aboard.
*January 2002FBI receives an additional $78 million to accelerate Trilogy.
*February 2002Joint Application Development planning sessions begin; Sherry Higgins hired. *August 2002Matthew Patton hired by SAIC as security engineer.
*November 2002SAIC and FBI agree on baseline requirements; Patton [above] leaves
SAIC.
*December 2002FBI receives another $123.2 million to complete Trilogy.
*September 2003GAO reports that FBI needs an enterprise architecture.
*December 2003Zalmai Azmi becomes acting CIO; SAIC delivers VCF.
*March 2004Arbitrator finds that of 59 problems, 19 were FBI changes to requirements and 40 were SAIC errors.
*June 2004FBI asks SAIC to develop Initial Operating Capability (IOC) for $16.4 million; FBI contracts Aerospace Corp. to evaluate the VCF.
*January 2005Field trials of IOC begin; Aerospace Corp. delivers its report.
*February 2005 Final Office of the Inspector General's report on Trilogy comes out;
Senate hearing, 3 February.
*April 2005 FBI officially kills the Virtual Case File.
*May 2005Mueller announces a new software project called Sentinel.
*December 2005Contract for phase one of Sentinel to be awarded.