After flight delays, FAA may add backup system
In the wake of a computer failure that caused delays for 646 U.S. flights on Tuesday, the Federal Aviation Administration plans to upgrade its decades-old technology for flight-plan processing and potentially add a third backup system.
The FAA's central system in Atlanta for handling all U.S. flight plans went down Tuesday because of a corrupted file, according to a FAA representative. Then, when the administration's secondary and lone backup system in Salt Lake City got bogged down with repeated re-entries of flight plans from the airlines, the malfunction caused major delays at airports in Atlanta, Boston, Washington, D.C., and Chicago.
By Tuesday evening, the flight delays were under control, but the computer system in Atlanta wasn't functional until about 1:30 a.m. EST on Wednesday, officials said.
Now, FAA representatives said that by September, it plans to add more computer memory to its data communications network known as National Data Interchange Network (NADIN). And by early next year, the FAA plans to completely upgrade the decades-old data communication network with new hardware and software.
"The big difference is that (the new system) has a lot more memory, so what happened yesterday could never happen again," said FAA spokeswoman Diane Spitaliere.
Laura Brown, another FAA spokeswoman, also said that the administration is considering adding a third backup system for NADIN at its New Jersey research and training facility.
At the heart of the problem--and a potential source of other trouble--was old technology at the FAA, officials said. One representative referred to NADIN as "'70s-era technology" that was bought from a third party and then maintained in-house at the FAA. (Another representative said the data communications network was put into operation about 15 years ago.) Despite its age, the failure--sourced to an IPX9000 packet switch--was caused by a human error that "resulted in the wrong configuration data being loaded onto the switch," officials said.
The cause for the flight delays came from the backup system in Salt Lake City that should have been able to take over operations for the Atlanta system smoothly. But it slowed to a crawl under the weight of airlines re-entering their flight plans repeatedly once they got an error message from the first system. The FAA had to call the airlines to tell them to stop hitting re-enter; the system didn't have enough memory.
The FAA is also upgrading host computers in 20 "En Route Centers" throughout the country. Those centers process flight plans sent from NADIN for planes that are flying in high-altitude air space over their area.
"Those computers were very old and running a language that only a few people left alive knew about," Brown said.





http://php.app.com/fed_employees/search.php
check out this article from March 2005...
http://www.networkworld.com/newsletters/servers/2005/0321server1.html
....
"For Andy Isaksen, computer scientist for the Federal Aviation Administration in Atlanta, having servers that are up 24/7 is imperative."
"Isaksen uses two Philips DS714 mainframe computers as his message-switching network. The DS714 was originally manufactured in 1968 and upgraded with new processors in 1981. Since then, they have been getting increasingly harder to maintain, support and write code for. "
...
"Isaksen will install two Stratus FTservers in his production network - one in Atlanta and the other in Salt Lake City. Two other identical servers are part of a test bed for running applications Isaksen and his staff write."
...
?The two centers run in a load-shared mode and at any instance can take over for each other," says Isaksen. "If one server dies, the other one almost instantly takes over so there is no loss of service to the aviation community."
...
"Isaksen will be deploying the Stratus servers to the field in the middle of next month but doesn?t expect they will go into production until the first of next year. "
So did they push off the upgrade for over 2years and still on the were they still on the Philips machines?? Or, did the Stratus servers fail??
I can understand human error.... to a point. I hope it's a reminder to all of us to be realistic about our plans for the redundancy of our critical services.
Talk about having a backup system. According to 1900.47B directive the FAA has contingency plans for the entire U.S. But they have never trained any one as it states in paragraph 15 of 1900.47B. I dare anyone to visit an ARTCC high altitude center, and at random walk up to a controller and ask them, If the center that boarders your airspace goes down or what the FAA calls ATC Zero what section of their air space do you assume?
All they know is that there is a contingency plan somewhere and they also know that there is an agreement with the next center about airspace. But they have no idea what it is because it has written on it For Official Use Only.
Maybe if the controllers had some training, when a center like Memphis goes to ATC Zero as it did again on Aug. 9th , with 5 centers boarding its airspace, just maybe they could have assumed Memphis airspace.
If went to a public meeting, at this meeting I asked the question, If Miami Center goes ATC Zero who will assume their air space? Rick Ducharme Deputy Vice President of Terminal Services said Jacksonville. But as I asked the next question Has any ARTCC center ever assumed another centers airspace? RICK DUCHARME said NO but yet this is their contingency plan.
Sure make you wonder how safe we are.
Please understand this, just look at FEMA.
these agencies are all protecting the busineses that they are to control.
Most of the hard work is already done. The House of Representatives and the Senate Finance and Commerce Committees have passed legislation on the funding portion of FAA reauthorization that would not only provide crucial funding to modernize our air traffic control system, but create jobs and generate real economic growth for our communities by funding many long-term airport projects.
If this bill is not passed this year, we will have to start all over again with a new Congress and a new Administration. The aviation industry simply cannot afford to wait that long. We call on you and your fellow Senators to finally pass the pending FAA re-authorization.