An $11 million air safety study sponsored by NASA that asked pilots to reveal mishaps and problems they encountered was so riddled with flaws that it could not provide useful insights about U.S. flight safety, according to a new review by a national scientific panel.
The research project ran for nearly four years and involved telephone interviews that lasted about 30 minutes with 29,000 pilots. But it was so flawed in its design and how it was run that its results were useless for measuring safety problems or trends, according to a report released Wednesday by the National Research Council.
The report is the latest critique of the now-defunct program. The project became known publicly two years ago when The Associated Press disclosed that NASA wouldn’t provide the results despite a Freedom of Information Act request. NASA told AP that disclosing its sensitive findings could harm airline profits and public confidence in the carriers.
When Congress demanded an explanation and expressed interest in continuing the project, NASA’s then-administrator, Michael Griffin, said the survey did not meet the space agency’s scientific and research standards and would not be resurrected.
The pilot interviews were conducted from 2001 to 2004.
NASA asked the National Research Council to evaluate the project’s methodology and analyze what the pilots said. However, the panel declined to analyze the results themselves, citing the difficulty in working with the censored data that NASA turned over.
NASA has said it censored parts of the data to protect commercial secrets and the anonymity of pilots. Their names were never included, but NASA officials worried someone could scrutinize information about plane types, flights and locations to identify pilots.
“We would have liked to have had the opportunity to see the entire data set,” said Clinton V. Oster, Jr., co-chairman of the National Research panel who is associate dean at Indiana University’s School of Public and Environmental Affairs. “We’ll never know what might have happened if we had.”
But Oster said the research project’s flaws extended beyond the censored data.
“The extent and magnitude of these problems raises concerns about the accuracy and reliability of the data,” the new report said.
The expert who helped design the survey, Jon Krosnick, said the research council published “untested and unsupported speculations about possible flaws in the study design, with no direct evidence that the findings are unreliable or inaccurate.”
NASA shut down the project amid concerns by federal aviation regulators when early results appeared to show greater numbers of incidents than other government monitoring systems recorded.
“While the survey team made mistakes, they were hampered by the agency’s lack of support for the program, which effectively condemned it to failure,” said Rep. Bart Gordon, D-Tenn., who led House Science and Technology committee hearings on the project.
“We still cannot conclude that a survey would or would not help make the skies safer. All we know is that the agency wasted millions of dollars in a halfhearted effort.”