US Flights delay — computer glitch?

F/N: Apparently, a similar breakdown happened in Canada on the 11th: https://simpleflying.com/canada-notam-disruption-2023/

Canada Had Its Own 'Unrelated' NOTAM Disruption Hours After FAA Ground Stop By Chris Loh Published 1 day ago Also taking place on Wednesday, January 11th, Canada's own Notice to Air Missions (NOTAM) system experienced a disruption. Operated by NAV Canada, this system outage occurred sometime during the morning, preventing the air navigation service provider from sending new updates to airlines. Taking to Twitter at 17:38 UTC on January 11th, NAV Canada issued the following statement: "NAV CANADA's Canadian NOTAM entry system is currently experiencing an outage affecting newly issued NOTAMs, and we are working to restore function. We are not currently experiencing any delays related to this outage. We are assessing impacts to our operations and will provide updates as soon as they are available." According to CBC News, NAV Canada also stated that no flights were delayed as a result, and that the system came back online in the early afternoon. More specifically, at 19:17 UTC, the agency issued a statement on Twitter noting that its system had been restored. While the FAA and NAV Canada both experienced failures of their NOTAM systems, it would appear that the FAA's outage caused more disruption to Canadian flights than the NAV Canada outage. Indeed, while NAV Canada reported no delays resulting from its outage, airlines like Air Canada and WestJet confirmed disruptions to their transborder services (flights between the US and Canada). While Air Canada said that it could not determine the extent of the impact, WestJet told Global News that six of its flights faced delays as a result of the FAA outage.

KF kairosfocus

F/N: I see more here: https://www.latimes.com/world-nation/story/2023-01-11/flight-delays-us-after-faa-computer-outage

The breakdown showed how much American air travel depends on an antiquated computer system that generates alerts called NOTAMs — or Notice to Air Missions. Before a flight takes off, pilots and airline dispatchers must review the notices, which include information about weather, runway closures or construction and other information that could affect the flight. The system was once telephone-based, with pilots calling dedicated flight service stations for the information, but has moved online . . . . The NOTAM system broke down late Tuesday and was not fixed until 6 a.m. Pacific on Wednesday. The FAA took the rare step of preventing any planes from taking off for a time, and the cascading chaos led to more than 1,300 flight cancellations and 9,000 delays by late afternoon, according to flight-tracking website FlightAware . . . . “Periodically there have been local issues here or there, but this is pretty significant historically,” said Tim Campbell, a former senior vice president of air operations at American Airlines and now a consultant in Minneapolis. Campbell said there has long been concern about the FAA’s technology, and not just the NOTAM system. “So much of their systems are old mainframe systems that are generally reliable but they are out of date,” he said. John Cox, a former airline pilot and aviation safety expert, said there has been talk in the aviation industry for years about trying to modernize the NOTAM system, but he did not know the age of the servers that the FAA uses. “I’ve been flying 53 years. I’ve never heard the system go down like this,” Cox said. “So something unusual happened.” According to FAA advisories, the NOTAM system failed at 5:28 p.m. Pacific time on Tuesday, preventing new or amended notices from being distributed to pilots. The FAA resorted to a telephone hotline in an effort to keep departures on track overnight, but as daytime traffic picked up Wednesday, it overwhelmed the telephone backup system. The FAA ordered all departing flights grounded early Wednesday morning, affecting all passenger and shipping flights.

We see a third element, a reversion to phone hotlines, overwhelmed. That, too was known at outset. This now becomes a case of a habit of metering out information in ways that in the end will undermine the respect of the public, not just travellers. This is a window on a much wider issue of transparency, truthfulnes and trustworthiness. And we also see numbers on the scope of cancellations and delays, this was not no bid deal. KF kairosfocus

F.N: CNET reports https://www.cnet.com/news/thousands-of-flights-delayed-after-faa-outage/

Thousands of Flights Delayed After FAA Outage Caused by 'Damaged Database File' An overnight outage grounded flights across the US, but delays persist as air traffic resumes. The FAA says it traced the outage to a damaged database file. Jan. 11, 2023 9:46 a.m. PT

FAA tweet:

The FAA ?? · Jan 11, 2023 @FAANews · Follow Replying to @FAANews Update 5: Normal air traffic operations are resuming gradually across the U.S. following an overnight outage to the Notice to Air Missions system that provides safety info to flight crews. The ground stop has been lifted. We continue to look into the cause of the initial problem The FAA ?? @FAANews · Follow Update 6: We are continuing a thorough review to determine the root cause of the Notice to Air Missions (NOTAM) system outage. Our preliminary work has traced the outage to a damaged database file. At this time, there is no evidence of a cyber attack. (1/2) 7:31 PM · Jan 11, 2023

They would have known this at outset. KF kairosfocus

Vivid, yes, she should have been properly escorted and treated with dignity. Instead she was publicly arrested for a traffic violation. Policing agencies will be well advised to avoid obvious one-sidedness. KF kairosfocus

KF Forgot to add that the mother of the only one who was killed was arrested on the same day they were spewing their lies. Vivid vividbleau

KF Got it. I am just so tired of being gaslighted by the massive propaganda machine. Five officers killed on Jan 6th a blatant, intentional lie.The vote that allows babies to be killed after birth , we are no different from the worshipers of Baal “Temple prostitution and sensual forms of worship were common in Baal worship as was occasional human sacrifice, namely children, who were burnt alive to appease the god of fertility” We are not a civilized nation. Vivid vividbleau

KF/21 :-) chuckdarwin

Vivid, 5 above, let us refrain from extreme comparisons. There is too much narrative manipulation yes, but it is not yet Goebbels on a routine basis. That said, I am very troubled by what seems to be a playing out reichstag fire type incident. KF kairosfocus

AC, no I did not, and V would be wrong to label a spokesperson as Goebbels who was an arch propagandist. US Govt spokespersons are increasingly propagandistic but are not at his level yet. My main reference is to arstechnica, which came out yesterday, at last some facts. From the outset they should have been as forthcoming as the Philippines. They instantly knew there were corrupt files, it seems in the main database and its first backup. The wider issue of software development hell needs to be addressed, as well as budgets. Too many big ticket software projects fail or are sub par. KF kairosfocus

@kairosfocus:

AC, you too, Breitbart is not a neo-nazi hate site and pardon your hostility and accusatory attitude are showing.

Oh please, you know very well that I was talking to V, who made the comment "Goebbels aka let’s “circle back” , another DIE hire (...)". V is not a snowflake. No need to whine like a leftist feminist. Btw. I was referring to a Retures article which is linked in one of the linked articles you've posted: https://www.reuters.com/technology/faa-has-struggled-modernize-computer-air-traffic-operations-2023-01-12/ AndyClue

F/N2: From this article, it looks like they are in system development hell:

FAA tech problems were previously described in a March 2021 report by the US Department of Transportation Office of Inspector General. The report discusses the FAA's Next Generation Air Transportation System (NextGen), "a multibillion dollar infrastructure project aimed at modernizing our Nation's aging air traffic system to provide safer and more efficient air traffic management." "NextGen's actual and projected benefits have not kept pace with initial projections due to implementation challenges, optimistic assumptions, and other factors," the report said. Summarizing the results of the Inspector General review, the report said the "FAA has struggled to integrate key NextGen technologies and capabilities due to extended program delays that caused ripple effect delays with other programs. In addition, the Agency has not fully made use of its own internal NextGen benefits analyses to help prioritize future implementation decisions. Finally, critical controller automation tools are not yet in use, which hinders FAA's ability to test and evaluate the full impact of new technologies. Leveraging these lessons learned will be critical for deploying NextGen's advanced capabilities, achieving benefits, and modernizing the NAS in a timely manner."

Notice, how hard it is to get a complex, deeply integrated system right. kairosfocus

F/N: Well, well, well, went looking for Wiki, but found "der sturmer, 2" -- NOT: https://arstechnica.com/tech-policy/2023/01/faa-outage-that-grounded-flights-blamed-on-old-tech-and-damaged-database-file/

FAA outage that grounded flights blamed on old tech and damaged database file Notice to Air Missions outage puts spotlight on FAA's struggles with technology. Jon Brodkin - 1/12/2023, 12:41 PM A Notice to Air Missions system outage that grounded flights across the US yesterday morning seems to have been caused by a damaged database file, the Federal Aviation Administration said last night. "The FAA is continuing a thorough review to determine the root cause of the Notice to Air Missions (NOTAM) system outage," the FAA statement said. "Our preliminary work has traced the outage to a damaged database file. At this time, there is no evidence of a cyber attack. The FAA is working diligently to further pinpoint the causes of this issue and take all needed steps to prevent this kind of disruption from happening again." . . . . CNN also wrote that one corrupt file was found in the main NOTAM system [--> corrupt data in main system], and another corrupt file was found in the backup system [--> more corrupt data in the backup, was this the same file? how corrupt, why?]. CNN provided more details as follows:

In the overnight hours of Tuesday into Wednesday, FAA officials decided to shut down and reboot the main NOTAM system—a significant decision, because the reboot can take about 90 minutes [--> 90 + 30 = 2 hours], according to the source. They decided to perform the reboot early Wednesday, before air traffic began flying on the East Coast, to minimize disruption to flights. "They thought they'd be ahead of the rush," the source said.

When the system came back online, "it wasn't completely pushing out the pertinent information that it needed for safe flight, and it appeared that it was taking longer to do that," CNN quoted its source as saying. FAA struggles with technology The source blamed old tech infrastructure. "Because of budgetary concerns and flexibility of budget, this tech refresh has been pushed off," CNN quoted the source as saying. "I assume now they're going to actually find money to do it." [--> where have budgetary priorities gone instead, post 9-11 this is critical infrastructure, or is it?] Another FAA computer problem in the En Route Automation Modernization (ERAM) system caused flight delays at major airports in Florida less than two weeks ago. [--> so, we have more than one relevant incident] On its website, the FAA says it is "modernizing the NOTAM system to improve the delivery of safety critical information to aviation stakeholders," with the goal of "provid[ing] pilots, flight crews, and other users of the National Airspace System (NAS) with NOTAMs that are relevant, timely and accurate." The FAA has struggled to modernize its computer and air traffic operations, a Reuters article pointed out today. "In October, for example, the FAA said it was working to end a long-ridiculed, decades-old practice of air traffic controllers using paper flight strips to keep track of aircraft. But adopting the change at 49 major airports will take the FAA until late 2029," Reuters wrote. [--> seven more years? How long to write a tablet, touch screen app and integrate?]

kairosfocus

AC, you too, Breitbart is not a neo-nazi hate site and pardon your hostility and accusatory attitude are showing. If you have something genuinely substantial to say, kindly put it on the table with its warrant. Even HuffPost or Wikipedia would do. KF kairosfocus

CD, you really need to learn to respect others and basic rights including to innocent reputation. Waltzing in with instant, unsubstantiated dismissive remarks on conspiracy theorising is out of order and you must know that. Then oh your style, is a squid ink cloud distractor on being called out. Lessee: your + right is obvious, to + innocent + reputation is just as obvious. Joining the two is perfectly coherent and comprehensible. Rights imply duties, and that is reasonable. Waltzing in with accusations or ad homs just so, no need to regard the reputation of the [obviously despised] other, speaks. It is now far too common and is corrosive of civilisation. There is a serious issue of sci tech news on the table. Do you have something substantial to say, apart from you personally were lucky to avoid serious delay? [That has no bearing on the overall picture.] KF kairosfocus

PS, contrast the Philippines: https://www.aerotime.aero/articles/philippines-air-traffic-recovering-following-atc-outage

Following a power outage at one of the Air Traffic Control (ATC) centers in the Philippines on the first day of 2023, air traffic is recovering in the country and its main international airport, Ninoy Aquino International Airport (MNL). “At around 9:49 AM local time [GMT +8 – ed. note], the Air Traffic Management Center (ATMC) which serves as the facility for controlling and overseeing all inbound and outbound flights and overflights within the Philippine airspace, went down due to power outage, resulting in loss of communication, radio, radar, and internet,” said Jaime Bautista, the Secretary of the Philippine Department of Transportation (DOTr), in a statement late on January 1, 2023. At one point in time, there were zero airborne flights above the Philippines, according to flightradar24.com.

Notice, straightforward. Power went down, presumably including backup (a bad sign). Notice, too, Internet is integrated with the network. I suspect -- just on security -- the US ATC system should run over the general telephone network, but isolated from the common Internet. Of course, I could be wrong, and if the ordinary Internet has been brought in to cut costs, that would be a vulnerable access point for hacking. However, the main point of lack of transparency is made. Sliding down on the article about the Philippines:

“The primary cause identified was a problem with the power supply and the degraded uninterrupted power supply which had no link to the commercial power and had to be connected to the latter manually,” [ --> why is that? something is wrong there] elaborated Bautista. Additionally, the outage was followed by a power surge which affected the ATC equipment. At 4:00 PM local time (GMT +8), the ATMC resumed partial operations and an hour and 50 minutes later, the center resumed full operations while equipment was still being worked on.

This is one day after the event. kairosfocus

Folks, getting back to the issue. Let us notice, keeping planes on the ground and rebuilding a known good present state in the context of reloading the software, restored "normal" function. That is suggestive. The system generally works reasonably reliably, but something put it into an untrustworthy state. It looks like a hack has not been eliminated, given some remarks; which could raise questions of cyber hack or system corruption by some sort of invasive malware. Which, could have been lurking then triggered. Or, something was coming from the network that was not just isolated corrupt data. Could there have been an avalanche of echoed messages overwhelming the network? (Denial of service of some sort, through network overload.) Assuming a hot backup, that could cripple that too, forcing shut down, rebooting a known good initial state from offline storage, and rebuilding a current state from trusted data. We need sound information starting with clarification, was this a hack, and if so who . . . given recent known hacks. Of course, this was specifically software not hardware. The silence and vagueness suggest something embarrassing happened or something that if it were public knowledge would trigger massive public anger. Something is odd here (knowing the tendency to manipulate public information) and needs attention. KF kairosfocus

Why are they in a hospital bed? Neither one of them gave birth. Pure theatre just like the time when he rode to his office in a gas guzzler SUV and got out a few blocks away and got on his bike and road the 2 blocks virtue signaling how committed he was to carbon reduction. Pure theatre. Clown show https://pbs.twimg.com/media/FmM4t-vXgAAtnpH?format=jpg&name=medium Vivid vividbleau

Wait until planes start falling from the sky (sarc) FAA priorities https://www.breitbart.com/clips/2018/06/02/aviation-attorney-to-tucker-carlson-faa-is-putting-diversity-ahead-of-safety/ https://www.breitbart.com/politics/2023/01/11/faa-2023-budget-features-focus-climate-crisis-racial-equity/ https://www.breitbart.com/politics/2021/10/14/transportation-secretary-pete-buttigieg-on-paid-leave-since-mid-august-amid-supply-crisis/ https://nypost.com/2023/01/11/37-hour-amtrak-train-delay-prompts-passengers-to-call-911/ Vivid vividbleau

@Vividbleau:

The FAA priority’s https://www.breitbart.com/politics/2023/01/11/bumbling-faa-focused-on-lgbtqi-inclusivity-herstory-gender-neutral-language/ Vivid

Der Stuermer is confused. The priority of the FAA is to modernize their infrastructure. FAA's systems have been a failure for a very long time... long before the leftists' push for gender neutral language. Also I don't think the FAA's communication department can help out fixing the infrastructure, they are not engineers. AndyClue

I’m not sure what the situation in Eastern Europe has to do with an FAA glitch. This, on the other hand, is a doozy:

However, you are yet again in violation of the right to innocent reputation by projection to the despised other.

Sounds like something out of an abstruse Derrida tome. KF, you really need to lighten up…… chuckdarwin

CD, no one has suggested a conspiracy to shut down US flights, though there has been concern on US funding of Ukraine (which DOES have a reputation for extraordinary corruption); you will note, please, my remark above that such funding may be advisable given the dominoes waiting to topple behind -- and I am concerned Putin may escalate sharply, as happened in 1939/40 with Finland. However, you are yet again in violation of the right to innocent reputation by projection to the despised other. I suggest reconsidering. KF kairosfocus

“You guys can make a devious conspiracy out of anything. I’m gonna nickname you “The Mountain out of a Molehill Gang.” It’s really pathetic……” What conspiracy theory? Vivid vividbleau

I flew from Minneapolis to Vegas yesterday. It was no big deal. You guys can make a devious conspiracy out of anything. I’m gonna nickname you “The Mountain out of a Molehill Gang.” It’s really pathetic…… Viva Las Vegas—Elvis lives…. chuckdarwin

Vivid, infrastructure software is not showy and this one has to have a huge, networked back-end database that is updated in real time. I saw something on reloading the system, suggesting they reverted to a known good point and dealt with the incremental updates on airport etc status. I suspect corrupted information and maybe broken networks? As for billions to Ukraine, I am a little realpolitik, backing a bad regime that blocks a worse may avert a catastrophic war, and buys you leverage to promote reform. If Ukraine falls, Belarus is next with the Baltics behind. If Finland and Sweden want to join NATO and Switzerland wants to upgrade its silent de facto informal quasi membership, they are worried. Then, the Chinese are watching, with an eye on Taiwan. The corruption is bad, the undemocratic acts are bad, WW4 -- WW3 was the Cold War -- is horrifically worse. KF kairosfocus

The FAA priority’s https://www.breitbart.com/politics/2023/01/11/bumbling-faa-focused-on-lgbtqi-inclusivity-herstory-gender-neutral-language/ Vivid vividbleau

"They can give billions upon billions to the corrupt, undemocratic, Ukraine regime" ...and toss around billions and billions in the name of Climate Change and not affect the temperature at all. Andrew asauber

KF Evidently they have no back up systems. They can give billions upon billions to the corrupt, undemocratic, Ukraine regime but nothing for a crucial back up system for the FAA. Makes one wonder how much dirt they have on the corrupt Biden crime family. Vivid vividbleau

No vividbleau

I see something about reloading so maybe they had to roll back to a known good point and build forward? But what triggered the choke-up? Software is not a matter of throw $$$$ at it, you need solid competence, so if a loss of key people due to the pandemic mismanagement is involved this may be onward cascaded but delayed failure. And if this is just the first swallow of the summer . . . But then, we need facts, straight facts, we are getting vague statements and mixed messages, hack vs no external interference, etc. Is there a press conference by a senior manager? kairosfocus

Goebbels aka let’s “circle back” , another DIE hire, will be on soon. She is one wrung lower on the intersectionality ladder than Pete so I am sure we will get some truthful answers, not! Vivid vividbleau

KF Obviously it’s a system error the Transportation Sec floated a hack. More likely one of the causes had to do with Covid and the Vax mandate I understand a lot of systems software people retired. So they are understaffed. Like everything else this administration does we will be gaslighted and they will blame “not enough money we need more” This is not his first rodeo https://www.foxnews.com/politics/buttigieg-battered-crises-first-two-years-transportation-secretary-prime-example-failing-up He couldn’t even fix the pot holes in South Bend and one of the first things he did after his appointment was to go on maternity leave Vivid vividbleau

Vivid, could you explain? Is there a bug that has crawled out because some threshold was crossed? Are networks overloaded? What is the detailed report or credibly informed discussion? I know no sufficiently complex software can be bug free, so graceful degradation should be built in. KF kairosfocus

Clown show. This is what happens when DIE (DEI) trumps competence. The American people are getting the government they deserve. We have just finished the mid terms and the voting public voted for this, every vote cast for a Dem was a vote for Biden and his administration. Let it all burn down. Vivid vividbleau

US Flights delay — computer glitch? kairosfocus