The National Weather Service had a serious, systematic internet failure on Tuesday morning, making its forecasts and warnings inaccessible to the public, limiting the data available to its meteorologists.
The shutdown addresses systemic, long-standing systemic issues related to its IT infrastructure that the agency has struggled to address as its service requirements have only increased.
In addition to Tuesday morning shutdowns, the weather service has faced a number of recurring issues with its Internet services in recent months, including:
– Lack of bandwidth, which forced him to offer և implement restrictions on the amount of data downloaded by his customers.
– Launch of a radar site that did not work well և angered users.
– Flooding at its data center in Silver Spring, Massachusetts, depriving it of key ocean nest views. և,
– Numerous shutdowns of NWS Chat, its program for transmitting critical information to broadcasters and emergency managers, based on severe weather events.
Problems with the stability and reliability of the weather service information dissemination infrastructure date back to at least 2013, when The Washington Post began reporting on the issue.
The Weather Service seeks to assess and implement solutions to those problems that affect its ability to protect its life and property.
The shutdown on Tuesday morning meant that the weather service’s leading website, weather.gov, crashed, cutting off access to forecasts and warnings.
“There is a serious, national disruption affecting the distribution of NWS products,” the Weather Service Weather Forecast wrote in College Park, MD.
Weather Service Central Operations Center at 5 p.m. On the 11th, it published a newsletter citing nationwide failures, including forecast offices that lose contact with the agency’s networks, “affect product distribution, data retrieval,” inactive sites, and have no access to NWS Chat.
The lack of data limited the model data և observations that weather service meteorologists could use to make predictions.
Weather forecasters and weather service voters have complained on their Twitter page about the disruptions, with many citing chronic problems with its Internet services.
“Why do such things happen all the time?” It is unforgivable at this moment. NWS people are constantly dealing with IT hurdles to spread their message in recent months. “Frequency and complexity are about the absolute worst I’ve ever seen,” wrote Matt Lance, a Houston-based aerospace expert.
“There is absolutely no word on Twitter that can describe how insane it is that in 2021 the richest and most powerful government on Earth will not be able to save its citizens weather forecasts due to internal internet disruption.” You wrote Celine. , A student of meteorology at Cornell University.
“The perpetual technological problems that NWS has to deal with are completely unacceptable. “The whole country is in a state of disarray when it happens,” wrote disaster expert Samantha Montano.
“The shutdown of @NWS is part of our country’s massive infrastructure problems. “It’s hard to imagine meaningful climate resilience without our literally crumbling bridges, broken roads և referring to the 1995 data service,” wrote North Carolina climatologist Katie Dellon.
“NWS ‘seven-hour shutdown was heading for the peak of the severe weather season… … it was so successful it was a very quiet evening. Whether or not fiber cutting is not the beginning or the end of IT issues at NWS. “I would urge Congress to investigate before the bubble appears,” wrote Victor Gensin, a professor of meteorology at the University of Northern Illinois.
On Tuesday morning, Internet problems seemed to be solved, but they shed new light on many other information technology problems that the weather service has faced in recent weeks and months.
The weather service chat system has proven to be one of its most unreliable systems, failing in many cases, including dangerously severe weather.
The problems became so flawed that one Weather service attempted to opt out of NWS Chat for an external program, a move planned by Weather Service headquarters in Silver Spring, Md.
On March 15, the Birmingham Weather Service in Alma sent an e-mail to its media partners announcing that they had decided to switch to the Slack messaging program immediately before the March 17 tornado, which unleashed nearly 50 swingarms.
“For the sake of public safety, factors beyond our control, NWS (Birmingham) will replace Slack Chat as our real-time communication project until NWS Chat is established as a stable, reliable, reliable backup service. on the spot, “reads a warning sent by air coordinator De on De Bloch.
An e-mail from The Post stated that NWS Chat would become the office’s new backup service, referring գործընկեր media partners ղեկավար emergency managers to join the Slack group, which will be operational the following morning.
“We believe this is the best option for all of us right now,” De Bloch wrote.
The Birmingham Weather Service declined to comment on the move to Slack, but was instructed from above not to do so again.
“The offices have been instructed not to buy their alternative platforms. This function is the responsibility of the NWS headquarters, not the individual offices, “wrote Weather Service Director of Public Affairs Susan Buchenan.
A second “high-risk” tornado was forecast last Thursday, but Birmingham Weather Service was asked to use NWS Chat instead of Slack this time.
Knowing that heavy use disrupts the chat program, the weather forecasters asked partners outside the affected areas to stay away from it to maintain bandwidth.
“If you are not in a serious wx risk area today, please do not use NWSChat,” wrote Rick Smith, warning meteorologist at the Weather Service in Norman.
Many meteorologists on Twitter responded to Smith’s call to relieve stress on the system. However, the chat service was still dark at a time when deadly tornadoes were sweeping south.
Span Ames Spen, a veteran air conditioner whose home was hit while he was covering the tornadoes live, said during his broadcast: “I think NWS Chat is broken again.”
Lab osh son Onson, another highly respected Alabama aeronautician trusted to save lives during the 2019 Lee County tornado, tweeted: “The fact that we can not use NWS’s reliable chat platform is unbelievable and dangerous for those we serve. It is the year 2021. “
The NWS Chat also fell for a while on Saturday night when tornadoes tore through Arkansas, Texas, Mississippi, Tennessee, and a deadly torrential downpour swept through Nashville.
“And down, my favorite NWS Chat,” tweeted Daryl Herzman, a systems analyst at Iowa State University who first helped develop and implement NWS Chat years ago. “I wish I could find a mechanism to help them fix it. (Dozens) of unanswered emails so far. “
In a statement issued by Buchanan, the Weather Service described last week’s problems as “intermittent sluggishness, temporary disconnections”, acknowledging that “we recognize the need and need for this communication տեղ coordination with local partners”.
Buchanan attributed the problems to “increased web traffic due to severe weather in the Southeast due to the loss of one data center on March 9 due to the explosion of a silver fountain at NWS headquarters.”
But Troy Kimmel, a senior professor of meteorology at the University of Texas at Austin, says the chat system is not a temporary problem that has plagued it for years. “It must be increased with 100% efficiency. It is unforgivable, ”he said. “This thing will come back with congressional inquiries, if it hasn’t happened yet.”
Instead of fixing problems with NWS Chat, the weather service, realizing its instability, prefers to pursue other options.
“The good news is that this spring we will officially launch a demonstration project to assess the viability of commercially available, off-the-shelf products as a long-term replacement for NWS Chat,” Buchenan wrote. “We are working to improve the stability of the system in the short term.”
Problems with weather service internet systems have persisted for years, in part due to growing user demand, which the agency has struggled to address.
In December, due to the growing scarcity of bandwidth, the Weather Service offered to limit users to 60 minutes per minute on its large number of sites.
Voters complained about the quota և Earlier this month, the Weather announced that it would instead set a data limit of 120 queries per minute, with only the servers serving the model starting April 20.
“This solution will not affect all other NWS sites,” the weather service said in an email to its partners.
The e-mail states that the Weather Service intends to upgrade its data server և network architecture using funds provided by Congress.
At the same time, on March 9, the central office of the Weather Service at the “Silver Spring” “ruptured the water pipe, which caused a significant, widespread flood”, which affected the data center, the weather service said in a statement.
“Some NWS data have stopped flowing, including data from ocean plants,” the statement said, noting that some plants were being used to “detect and locate a seismic event that could cause a tsunami.”
Former Acting Director of the National Oceanic Atmospheric Administration Neil obacobs, who oversees the Weather Service, said many of the agency’s Internet infrastructure is connected through non-cloud service providers such as Amazon Web Services, Microsoft and Google Cloud.
“I have requested in writing that NWS transfer these applications to our Cloud Partners. That’s part of my internal strategy, “the Trump administration-appointed Jacobs told The Post by e-mail before leaving office.
In July 2020, NOAA released its Cloud Strategy, which stated: “Our data volume և speed is expected to increase as the new geometric surveillance system և increases its data acquisition capabilities, putting a premium on our capacity և IT infrastructure and services to support this growth.” “Upgrading our infrastructure requires the use of cloud services as a solution to meet future demand.”