NEWYORK: New York city Mayor Eric Adams said that crowd strike is cyber security software prevalent in computers across the globe and their goal is to identify if there is a problem and to immediately take action. Early this morning, they sent out a software update that inadvertently took systems offline.
Speaking at briefing on IT outage, Mayor Adams said that this was not a cyber attack that was a hit or a cyber hit on our city infrastructure.
“Our city’s IT and security teams, led by the Office of Technology and Innovation, have been working nonstop to troubleshoot problems and restore security,” he said.
Mayor Adams said that CTO, Matt Fraser, has been directly in touch with CrowdStrike to get real-time updates and identify the fastest path to getting all systems up and running as quickly as possible. “I just really want to emphasize this was not a cyber attack. This was an update that took systems offline,” he said.
He said that the preparation is so important that the chief of staff is in charge of doing to make sure that we can immediately respond if there is an IT disruption of this magnitude or if there is a cyber attack.
“We have to be prepared, and this is what the drills are for. There are no calls being held or missed, and there is no backlog at the FDNY and FDNY is reviewing to ensure this remains the case. We want to ensure New Yorkers that 911’s call systems have not been impacted. Our infrastructure and emergency operations, they are all in place and we are going to continue to do that,” he added.
Mayor Eric said that life saving complaints that come through 311 or city agencies are being prioritized. Our traffic signals are working and are fully operational as is the Staten Island Ferry.
“Our water systems are fully operational. Our Summer Rising program will continue as scheduled, including bus service. Although we want to tell parents, you may see disruption on the bus tracking app. The systems are still operating,” he said.
“We expect to continue to see cascading effects of the outage throughout the day. New York City Office of Technology will be doing everything possible to minimize potential service disruptions and restore service to our system as soon as possible. Our city agencies’ IT department will be prioritizing this work today to get things back up and operating online based on the priority of those particular systems. We want to make sure New Yorkers are safe and they can rely on the city services they need,” he said.
“This is a developing situation. We are assured by the coordination that’s taken place from all the agencies involved, as well as the entities that are responsible such as CrowdStrike to make sure we can get this underhand without a minimum amount of interruptions in our city,” he added.
Chief Technology Officer, Mayor’s Office of Technology and Innovation Matthew Fraser said that as we started to see the symptoms manifest, we contacted our partners, both at Microsoft and CrowdStrike.
“Subsequently, we got down to the root of the issue. The thing with technologies like this is in order to be safe and be able to respond to threats that evolve in a continuous basis, you need tools that are capable of being updated in that way. As a result, a tool like CrowdStrike, which gets updated real time, when they push a patch, if that patch goes wrong, this is a perfect example of how bad that can be, he said.
“One patch goes out at about 12 o’clock a.m Eastern Time, and it’s pushed between 12 and 1.30 a.m. Eastern time. In that period of time, we had a number of workstations, a number of computers and servers that were impacted. After notifying CrowdStrike and working with them, by 1:30, they had stopped deploying that patch. Since then, we’ve been in a state where we haven’t seen any additional machines impacted. We’re now in a state where we’re going through the recovery,” he said.
Fraser said that unfortunately, the recovery at this moment was a fairly manual process that requires physical touch to most of the machines.
“We have a team of people that are working across the city in real time, trying to get things back up and running as quickly as possible. As the mayor said, most of our critical needs are in place. Most of our critical services, all of our life safety-related services remain up and running. There’s no impact to 911, no impact to 311, no impact to police, fire, EMS, radio or dispatch operations. Everything is running as expected. In addition to that, our operational technology networks and our SCADA networks, things that supply our water systems, things that operate our traffic lights, none of those systems are also impacted,” he added.
To a question, Fraser said that the way these technologies work is that there’s a control file that gets deployed into the agent that runs on every computer and server. He said that when that control file got deployed, part of the control file was corrupted so when it went to execute and apply the update, the Windows system itself panicked, which is what causes the blue screen that says that they commonly referred to as the blue screen of death.
“That update that went in, that came as a control file into the agent, is the thing that triggered the symptoms that we’re seeing,” he said.
Fraser said that the essential services and life safety-related services are not impacted. However, there are other services that are in the city that may be impacted. “We’re moving through and we have a pathway to recovery, and we’re confident that soon we’ll be back to a normal operating state for the services that may feel slightly impacted, he added.
