For three days last month — after a botched update from the security software firm CrowdStrike brought down an estimated 8.5 million Windows devices around the world — a roving band of IT experts descended upon University of Maryland Medical System hospitals and outpatient centers to root out devices stricken by the dreaded “blue screen of death.”
“There’s a picture that somebody took of us in an office building,” said Joel Klein, the system’s senior vice president and chief information officer, “where it was this combo of PC techs, project managers and people like me who don’t normally get into the guts of computers.”
The outage, which temporarily paralyzed operations at hospitals, 911 call centers, airlines and businesses globally — costing companies an estimated loss of more than $1 billion in revenue — began just after midnight on July 19. After initial fears of a massive ransomware attack were dispelled, word spread quickly that the fix would require IT workers to manually delete the faulty file from each affected computer.
At the University of Maryland Medical System — where nearly two-thirds of the laptops, workstations and servers across 50 sites were rendered useless by the CrowdStrike update — this was a herculean task that required thousands of man-hours to correct.
“Our team members are consistently focused on delivering high quality, world-class patient-centered care across the System,” Dr. Mohan Suntha, president and CEO of the University of Maryland Medical System, said in a statement. “I am incredibly proud of the way that our organization was able to respond to and navigate the challenges associated with CrowdStrike, throughout the day, over the weekend and into early the following week.”
Here’s a timeline of how one of the most hectic stretches in IT history played out for one of Maryland’s largest hospital systems, assembled from interviews and information provided by a system spokesperson.
July 19, 12:09 a.m.
The first device in the medical system received the corrupted update file from CrowdStrike. Over the next hour and a half, nearly 20,000 of approximately 33,800 devices across the system received the file and became unusable. Individual troubleshooting didn’t work. Phone calls and tickets streamed into the IT help desk to report the problem.
1 a.m.
Klein got a phone call, alerting him of the outage. Soon, Kristie Snedeker, vice president of Shock Trauma, and Adam Canterbery, senior manager for end-user computing, also were woken up and jumped on calls to begin understanding what was happening.
2 a.m.
All hospitals and sites across the medical system were directed to begin “downtime procedures” — non-technological protocols that replace procedures that require computers or other technology to complete.
Computers are intertwined with nearly every facet of health care delivery. They allow doctors and nurses to almost immediately access a patient’s entire medical history, including what prescriptions they’re taking and any allergies they have, and quickly send orders to labs and pharmacies. But even though technology makes health care safer and more efficient, Klein said, patients can be treated without it.
“You can still go take out somebody’s appendix. You can still do CPR. You can still provide lifesaving care,” Klein said. “It’s just a lot slower, a lot more cumbersome, and the handwriting is worse.”
In some ways, it’s a return to how hospitals worked in the 1980s — when Klein worked in emergency medicine and when doctors and nurses had to ask patients for a list of their medications and track their symptoms on paper charts, instead of computers.
In the early hours of July 19, hospital officials had to figure out the extent of the disruption. Would nurses be able to clock in and out? MRI and CT machines might be working, but were they able to push their results to the platform where a radiologist could read them? How about security cameras — were they functioning correctly?
These are questions that are hard to answer when it’s 3 a.m. and everyone is asleep, Snedeker said. As she drove to Shock Trauma in downtown Baltimore, she called the hospital’s department heads.
“This is what is happening,” she told them. “I need you to wake up and I need you to get an idea of how things are working in your space and then I need you to report in about 20 minutes.”
3 a.m.
Canterbery and his co-workers identified a way to break through the computers’ blue screens of death and delete the file causing the problem — but the process was a doozy. It involved entering a 48-digit key to get past the encryption software on each device. The key for each device across the system is unique and can be accessed only through a secure server.
Two hours later, Canterbery had drafted a step-by-step process IT workers could follow to manually unlock each device. He distributed the document to technical teams across the system to begin the remediation effort. Computers and workstations critical to patient care — such as those in the operating rooms, trauma resuscitation unit and post-anesthesia care unit — were prioritized.
6:30 a.m.
Kayla Zellous, a licensed practical nurse on the orthopedic floor at the University of Maryland Baltimore Washington Medical Center, got a call from her supervisor, asking whether she could work that day, even though she was scheduled to be off.
“Today is going to be a little bit different,” her supervisor told her when she arrived at work.
The prospect of a day with little access to computers was initially intimidating, Zellous said. The 27-year-old has been a nurse for only two years. While some of her co-workers remembered the days of paper charting, that was new for her. Computers in four of the patient rooms where she was assigned to work that day were functional, but she couldn’t use them to order labs or communicate with the pharmacy.
Later that morning, Zellous and her co-workers pulled all of the computers on the floor into the hallway so they would be easier to access for the IT workers running around to unlock them. By 4 p.m., most of the unit’s computers had been rebooted and were ready for use, she said.
“It was a difficult time, but I definitely have a humor about myself. I don’t let myself or the people around me get down,” Zellous said. “I really built up the morale that day, because I could tell a lot of people were like, ‘What do we do?’”
9 p.m.
Word of another strategy to unlock computers affected by the CrowdStrike update got around to the medical system. The new process was simpler and faster — but required a USB flash drive for each computer.
Canterbery and his co-workers tested the new method and wrote up new instructions for unlocking computers. Then, Canterbery went on Amazon to make an overnight order of flash drives, which arrived at his front door at 4 a.m. the next day. He later bought an additional 100 drives from Micro Center.
“I think in total, I spent a little over $500 on flash drives, not counting what we had already,” Canterbery said.
Saturday and Sunday – July 20 and 21
At around 7 a.m. the next day, the first shift of IT workers arrived at hospitals and outpatient centers across the system and got to work.
The time it took to unlock each computer varied. Some took just a few minutes, but others required workers to suit up in personal protective gear and enter a patient’s room, then use a key to unlock a cabinet to access the device.
Even though IT staff members were required to work only one shift, Canterbery said, many worked through the second shift, all the way to 10 p.m. More than 200 other employees who weren’t scheduled to work volunteered their time to help with the remediation effort. On Monday, July 22, technical teams set up kiosks in the system’s hospitals and outpatient centers to unlock any stragglers.
Moving forward, some system hospitals held meetings to talk about the response and what could have been done differently. It might be helpful for the medical system to review its emergency IT playbook on a more regular basis, Canterbery said, and possibly expand it to include more scenarios.
By the time the weekend ended, Canterbery was about ready to collapse, he said with a laugh. Between Friday morning and Sunday evening, he had walked nearly 50,000 steps — or more than 24 miles. The weekend was grueling, but it also had some bright spots.
Most of the time, Canterbery said, IT workers just hear from people when things go wrong. But on July 19, they were the heroes. Nurses and other clinical staff members were excited when they saw them arriving on their units. They thanked them, and some offered food and drinks to the workers.
“It was nice,” Canterbery said.
Around the Web
