Christopher Fagnant was in Michigan when the ransom notices flashed across screen after screen at Qualtek Manufacturing.
It was the start of a long and costly cyber nightmare.
As company president, he’d heard from the firm’s IT specialist after the first signs of trouble, about 45 minutes earlier. It was 8 a.m. Aug. 31, the Thursday before Labor Day weekend, when the Springs-based manufacturer learned it had been hit with ransomware.
Its data and systems were being held hostage by hackers, and they’d end up paying a high price to get back to business.
Nearly two months later, Fagnant talks about the emotional stress and lingering anxiety, but they’ve reached the point “where it’s no longer every day there’s still a computer that needs to be brought back up or there’s still this that needs to be fixed.”
But he’s taking the unusual step of talking publicly, and in detail, about falling victim to ransomware so that other Springs companies can avoid the same ordeal.
“It’s kind of like anything else — you air your dirty laundry and you find out everybody else was kind of just keeping it under wraps,” Fagnant said. “But I’d say, more than that, it’s that I don’t want to keep my mouth shut because I don’t want it to happen to somebody else.”
Qualtek was not inattentive to cyber precautions. It had a cybersecurity assessment early this year and had been fixing a list of deficiencies.
But that Thursday morning launched an expensive lesson on closing the gaps.
“In the grand scheme of things [the cybersecurity assessment] didn’t help us at all — but it should have,” Fagnant said. “They told us everything we were supposed to do to defend ourselves from an attack, but because we didn’t do certain things right — like the backup — we were still vulnerable.”
Qualtek had onsite and off-site backups, but they discovered the hackers had deleted their onsite backup. And due to a miscommunication, their off-site backup in the cloud fell far short of what they expected.
“What we determined was, the last useful image of our server was from six months prior,” Fagnant said.
So the hackers were holding half a year’s worth of data hostage, along with everything else on the server: digital work instructions, accounting software, email, and Qualtek’s material requirements planning system, which included order entry, invoicing, work order generation, shipping and documentation.
“None of that was accessible,” Fagnant said. “We knew relatively quickly that we were going to have to pay a ransom.
“It was one Bitcoin per [system] that we needed to unlock, and we were lucky — sort of,” he said. “There’s two ways to look at it. One, we shouldn’t have all the stuff we need on one workstation — i.e. one server — but two, we only had to pay one Bitcoin to get that one workstation back.”
The day they paid the ransom, one Bitcoin was worth $4,850.
While the FBI says it does not support paying a ransom, in part because there’s no guarantee the victim will get their data back and in part because it “emboldens the adversary,” a 2016 IBM study showed that 70 percent of ransomware victims pay up.
EosEdge Legal cyberlaw attorney Doug DePeppe said the ransom “is a business decision for the victim.
“While paying the ransom promotes more attacks, if the encrypted data risks destroying the business unless the ransom is paid, I suppose there are not many good options,” he said in an email.
For Fagnant, it was “not a money thing — $4,800 to get your entire business back, it was sort of a no-brainer.”
But when you’ve never used Bitcoin, how do you navigate the payment?
“The screen basically gives you those instructions and says in order to pay, you need to pay via Bitcoin. But in order to pay via Bitcoin, you need a Bitcoin wallet … and normally it takes a number of days to set one of those up, even if you have the money to just put in there,” Fagnant said.
In the end it took help from five organizations — and a couple of false starts — to get that job done.
In the meantime, Qualtek leadership worked frantically with their IT support company as well as the firm that did their cybersecurity assessment, Springs-based IT consultants Navakai, who they’d been interviewing to take over IT support.
“The whole day was just going as fast as you can — super high stress trying to do as many things as we can to get this stuff up and going,” Fagnant said.
Alongside the highly technical rescue operation, Qualtek had to turn back the clock a few decades on operations, hand-writing detailed documentation for parts that needed to be certified.
Fagnant said customers were “empathetic and understanding … but I think if I read between the lines, everybody in the back of their minds is going, ‘How do you let this happen?’ And they’re right … It would’ve been a lot easier if we’d had a [server] image from even a week before. Then it would’ve been: ‘I’m not going to pay the hackers and I’m not going to keep fueling this broken system that allows them to do this’ — but we didn’t. We just weren’t in that position.”
Qualtek had secured cyberliability insurance just a couple of months before the attack, and without it, “right now, at least, I know I’d be about $45,000 in the hole,” Fagnant said.
Qualtek has a $10,000 deductible to pay, and Fagnant has advice for other companies: “Trust but verify. That’s it.
“Make certain that you know what you have in place. … Whatever you think you have, make sure you know what it is and don’t be afraid to spend what sounds like a lot of money now to save yourself a lot of money later.
“The off-site backup process has to be robust. … That’s really what I’ve been going around telling people: If it’s on your list of things to do, make it the thing you do today.”
DePeppe also emphasized preparedness.
“Establish vendor and cyberlaw relationships now, and investigate insurance options,” he said. “Network backup is a key preparation step.”
Regularly backing up data and verifying its integrity also heads the FBI’s list of recommended defenses. Its other recommendations include:
• Secure backups and make sure they’re not connected to the computers or networks they’re backing up;
• Be suspicious of links in emails and do not open attachments in unsolicited emails;
• Only download software from sites you know and trust;
• Keep application patches up to date for operating systems, software and firmware;
• Set antivirus and anti-malware solutions to update automatically; and
• Report any ransomware attack to the local FBI office or file a complaint with the Internet Crime Complaint Center at IC3.gov.