This LA hospital’s computers weren’t backed up? DR fail made “ransomware” pay

[steve2470]

http://www.computerworld.com/article/3034721/healthcare-it/hollywood-hospital-ransomware-40-bitcoin-itbwcw.html

(I posted about this a few days ago, apparently they paid up, original article title)


A Hollywood hospital has had to pay a ransom to get its data back from hackers who encrypted everything on its computers. The medical center's CEO confessed to spending around $17,000 in Bitcoin—this after a week spent failing to restore important health data, email, and other critical stuff.

In other words, Hollywood Presbyterian Medical Center failed at disaster-recovery (DR). Sounds like it either didn’t have any backups, or the restore didn’t actually work.

Oopsy daisy, hashtag-fail, etcetera. In IT Blogwatch, bloggers see a lesson for all of us: Backups aren’t backups unless you can restore them!

.......

Here’s a local take. Courtesy of Richard Winton— Hollywood hospital pays $17,000 in bitcoins to hackers who took control of computers:

more at link

[Erich Bloodaxe BSN]

1. Still, 17k is a lot less than the original demands.

They got off cheaply, and can learn from it. Do your damn backups, and be ready to drop in new drives if needed.

[joeybee12]

2. Remind me never to get sick...

I'm sure this isn't an isolated incident with for profits hospitals now trying to maximize those profits.

[Rex]

3. This is why the cloud is a good thing. Local backups are not good enough.

Back it up somewhere, where the hackers have to go into an entirely different set of security protocols. Local is always bad without a secondary offsite somewhere.

So we see our very first cyber-terrorism using bitcoin as the ransom transfer? Not good and to pick on a hospital makes it twice as evil imo.

[awake]

4. I am sure the hackers got in through a "BackDoor"

just like the one that the FBI wants Apple to create for the iPhone

[Sam_Fields]

6. All they do is create an email with a payload or link

and if the user opens it then the ransom ware is downloaded and it starts encrypting files. Microsoft should create something so that changing file names requires a password.

[LiberalArkie]

5. What has been happening is a computer picks up a trojan (might I add usually an executive) or high

management that when plugged into the corporate lan injects the trojan. The trojan/virus/ whatever you want to call it starts encryption of all the data. The "hackers" wait sometimes months or years. The servers keep doing backups but the backups are all encrypted thus useless.

[Egnever]

7. I don't think that is what happened here

Reading the statement from the hospital. It appears that it was easier and faster to pay the ransom than to do the full restore.

Also once the files are encrypted people notice pretty quickly as they lose access. If this acts like most of the other similar ransom ware this is not something that sits for months it begins immediately to start encrypting files and aggressively goes after any network locations it can get to.

Sounds like from the statement because it was a hospital with patients affected they wanted to be up and running as quickly as possible, so they just paid the ransom.

Somewhat sad that it is easier to pay than to restore but in a hospital setting I can see how it could save a ton of the to get you back to functionality.

[LiberalArkie]

10. Well if you are a 24/365 business and your last good backup (before the trojan hit) is 1 week old,

or even 2 days old you are in trouble. Until people that take corporate laptops home understand what can happen, this will continue to happen.

[backscatter712]

8. Who are the dumbfucks running that hospital's IT department?

No backups, no protection against malware.

They got completely pantsed, because their IT people did not have their shit together.

How can the IT at a FUCKING HOSPITAL not have a proper backup?

Heads had better roll for this.

In a properly run IT department, infected computers get wiped and re-imaged. The data's all kept on servers, and the servers are well-guarded, and properly backed up. A malware problem on a workstation can get fixed in a half hour. It would take a few hours if a server got pwn3d to nuke it and restore from backup, while operations continue on a backup server.

For something as critical as patient records, you don't just have one server. You have multiple redundant servers. All backed up. And you practice the backup plan. Practice a backup and practice a restore, so when something happens, you know what to do and can unfuck your server quickly.

Incompetence. Complete incompetence.

[dilby]

11. It's real simple, hospitals want to maximize profits.

IT security is a cost that does not bring in any revenue. It's why a hospital won't bat an eye at paying for a super expensive piece of medical equipment that they can use to run tests they will charge a patient for and get reimbursed by insurance. But they won't spend money on backups, new OS, virus and malware protection or IT personnel.

[Egnever]

12. I dont think that is the case here

I think it was quicker to pay the ransom than to do a full restore.

At least that is what the statement from the director seems to imply.

[Logical]

14. Makes sense. Nt

[backscatter712]

15. Sounds like the director's desperately trying to cover his ass.

If the IT guys had their shit together, it would have been easier, and cheaper, to restore from backup.

Sounds to me like one of two things.

1. They didn't do a backup at all, which wouldn't be out of character for a for-profit hospital run by a Rick Scott wannabe looking to cut corners at every opportunity. In this case, they cut the wrong corner and it cost them dearly.

2. They half-assed the backups. They ran some program that allegedly put stuff on a backup server, or on backup tapes, but when the system shit itself due to the malware, and they tried the restore, it barfed because they didn't know what they were doing.

They certainly didn't have things like redundant servers, or proper firewalls for the servers, or IT that knows how to protect their servers from malware. Or a backup system that actually works.

They spent a full week fighting with the system trying to put it back together before paying the extortion money.

In the end, they should have had an IT department capable of doing a server wipe, re-image, and restore from backup in a few hours, or an all-nighter in an emergency.

[Egnever]

16. Including all workstations that connect to it,?

[backscatter712]

17. All the data should have been on the servers.

Getting the malware off of workstations should have been a matter of wiping them and re-imaging them.

The fact that they had to pay bitcoin to decrypt their data shows they fucked up beyond belief.

[JustABozoOnThisBus]

9. At most of the companies I worked at, all did regular backups...

... only one did testing of recovery.

One company had done "incremental backups" since forever, with many of the old backups on tape that could no longer be read. Would the company do a one-time "full backup" to create a baseline data recovery point? No, too expensive.

Generally, most of the companies could recover formal data bases (Oracle, SQL Server, IMS, etc), but the ordinary files were hit or miss.

Data management and security seem to always be afterthoughts.

[WillowTree]

13. The HIPAA implications for this hospital are enormous. The fines could be colossal.

[backscatter712]

18. The fines should be colossal. There's no excuse for this. n/t

[NobodyHere]

19. Blame the victim much?

[backscatter712]

21. Their IT department should have been prepared for this.

For one thing, they should have had proper backups of their important data, and had practiced backups and restores using their backup systems. Their backups didn't work. Failure number one.

For another thing, they should have had their important data on protected servers, preferably out of the reach of this sort of malware. They didn't. Failure number two.

For a third thing, they should have had a disaster recovery plan. They should have been able to wipe and reinstall a malware'd server or workstation, restore all its data, and had it back running in a matter of hours. The fact that this malware took their systems down for a full week shows their disaster recovery plans were absent or didn't work.

For a fourth thing, they should have been keeping their systems updated, to patch the security holes that this sort of malware exploits. They didn't do that.

Malware's out there. Everyone who works with computers for a living has seen it. They should have been prepared, but weren't.

If I were directing this hospital, I'd be looking very hard at firing the IT department head, and everyone else whose job it was to keep those computers working.

[NobodyHere]

22. Sounds like the prevention costs more than the cure in this case.

[backscatter712]

25. This is vital data. Confidential patient data, and if it gets fucked up, people die.

They should count their blessings that they were able to get it back after only paying $18 grand. It could have been much, much worse.

They might have fucked up so badly they wouldn't have been able to recover their data at all, or their data could have been corrupted.

It could have cost them millions. And it still might cost them - it's a huge PR black eye for them, and regulators are going to crawl up their ass and make them fix this shit before they can put it behind them.

It's a hospital, not a video-game startup. They need an IT department that keeps their systems locked down and secure. What I saw in this case is pretty horrible negligence on the part of the hospital. They have a responsibility to their patients, and let them down.

[hobbit709]

23. I blame the hospital for lousy computer security.

My security software warns me if anything even THINKS about trying to get into my system.

There's no excuse for stupidity.
ID ten T
PICNIC
PEBKAB

[NobodyHere]

24. How bout blaming the hackers?

[hobbit709]

26. Hackers wouldn't have been able to get in if there was decent security.

Do you blame thieves if you leave your front door wide open and lit up?

There's bad guys out there, why make it easy for them.

I do this for a living and I get this from people all the time "I have an antivirus program, how did it get in?"
"Because as soon as you clicked on the malware you gave it permission and it's now 5 minutes too late"

[RKP5637]

27. Someone on an earlier post said the backups were likely contaminated too if the

main server was being encrypted. To me, that seems to make sense? Is it a progressive encryption file by file with altered file names but still readable and then a command invokes all files encrypted?

[backscatter712]

28. It all depends...

If they were doing daily backups, or had redundant servers, chances are good that they could have nuked the malware, and restored the files from backup, and be good to go.

[backscatter712]

30. There's only so much you can do about the hackers.

They're laughing it up over shots of vodka somewhere in Russia. Ideally, you'd catch them and throw them in prison for this shit, but that's a tall order.

Practically, it falls on the IT people to take action to protect their systems against this shit.

That means running anti-virus/anti-malware software. It means firewalling the hospital network to block malware from doing its mischief. It means educating users to do things like not clicking on strange attachments. It means having redundant servers. It means doing at least daily backups, and practicing doing things like restoring corrupted or lost files when things get hosed.

A good IT department can keep their systems clean, and knows how to quickly unfuck a system when the worst happens. This hospital's IT dept. didn't.

[backscatter712]

29. I thought it was PEBKAC

Problem Exists Between Keyboard And Chair.

[dembotoz]

20. Perhaps they never really tested what they bought.

from what i understand the failure of backups is much greater than it could be

testing is good

[steve2470]

31. Ars Technica article and comments: Hospital pays $17k for ransomware crypto key

http://arstechnica.com/security/2016/02/hospital-pays-17k-for-ransomware-crypto-key/

Hollywood Presbyterian Medical Center, the Los Angeles hospital held hostage by crypto-ransomware, has opted to pay a ransom of 40 bitcoins—the equivalent of $17,000—to the group that locked down access to the hospital's electronic medical records system and other computer systems. The decision came 10 days after the hospital lost access to patient records.

"HPMC has restored its EMR on Monday, February 15th," President and CEO of Hollywood Presbyterian Medical Center Allen Stefanek wrote in a statement published by the hospital late Wednesday. "All clinical operations are utilizing the EMR system. All systems currently in use were cleared of the malware and thoroughly tested. We continue to work with our team of experts to understand more about this event."

The first signs of trouble at HPMC came on February 5, when hospital employees reported being unable to get onto the hospital's network. "Our IT department began an immediate investigation and determined we had been subject to a malware attack," Stefanek wrote. "The malware locked access to certain computer systems and prevented us from sharing communications electronically."

"Law enforcement was immediately notified. Computer experts immediately began assisting us in determining the outside source of the issue and bringing our systems back online," the statement said.

more at link

NO ACCESS TO PATIENT RECORDS FOR TEN DAYS ???!!!! If this CEO and IT director keep their jobs, it will be a miracle.