Kaseya VSA Attack

[MJ Shoer]

In light of the apparent attack on Kaseya this afternoon, we are opening this thread for members to discuss the attack and any impacts, remediations, etc.

Following are the posts in the Cyber Forum on the attack as of this writing:

Kaseya VSA, credit to @Matthew Lang

Kaseya VSA Ransomware Attacks July 2nd, 2021

Kaseya VSA Urgent Alert

Kaseya VSA Supply-Chain Ransomware Attack

Kaseya VSA supply chain ransomware attack, credit Silver Industry Partner Sophos.

What to expect when you’ve been hit with REvil ransomware, credit Silver Industry Partner Sophos.

Some additional links members have sent in:

Kaseya REvil Configuration Dump, credit @Ian Thornton-Trump CD
REvil ransomware gang executes supply chain attack via malicious Kaseya update, credit @Ian Thornton-Trump CD

Per @Chris Loehr, if you shut down a Kaseya VSA server, be certain to save all logs. This could be critically important.

Thanks to all who have been reaching out and sharing on this attack.

MJ

 

[MJ Shoer]

Latest update from Kaseya, as of 10 PM ET:

Update Regarding Ransomware Attack on VSA
July 2, 2021 - 10:00 PM EST
Beginning around mid-day (EST/US) on Friday July 2, 2021, Kaseya’s Incident Response team learned of a potential security incident involving our VSA software.
We took swift actions to protect our customers:

• Immediately shut down our SaaS servers as a precautionary measure, even though we had not received any reports of compromise from any SaaS or hosted customers;
• Immediately notified our on-premises customers via email, in-product notices, and phone to shut down their VSA servers to prevent them from being compromised.

We then followed our established incident response process to determine the scope of the incident and the extent that our customers were affected.

• We engaged our internal incident response team and leading industry experts in forensic investigations to help us determine the root cause of the issue;
• We notified law enforcement and government cybersecurity agencies, including the FBI and CISA.

While our early indicators suggested that only a very small number of on-premises customers were affected, we took a conservative approach in shutting down the SaaS servers to ensure we protected our more than 36,000 customers to the best of our ability. We have received positive feedback from our customers on our rapid and proactive response.
While our investigation is ongoing, to date we believe that:

• Our SaaS customers were never at-risk. We expect to restore service to those customers once we have confirmed that they are not at risk, which we expect will be within the next 24 hours;
• Only a very small percentage of our customers were affected – currently estimated at fewer than 40 worldwide.

We believe that we have identified the source of the vulnerability and are preparing a patch to mitigate it for our on-premises customers that will be tested thoroughly. We will release that patch as quickly as possible to get our customers back up and running.
I am proud to report that our team had a plan in place to jump into action and executed that plan perfectly today. We’ve heard from the vast majority of our customers that they experienced no issues at all, and I am grateful to our internal teams, outside experts, and industry partners who worked alongside of us to quickly bring this to a successful outcome.
Today’s actions are a testament to Kaseya’s unwavering commitment to put our customers first and provide the highest level of support for our products.
Fred Voccola, CEO
Kaseya

 

[MJ Shoer]

All,

We have a member of the CompTIA ISAO who has been hit hard by the Kaseya attack. He is a small MSP and longtime CompTIA member. It would be wonderful if anyone is able to lend this MSP any help at all, even if simply moral and emotional support as he works to recover from this.

Here is the rough situation:

All clients hit.
40-60 servers impacted.
400-500 PCs impacted.
Backup recovery keys are stored on a compromised server.

As he said to me this afternoon, lots of lessons learned and lots of things he is documenting to improve upon to try to prevent a recurrence of something like this.

Right now, he is most importantly working to put together a response plan as his plans are on compromised servers and inaccessible. If any member can help him get a plan in place and execute on it to aid his recovery efforts, he will be very grateful for the support.

Please reach out to me directly if you'd like to be put in contact to help. Thank you!

MJ

 

[Jason Slagle]

What do they need? I can offer compute, moral support, forensics etc. I'm a bit tied up until tomorrow with family stuff but will have some cycles I can throw around tomorrow.

 

[Mark Mitchell]

Hi MJ,

Happy for my details to be passed onto the MSP, appreciate the time difference may make my input a little difficult/delayed but I can be used for advice/sounding board if required during response/recovery and thereafter.

Cheers
Mark

 

[Mike Semel]

To all involved in this incident and others that are sure to come,

THIS IS WHY YOU HAVE INSURANCE AND AN ATTORNEY. Your clients are likely to come after you for damages based on advice from their insurance and lawyers.

For the past few years I have co-spoken at cybersecurity conferences with an attorney that specializes in data breaches and compliance violations. He has taught me a lot. Communications involving attorneys may be protected by attorney-client privilege and not accessible through subpoenas. Without an attorney, everything you and your employees say or write (including internal and external emails, Facebook groups, and forums like this) may be discoverable by subpoena and become evidence against you in a lawsuit.

1. Of course you want to help your clients, but you need to be very careful about what you say and do, because your best customers may now be your biggest adversaries. Your attorney can help craft language that will not come back against you.
2. Your Errors & Omissions and Cyber Liability insurance may be able to cover the costs of an attorney that specializes in breaches and compliance violations (ransomware is a reportable HIPAA violation, reportable under the New York SHIELD Act, and other laws). Insurance may also cover additional costs to remediate the problems.
3. Even if you don't have insurance that will pay for a lawyer, now is the time you must quickly engage with one. Your business and assets depend on it. Remember, if you own your business and it fails, your employees will have new jobs tomorrow and you will be the one left with the bills, bank loans, tarnished reputation, and perhaps bankruptcy.
4. This is a good wake-up call for those that haven't taken HIPAA and other regulations seriously,. Any investigation and lawsuit will include relevant compliance requirements, so if any healthcare clients, financial services, defense contractors, or other regulated clients have been affected, your compliance will be inspected. First, your insurance company may investigate you to ensure you qualify for them to cover your claim - which could cost millions of dollars if they refuse. Then a lawsuit will demand all compliance-related documentation. Even if you say you are compliant, can you prove it with documentation that will stand up in court?
5. Check out my article MSP SUED! at ChannelPro.

I hope this helps.

Mike Semel

 

[Paul Blough]

They have to start by reaching out to their Cyber Liability Insurance provider. They will dictate what any of us may be able to do to assist, if anything. If they don’t have cyber liability insurance that will be a difference scenario. As per Mike’s comments above, they need to get a qualified attorney engaged as well. We all want to help but if we are not careful we may end up causing more trouble for them or giving them bad advice. A quick example, we got called in as the third IT responder when a local doctor’s office got ransomed. The first IT company formatted the infected servers before they checked the backups which were also ransomed. The second company was brought in after that but it was beyond their expertise and 5 days later we were brought in. At my first meeting I asked them if they had contacted their cyber insurance provider and the doctors office said they didn’t have any. I know that was actually unlikely since it’s wrote into their malpractice insurance often now. I challenged them to double check and they did have insurance after all. The insurance company wanted to pay the ransom to get the data back but we couldn’t because the servers had been formatted and were unrecoverable after that. The backups were also unrecoverable. The doctor ended up losing $600,000 in medical billing that he no longer had documentation to back up. Moral of the story is get the right advice before you do anything.

 

[MJ Shoer]

Thank you all for your offers and your insights. The MSP in question is able to see these posts and is getting this input. We have also had dozens of MSPs and vendors reach out to offer assistance. I am so pleased and proud of the CompTIA ISAO for their response to this attack and an MSP who has been impacted. We're all in this together and this proves it. I appreciate all of your support and offers of help.

MJ

 

[Raffi Jamgotchian]

To all involved in this incident and others that are sure to come,

THIS IS WHY YOU HAVE INSURANCE AND AN ATTORNEY. Your clients are likely to come after you for damages based on advice from their insurance and lawyers.

For the past few years I have co-spoken at cybersecurity conferences with an attorney that specializes in data breaches and compliance violations. He has taught me a lot. Communications involving attorneys may be protected by attorney-client privilege and not accessible through subpoenas. Without an attorney, everything you and your employees say or write (including internal and external emails, Facebook groups, and forums like this) may be discoverable by subpoena and become evidence against you in a lawsuit.

1. Of course you want to help your clients, but you need to be very careful about what you say and do, because your best customers may now be your biggest adversaries. Your attorney can help craft language that will not come back against you.
2. Your Errors & Omissions and Cyber Liability insurance may be able to cover the costs of an attorney that specializes in breaches and compliance violations (ransomware is a reportable HIPAA violation, reportable under the New York SHIELD Act, and other laws). Insurance may also cover additional costs to remediate the problems.
3. Even if you don't have insurance that will pay for a lawyer, now is the time you must quickly engage with one. Your business and assets depend on it. Remember, if you own your business and it fails, your employees will have new jobs tomorrow and you will be the one left with the bills, bank loans, tarnished reputation, and perhaps bankruptcy.
4. This is a good wake-up call for those that haven't taken HIPAA and other regulations seriously,. Any investigation and lawsuit will include relevant compliance requirements, so if any healthcare clients, financial services, defense contractors, or other regulated clients have been affected, your compliance will be inspected. First, your insurance company may investigate you to ensure you qualify for them to cover your claim - which could cost millions of dollars if they refuse. Then a lawsuit will demand all compliance-related documentation. Even if you say you are compliant, can you prove it with documentation that will stand up in court?
5. Check out my article MSP SUED! at ChannelPro.

I hope this helps.

Mike Semel

Mike,

Curious on your thoughts... many of us use the same few insurance agents and carriers, and even attorneys. Are they potentially going to be overwhelmed in this sort of situation?

 

[Dave Alton]

MJ,

Please let me know what I can do. All of this "advice" can be extremely overwhelming in the moment. It is easy to just be so frustrated that you throw your hands up. The folks here are offering extremely sound advice and it should be heeded, but like with any project you have to break it down into manageable chunks. You won't be able to do everything all at once. Listen to the advice of others. Reach out and tap those that are experts in what you need, then make a plan and work the plan.

Rely on your team and community to help you to the best of their ability but remember you have to do what is right for you. Take your time and work the problem and break it down.

Don't get stuck on a little thing that doesn't provide value.

Hope that helps and if there is something I can do to help please reach out.

Thanks!

Dave

 

[Dave Alton]

To all involved in this incident and others that are sure to come,

THIS IS WHY YOU HAVE INSURANCE AND AN ATTORNEY. Your clients are likely to come after you for damages based on advice from their insurance and lawyers.

For the past few years I have co-spoken at cybersecurity conferences with an attorney that specializes in data breaches and compliance violations. He has taught me a lot. Communications involving attorneys may be protected by attorney-client privilege and not accessible through subpoenas. Without an attorney, everything you and your employees say or write (including internal and external emails, Facebook groups, and forums like this) may be discoverable by subpoena and become evidence against you in a lawsuit.

1. Of course you want to help your clients, but you need to be very careful about what you say and do, because your best customers may now be your biggest adversaries. Your attorney can help craft language that will not come back against you.
2. Your Errors & Omissions and Cyber Liability insurance may be able to cover the costs of an attorney that specializes in breaches and compliance violations (ransomware is a reportable HIPAA violation, reportable under the New York SHIELD Act, and other laws). Insurance may also cover additional costs to remediate the problems.
3. Even if you don't have insurance that will pay for a lawyer, now is the time you must quickly engage with one. Your business and assets depend on it. Remember, if you own your business and it fails, your employees will have new jobs tomorrow and you will be the one left with the bills, bank loans, tarnished reputation, and perhaps bankruptcy.
4. This is a good wake-up call for those that haven't taken HIPAA and other regulations seriously,. Any investigation and lawsuit will include relevant compliance requirements, so if any healthcare clients, financial services, defense contractors, or other regulated clients have been affected, your compliance will be inspected. First, your insurance company may investigate you to ensure you qualify for them to cover your claim - which could cost millions of dollars if they refuse. Then a lawsuit will demand all compliance-related documentation. Even if you say you are compliant, can you prove it with documentation that will stand up in court?
5. Check out my article MSP SUED! at ChannelPro.

I hope this helps.

Mike Semel

Mike,

I appreciate your helping here. I was hoping you could shed some light on the RFT platform? I heard that there may be an exposure path for those of us that run Network Detective or Compliance Manager. Do you have any insight as whether this is true or not. I have reached out to support but have not heard back. I know everyone is busy but I wanted to make sure I do not have any exposure.

Thanks,

Dave

 

[Susan Kostbar]

As a non-technical CompTIA staff member in Corporate and CompTIA ISAO membership, a big THANK YOU, for sharing guidance and advice to this forum. As a reminder, CompTIA announced a member benefit with competitive rates for business and cyber insurance with GCG Financial and The Hartford. Go to TheHartford.com/CompTIA or call 855-658-0528.

 

[MJ Shoer]

An important update from the FBI and CISA:

CISA-FBI Guidance for MSPs and their Customers Affected by the Kaseya VSA Supply-Chain Ransomware Attack

Also, Huntress Labs has one of the most thorough and up-to-date blogs on the attack, with analysis from their threat hunting team. Huntress is also "hosting a live webinar on Tuesday, July 6 at 1pm ET to provide additional information. Led by Huntress founders and members of our ThreatOps team, we'll share our latest technical analysis, answer your questions and offer guidance to help businesses respond to these large-scale ransomware attacks. You can click here to register."

I recommend that any member impacted (we still only know of a single member hit) or any member concerned about supply chain attacks against MSP tool vendors to register and attend this webinar.

 

[MJ Shoer]

SophosLabs Uncut just sent over the most detailed analysis of this attack that I have seen so far.

Please review the post:

Independence Day: REvil uses supply chain exploit to attack hundreds of businesses

From the Lessons Learned section of the post:

The tactics to evade malware protection used here—poisoning a supply-chain well, taking advantage of vendor carve-outs from malware protection, and side-loading with an otherwise benign (and Microsoft-signed) process—are all very sophisticated. They also show the potential risks of excluding anti-malware protection from folders where automated tasks write and execute new files. While zero-day supply-chain exploits are rare, we’ve already seen two major systems management platforms exploited in the past year. While Sunburst was apparently a state-funded attack, ransomware operators clearly have the resources to continue to acquire additional exploits.

 

[Scott Barlow]

Great infographic if anyone is interested in sharing the differences between the recent attacks on SolarWinds, Microsoft and Kaseya.

 

[MJ Shoer]

Kaseya Ransomware Attack: Guidance for Affected MSPs and their Customers

Newly released guidance from CISA for impacted companies.

 

[MJ Shoer]

Breaking news from the New York Times:

Russia’s most aggressive ransomware group disappeared. It’s unclear who disabled them.

Key comments by reporter David E. Sanger:

"Just days after President Biden called President Vladimir V. Putin of Russia and demanded that he act to shut down ransomware groups that are attacking American targets, the biggest of them has gone off-line. The mystery is who made that happen."

"...around 1 a.m. on Tuesday, when the group’s sites on the dark web suddenly disappeared."

"There were three main theories floating around about why REvil, which seemed to revel in the publicity and reaped huge ransoms — including $11 million from JBS — suddenly disappeared.

One is that Mr. Biden ordered the United States Cyber Command, working with domestic law enforcement agencies, including the F.B.I., to bring it down. Cyber Command proved last year that it could do just that, paralyzing a ransomware group that it feared might turn its skills to freezing up voter registrations or other election data in the 2020 election.

The second theory is that Mr. Putin ordered the group taken down by Russia. If so, that would be a gesture toward heeding Mr. Biden’s warning, which he offered, in more general terms, when the two leaders met June 16 in Geneva.

And a third is that REvil decided that the heat was too intense, and took itself down to avoid becoming part of the crossfire between the American and Russian presidents. That is what another Russian-based group, Darkside, did after the ransomware attack on Colonial Pipeline, the U.S. company that had to shut down the gasoline and jet fuel running up the East Coast in May."

This will be a very interesting story to follow.

 

[MJ Shoer]

Interesting blog post from the TrueSec blog on how the Kaseya attack was executed:

How the Kaseya VSA Zero Day Exploit Worked