Attack on IT companies: Kaseya VSA
I was petrified on Friday evening, July 2 2021! I was just enjoying my last days of hybrid (the politically correct name for work and maybe fun) holiday. When I started to read the paper about hundreds of attacked companies through monitoring (RMM) system of Kaseya company. Attackers again abused our tools against us, administrators!
To be clear, we need proper instruments to provide effective help (with service, security, and automatization). Tools that allow us to watch and control all environments centrally. Such a tool can keep the order and safe time. You can make a configuration change in one place, and it can spread over thousands of computers thanks to that tool.
The best advantage of the central system, however, can be as well the weakest spot. When the system is overtaken by somebody with bad intentions, he can damage data on all computers that the system controls. And that’s what happened on Friday, July 2.
The attack against Kaseya VSA
Company Kaseya develops one of these central tools for management and monitoring (RMM – Remote Management and Monitoring) named Kaseya VSA. This tool exists in the form of cloud service as well as “on-prem” installation (i.e., a structure directly at MSP/MSSP)
Several critical errors were found in this tool shown by the Dutch Institute for Vulnerability Disclosure to the producer (Kaseya). They immediately started to fix them. Unfortunately, also cyber hackers (REvil/Sodinokibi) noticed the bugs and used them first (July 2, 2021) before any corrections could be made (July 11).
As a result, 60 IT companies were attacked, and more than 1.500 of their clients (Kaseya ransomware attack: What we know now) were encrypted. So far, about the “boring” explanation, let’s look at the exciting part of this attack.
Nightmare of IT companies
This type of “trouble” must be a nightmare for any IT company. On Friday afternoon, you end up working with a vision of an extended weekend, and a few hours later, all your customers are encrypted.
This is not your fault – you have had the software updated, you use strong passwords and multi-factor authentication, and yet all customers are encrypted.
All customers! That’s probably the worst thing about it. You could get one or two customers back on their feet in a week, but all customers? It will be for months. It’s an incredible asymmetry between how much energy is needed to cause damage and how much to repair.
From the customer’s point of view, it also doesn’t look good. He is not responsible for the situation at all. And his only option is to be bigger trouble for you than the others and get a higher priority for system recovery. Or ask for help from the competition and pay.💰
Every company has its “own” Kaseya VSA
We, fortunately, do not use the Kaseya products. However, we have other similar critical systems:
- N-able RMM – similar SW as Kaseya VSA provided in the form of the cloud. Earlier, the name was SolarWinds MSP RMM, but after the incident last year with other SW (i.e., “Hack of SolarWinds, FireEye and US Treasury Department [CZ only]”), the spinoff was created (separation to new company) (see “Video of monitoring system: how we monitor over 2000+ net devices”)
- ESET Protect – central console for product management (especially antivirus) from ESET. It can also control all connected devices (computers, servers, mobile phones).
- TeamViewer – software allowing the remote support of users (i.e., „TeamViewer: the experience after 4 years of using (part 2/2)“). It is possible to connect and control all paired devices.
- Devolutions Server – software for controlling all passwords and accessions to our customers (i.e., “Passwords management 2: we set a new system for password management”).
- Azure Active Directory – so far, we are not entirely “integrated” (for security reasons), but it will come in the future. Access to primary e-mail accounts can be critical, mainly due to the ability to reset the password, bypass multi-factor authentication, or control of managed PCs through Intune.
It is necessary to face the problem
I wondered if I would even write about the fact that we have similar critical systems to not scare our customers unnecessarily. However, the truth is that every company has its critical systems. That’s just the way it is. Without these systems, it would not be possible to do our job correctly.
The aim must be to face these challenges. Understand which systems are critical, what can happen when abused, and start to manage those risks. We should:
- Limit the number of critical systems – what I don’t necessarily need, don’t use.
- Minimize the criticality of the given systems – manage with the given system only what I need (e.g., only stations) and unused modules turn off.
- Divide systems by criticality and apply appropriate security measures – access measures (no or lowest possible access permissions), network availability (firewall, VPN, geo-filtering), 2FA, EDR, AV+sanbdox (i.e., “Network security: Tierig and PAW”).
- Choose your partners/suppliers carefully – they have a greater effect on our security than we often admit.
What was the successful part of the attack
This attack was not as sophisticated as against SolarWinds (“Hacking SolarWinds, FireEye, and the US Treasury Department [CZ only]”). However, it must be considered that the hacking against SolarWinds was carried out by a state-supported elite hacking group. On the contrary, this attack was carried out by an “ordinary” cybercriminal group.
Unlike other attacks by similar groups, this is a very sophisticated attack. The attackers used a publicly unknown vulnerability (maybe their own research?) and their own exploit. Usually commonly exploit bugs found by others are used by attackers (such as security researchers) and are often unable to write their own exploit (waiting for someone else to write it and then start abusing it).
Technical design of payload (malware)
I must say that the attackers took care of the payload (the way ransomware is deployed). When they attack a company manually, they can scan its environment and disable its antivirus protection (disable, uninstall, set exceptions).
However, when attacking automatically, all instructions (logic/behavior) must be built directly into the deployed malware. But how to do it if each victim has a different antivirus and its settings?
The initial phase of the payload (malware) used by the attackers was deployed through a monitoring system agent. However, it is often included among the exceptions of the antivirus system (so that the monitoring system does not break). At this stage, an old version of Windows Defender and code that allows data encryption were downloaded to the system.
Now you’re probably wondering why attackers would download antivirus into the system; it doesn’t make sense.😁 The point is that the old version of Defender, which they downloaded to the system and “installed”, contained a security flaw (they performed the so-called “Bring Your Own Vulnerability”).
Afterwards, this old Defender was launched. Due to a security flaw, it loaded the supplied data encryption code (DLL) and began encrypting all data.
The goal was to bypass the antivirus protection of the PC/server again. Antiviruses usually have an exception for Defender to avoid system instability due to multiple antiviruses. From the information of my acquaintances, I have confirmed that the ransomware has not been stopped by their AV.
It is also possible (I estimate) that Defender also has access to “protected folders” and could also encrypt this protected data.
Timing of attack
It is also worth mentioning the timing of the attack. The attackers launched the attack on Friday evening (July 2, 2021). At the same time, it was not a typical Friday but the Friday before the extended weekend when several US states postponed public holidays on Monday, July 5 – “Independence Day”. Here in the Czech Republic, Monday and Tuesday were also national holidays. Therefore, many people are on their way to vacation, cottage, family, etc. It is, therefore, more challenging to organize an “incident response”, and attackers have more time to attack.
What could attackers do better
Although the attack caused significant damage (over 1.500 companies were attacked), it could have been even worse. The 1.500 companies attacked came from about 60 hacked Kaseya VSA on-prem servers (central servers for customer management). While at the time of the attack, there were about 2.200 Kaseya VSA servers on the Internet. The attack could theoretically be 25 times more giant.
The question is why not all servers were attacked.🤔I think the attackers would have managed to do this before most of Kaseya’s VSA servers could be disconnected from the Internet. The attackers had the attack automated (I estimate) – i.e. it could be “done” in order of minutes. In an article describing the exploitation of “How the Kaseya VSA Zero Day Exploit Worked”, the author speculates that this may have been due to a “limitation” of the security flaw/exploit.
Backups usually survived
Backups of the victims usually survived so they could restore their data. That means the victims did not have such a need to pay a ransom.
Automated attacks (like this one) are not as creative in destroying backups as human operators (attackers). This is because each company has different backups (software, location, timing, protection) and programming a script (or artificial intelligence) that would cover most scenarios is laborious.
They were not stealing data from attacked companies
For common ransomware attacks, the rule is that attackers download copies of some data from the victims. Subsequently, the victims blackmail that if they do not pay, they will publish the data. However, they did not do it here, so they lost part of the “leverage” on the victims.
Of course, such a significant attack would require more infrastructure and preparation while they didn’t have much time. However, it would certainly not be unrealistic.
The affected companies didn’t have a backdoor
From an analysis of the used malware/ransomware, it appears (so far) (see “Independence Day: REvil uses supply chain exploit to attack hundreds of businesses”) that attackers did not make a backdoor or pump passwords on the affected networks. The victims thus needed “just” to restore data from backups and remove used malware from computers.
In the case of a normal recovery after a human-operated ransomware attack, there is a risk (unless the entire network is installed cleanly) that some attacker´s backdoor will be overlooked. They then return to the network and encrypt everything again.
How the Kaseya (producer) reacted
I do not envy the producers or his employees for this situation, and I hope not to experience something like this. During such an incident, there is stress, exhaustion, and remorse everywhere.
The speed of communication of the incident to customers
I really like that Kaseya managed to communicate this problem to his customers very quickly. It can be seen from the graph above that on July 3(approximately 24 hours after detecting attacks), most (about 90%) of the on-prem Kaseya VSA servers were shut down. This reduced the number of targets available to attackers and reduced the impact.
Information sharing
I also like how much information about the attack was shared by Kaseya (see https://helpdesk.kaseya.com/hc/en-gb/articles/4403440684689-Important-Notice-July-14th-2021).
Statement by the CEO
It is good that the CEO of the company also commented on the situation. However, I did not like the content of the message so much. It does not seem appropriate to boast that only “0.01%” of the customer base was affected. In my opinion, it was just luck, not their credit, that the attack was not significantly worse. At the same time, 0.01% means 60 SME partners and 1.500 companies that trusted Kaseya. Meanwhile, Kaseya had clumsy safety mistakes in their SW and, due to their laziness (see below), they caused severe problems to their partners.
How long does it take to release the actualizations
Kaseya was informed about the found vulnerabilities as early as the beginning of April (April 2, 2021 CVE were assigned). Nevertheless, even after three months, they did not correct all (the attacks were on July 2, 2021), only some of the bugs were corrected during this time, even though they were critical bugs.
Interestingly, only customers using on-prem servers were attacked during the attacks. I don’t know if cloud servers (SaaS) were more protected or just attackers didn’t attack them.
However, from the attack detection (July 2) until the release of updates (July 11), both cloud servers and on-prem servers were switched off. That means all customers’ networks (approximately 37.000 SMEs and 800.000 – 1.000.000 end companies) were unattended during this time.😕
Patches and cloud services were supposed to be available sooner according to the manufacturer’s promises, but gradually there were delays and various technical problems. Fortunately, currently (July 15), it looks like the patches have worked, and everything is already working.
A general approach to security at Kaseya
Even though Kaseya claims that security and data protection come first (who doesn’t say that, right?😅), articles and “testimonials” on the Internet claim that security at Kaseya wasn’t exactly a hit. For example, Marcus Murray says, “We found many different categories of exploitable vulnerabilities in the Kaseya VSA product that indicates a lack of understanding when it comes to basic security principles insoftware development” (see “Kaseya Failed to Address Security Before Hack, Ex-Employees Say”).
Brian Krebs then writes on his blog (“Kaseya Left Customer Portal Vulnerable to 2015 Flaw in its Own Software”) that Kaseya had a critical error from 2015 in her customer portal (it was said to be an old but still functional portal). The situation is even sadder as it is their own software.🤦♂️
Again, it turns out that a big company does not automatically mean a safe company.🙄
Possible twilight for ransomware?
Lately, the attackers were so clever. A ransomware attack on the American company Colonial Pipeline [CZ only] caused a fuel failure on the east coast of the United States. An attack on Irish health services [CZ only] caused health service failure across Ireland. And now the attack on Kaseya VSA, which has a global scope.
These attacks have seriously affected ordinary people and result in the political will to start tackling the problem. US President Biden reportedly had a conversation with Russian President Putin on cybercrime. He was to “ask” him to begin tackling cybercrime on its territory. A few days later (July 13), the websites of the cybercrime “organization” REvil, which is also responsible for the attack on Kaseya VSA, stopped operating (in the TOR network/darkweb) (“REvil: Ransomware gang websites disappear from internet”).
It seems to me that the ransomware groups have started to make serious problems due to these big attacks. Due to international political pressure, there will be fewer countries where they will be safe from justice. We’ll see how they approach it.
Some cybercrime groups are probably aware of this and have begun to ask their “franchisees” for information on which company they are going to attack and must approve it before the attack (“DarkSide ransomware will now vet targets after pipeline cyberattack“)
Conclusion
Uff, that was a lot of information again. 🤯I hope you enjoyed the article and believe it was helpful. If you have any information on the topic, I will be happy if you let me know in the comments or e-mail.
I wish you a peaceful summer and vacation. And keep your networks safe.
Martin
(July 25) PS: Kaseya company has acquired a universal decryptor that allows you to decrypt all data encrypted in this attack. The attackers originally wanted $70M for this decryptor. So far, we can only speculate whether any amount has been paid or whether the decryptor was obtained by political pressure. (source: https://www.bleepingcomputer.com/news/security/kaseya-obtains-universal-decryptor-for-revil-ransomware-victims/ ) . According to the attackers, they managed to encrypt over million devices, which is a significant number.🤯
Discussion