How To Recover a Network After a Ransomware Attack
I have devoted several articles to ransomware attacks. I wrote, what hackers can do in your network and how do they attack through IT providers. This time I will let you how to deal with the ransomware attack and how to restore the network to a usable form.☺ I would like to mainly share my observations and practical experience on how to deal with such a situation.
Imagine you are the administrator of a larger company (200+ employees) that has been infected with ransomware. This is not the case of ransomware arriving by e-mail and one of the employees launching it. Hackers have managed to steal login credentials, exploit an internet service, or place a backdoor into your network. (Emotet + Trickbot + Ryuk [CZ] ). They connected the network over the weekend, looked around for a while, and finally ran ransomware on several devices (these are large impact cases).
How To Detect Ransomware Infection?
„The “advantage” of ransomware attacks is that they are easily detectable.☺ Virtually anyone can detect them. Something will stop working as a result of the encrypted data. You can find out either from your monitoring system or from users who start calling. ☺
Why do I consider it an advantage? Espionage hacking is difficult to detect. It tends to be sophisticated, quiet and but a few companies have adequate experts and equipment for their detection. 2019 M-Trends report 54 days average (significant improvement compared to 2018 when 177 days were reported).
Where To Start When Ransomware Is Detected?
Most administrators would probably connect to the server and check whether the service isn´t “down”. Once they check the server, the administrator will find encrypted data (files with strange extension) and a ransom note (a message concerning who to contact to retrieve the data). Now you can be sure that your network has been attacked by ransomware.
Do you have a scenario for this case?
If yes, start following it and you don’t have much of a problem.
If not, the fun is yet to come. Questions will arise in your head … How could have it happened? … Were backups made yesterday? … Are backups encrypted? … What is infected? … What do I do?
In the meantime, your phones are ringing from users that they can’t use the system. Management joins in after a while. Everyone asks what is happening and how long will it take to fix it. Your stress levels will be so high, that your forehead will not be the only sweaty part.
Where to start in such a case? Meet the management, explain the situation, and tell them that the outage will last at least the whole day!
Why is it important? You´ll get more time and peace to solve the situation. Management, in turn, will know what the situation is and can try to adapt (send people home, start solving things by phone instead of email, pencil instead of IS). There is nothing worse than repeating that everything will work hour by hour (you are stressed, your workflow is not ideal and the management takes bad measures)!
Comment: Yes, the ransomware can be resolved in the order of tens of minutes/hours, but this applies to companies that are prepared. They have developed scenarios and train regularly.
Systematize
We are better off than at the beginning. We have gained time to solve the problem, the management is not pushing us with questions anymore and instead focuses on organizing the company. Now it is necessary to create a “rescue” team. For small companies (up to a few dozen PCs), you can probably do it by yourself. For larger companies, it will be necessary to involve your colleagues from the IT team. More people can get your business back and run sooner.
We have several tasks ahead of us:
- Detect the extent of infection – you need to identify everything you need to recover.
- How ransomware/hackers got into the network – so that ransomware/hackers would not return to the network within a few minutes.
- Restore network operations – the reason is most likely clear. ☺
Meanwhile:
- Tasks can be completed in parallel (although there are slight interconnections).
- Each task should have a leader (responsible person).
- Yet another person should be assigned only to manage the entire operation – team synchronization, making decisions and communicating with management and users.
- You should also consider whether to call in external experts (eg people from your suppliers). People who handle such “operations” regularly can reduce network recovery time and help avoid serious mistakes.
- If you manage a very large company, a full recovery can take several days. In such cases, work is usually done non-stop. Therefore, consider splitting into “shifts” right at the beginning. There is nothing worse than working 24 hours at a time (mental abilities being very limited) and having no one to replace you.
Comment: If you read my blog regularly, you know that IT security and hacking are more than just a job for me.☺ I will gladly join the operation to save your network, or at least advise. You can reach me either by e-mail (martin.haller@patron-it.cz) or through my colleague Martin Melich (+420 602 346 185). We can drive out anywhere in the Czech Republic within an hour if necessary.
How To Determine The Extent Of Infection?
Before you start connecting to all servers and systems, think about a few things:
- Are hackers offline? I came across the situation that the company was restoring data and hackers were encrypting other servers more than once. Therefore, disconnect the network from the internet!
- Disconnect the infected server from the network. The goal is to prevent it from encrypting other data that it has access to on the network.
- How are you going to connect to servers/services? If you plan to use a domain administrator account (DA), you run the risk of making things worse. Assuming the hackers didn’t get DA credentials, they could leave a “trap” on a compromised server. Once you connect to that server as a DA, the trap (malware) gains your authorization and can immediately spread throughout the network. How to resolve it:
- It is ideal to have a separate admin account for each server, which is the administrator of that given server only. Or to allow „RDP Restricted Admin mode“ (it has other disadvantages ☹). But both must have been set up before the attack.
- Disconnect the server from the network and log into it locally (via VMware/Hyper-V console/keyboard and mouse) with the help of cached credentials (malware cannot exploit them because the server is not on the network).
- Open the MMC with the “Local users & groups” snap-in on the trusted server and add a new local user with admin rights on the target server (you must do this through the FQDN due to Kerberos authentication). Then log in to the given server with a new user.
- Are backups infected? The first step is to make sure you didn’t lose your backups (so you know what you’re up to). Be especially careful during this operation to avoid endangering your backups (see above). Check the backups from a secure device and, if necessary, disconnect them from the rest of the network (eg if they are on a NAS).
What To Actually Look For On Servers?
That´s a good question. If you find encrypted data on the server and a ransom note outside the shared disk area, then the attackers got in. However, their absence does not mean that the server was not compromised/infected. In my opinion, it is ideal to combine the following:
- For servers that have been demonstrably compromised, look for the attacker activities. They usually don’t do much cleaning afterward. So you can find their scripts and the tools they used. Furthermore, the history of web browsers, Event logs (logged user accounts) remain, PreFetch information, recent documents, and sites, passwords/accounts stored in browsers and systems, caches, and logs from used software. This gives you a rough attack time-frame, a list of compromised accounts, and other servers/stations, the attackers could compromise.
- A random scan of all servers/stations: check the same things as above in the expected period (ransomware attacks I encountered have been performed as a “hit-and-run” so far – ie attackers do not stay in the network for a long time).
Where To Look Next
The search for compromised systems/devices will be significantly easier with:
- Central antivirus management: It will quickly show you which devices have detected malware. Even if AV does not capture the “cryptor” itself (software responsible for encrypting the data), it can still detect ransom note, data encryption (anti-ransomware modules), network attacks (firewall) and various “hacking” tools which hackers use to spread throughout the network.
- Network IPS/IDS: shows which devices the attacks were coming from. The devices from which the attacks came are compromised.
- Network analyzers/collectors provide an overview of what devices have communicated together, when, for how long, and what has the content. This is an incredible help in investigating security incidents.
Securing Leads
Don’t forget to write everything you find down in a structured way! As the number of information increases, it is easy to get lost in all of it. Take pictures of the important things as well. The notes/information will be useful for writing an incident report in the future.
If you need to reinstall or otherwise modify any computer/server, and it was a key device (in terms of clues and evidence), make its image (such as free FTK Imager). Likewise, before restarting a key server, dump its RAM for later analysis (such as FTK Imager, or DumpIT).
You can do a subsequent analysis of the disk image with Autopsy and RAM with Volatility. Both are free programs and it’s good to know them a bit.
How Did Ransomware Get Into The Network?
Most often (at least in my experience) ransomware enters the network via some vulnerable service published online. It would help to have a person amongst you who monitors what is happening in the field of IT security and keeps track of what vulnerabilities and exploits are “in”. Last year, these were RDP vulnerabilities (BlueKeep, DejaBlue [CZ]), RDP bruteforce, VPN implementations vulnerabilities (Pulse Secure Connect, FortiGate, Palo Alto GlobalProtect), Citrix ADC (CVE-2019-19781) and some vulnerabilities targeted at users in Adobe Flash, MS Office and web browsers (see „8 of the 10 Most Exploited Bugs Last Year Involved Microsoft Products“).
The information you found during the compromise identification phase will help you to correctly determine how ransomware has penetrated your network. Remember that the entry point may not be the first computer/server which has detected the ransomware. It could also happen that hackers gained access to your network X months earlier when you had an unpatched vulnerability present. They just placed a backdoor at that time or captured some login credentials. The attack/deployment of the ransomware itself was carried out with a few months’ delay (as I wrote in another article, recently there were more vulnerable networks than hackers were able to hack).
Once you know the attack vector, start to prepare for action. Implement them as soon as possible. Sooner then reconnecting the network to the Internet.😉
Resuming Operations
If you are lucky, only one server will be infected. It is very easy to restore from backups (assuming you have a backup of the entire server).
Unfortunately, in my experience, the entire network (Active Directory) is usually compromised. Hackers (currently even the ransomware itself) are very good at „lateral movement“ / „privilege escalation“ – ie. movement in the network and gaining access to everything (see „What Hackers Can Do In Your Network“).
In case they got as far as DC, the entire network has been compromised and you are faced with the decision of how to resolve it. Restore servers from backups and leave the stations? Restore servers from backups and reinstall workstations? Install everything from scratch? Let’s compare both options.
Install The Entire Network From Scratch
The solution recommended by experts. This choice is the safes and no one can object that you have neglected something.
A side benefit is that a clean installation will get rid of all possible old settings/programs/accounts (even backdoors) that you might have had in the environment and no one knew what they were for. ☺ You will have a better understanding of your environment and it will be safer than before the attack.
The huge time and financial demands are a disadvantage. You need to find a lot of installations. Search for people who are most likely retired and ask them what, how, and why has been set up. There is also a risk that transferring the old settings/data will include a backdoor as well, thus compromising a clean system. Theoretically, if the attackers were top tier professionals and cared for your network, the backdoor can remain, for example, in the firmware of a computer, printer, IP camera, IP phone, router.
In case you decide on a clean installation, you should ideally bring someone from IT security to help you build a resilient network from the start. It’s easier than securing things once all is done.
Restore The Infected Systems From Backups And Leave The Rest
This is a riskier and “more controversial” option. Some colleagues consider this variant to be a taboo (no-go). I believe it is a valid option. However, when choosing the right option, it is always necessary to consider the context of the whole situation. e.g. who was the attacker, how long did they operate in the network, where did they get, how critical the network is, how long the outage can last, what the restoration budget is, etc.
The controversial part of this method is that you never know whether hackers left a hidden backdoor (one or more) your network. Therefore, in case you do not reinstall everything (to restore credibility), you can not be sure that they will not return.
If you decide to pick this option (that you will restore/reinstall only a part of the network), I am sharing my list of steps to take (if you have any other ideas, I will be happy if you could share):
- Change all passwords of computers connected to AD (protection via Kerberos Silver Ticket and AD Sync).
- Change passwords for all users in AD and service accounts.
- Change Krbtgt account password 2x (protection via Kerberos Golden Ticket).
- Change passwords for all local accounts on computers.
- Change passwords for all other non-AD-connected systems if their database has been compromised (at least privileged accounts and critical systems).
- Change all passwords that have been stored on compromised systems.
- Change passwords for critical infrastructure (routers, backups, virtualization, etc.).
- Review all privileged groups in AD and GPO.
- Doublecheck your AD with BloodHound, it can reveal various methods of escalating privileges (GPO, ACL, computer sessions).
- Have all devices run a virus scan.
- Check the device through Autoruns (it can identify everything that starts automatically with the system).
- Review the FW rules on the perimeter and published services to the Internet.
- As soon as things start to calm down (the company starts to operate), it’s time to start looking at other places in AD. I can recommend the following lecture of Michael Grafnetter – Exploiting Windows Hello for Business or https://adsecurity.org/, where you will find a lot of quality information to fill in any lonely evenings.
It is also worth to introduce/strengthen monitoring/surveillance systems. The goal is to detect a possible subsequent attack or backdoor in time, which would be trying to “call” home. I wrote a little about it in my article „IT Security Life Cycle“.
What´s Next?
The investigation is over (you know how the ransomware got into the network as well as what was compromised) and the systems are restored? Now it remains to implement (proposed) defensive measures so that the attack does not repeat the same way and connect the network to the Internet once again.
The next step should be to analyze the logs from the routers/network analyzers and to try to find out whether the data leaked during the ransomware attack. Some of the groups behind the ransomware began to “extract” data from the compromised companies. Then the company is blackmailed for ransom. You should find out whether this is your case. It can also be guessed based on the used ransomware. Your management must prepare for this option (by choosing a strategy on how to respond to it) so that it can inform the concerned entities in time (suppliers, customers, customers, authorities).
Obtained information should serve as a basis to prepare a report, which you should be presented to the management. Management will ask why has this happened and whether you could have prevented it.
The last step is to work on increasing network security so that a similar incident would repeat in the future. At this point, you will have many ideas on how to increase the security of your corporate network. The management will be more open to listen to you and free up the budget for investments. If possible, recruit a security expert to make changes – his advice and knowledge will help you avoid missing something and implement the measures faster. ☺
Conclusion
I hope you´ve learned something interesting and at the same time, I wish that it remains just an informative article and that you never need it.☺ Please consider the recommendations and procedures as indicative – it always depends on the specific situation.
Did I miss something? Do you have a different opinion on the matter? Do you want to share your experience? I would be pleased to see your comments below the article.
Best wishes
Martin
Discussion