This year, we visited dozens of companies and discussed the security of their IT systems with them. These visits (security audit) were usually initiated by the management (they wanted assurance that IT is taking good care of the network), IT department (they wanted an additional pair of eyes), or hackers.đ
They were nice visits. We met great people, saw beautiful places, and expanded our overview of the state of IT in companies. I am writing about it in the past tense as I would like to reduce this activity for now. It is very time-consuming, and as an introvert, it takes me slightly out of my comfort zone.
Now I need to focus on the backlogs we have in our company. Complete the selection of suitable EDR (see my article âWhy I want to deploy EDR to all customersâ), implement central log management and checks for Office 365 tenants (within our monitoring system, about which I have already written, Monitoring system: what we can achieve with it). And lastly, writing final reports didnât interest me at all.đ´
I often mention that I see the way forward in IT security in mutual help and sharing information among ourselves. Therefore, I would like to share our insights and frequently occurring mistakes we encountered during audits. Hopefully, it brings you value.
Recording from Cybercon 2022
In September, I was invited as a speaker to CyberCon 2022 by NĂKIB (I appreciate it and thank them very much). My lecture was precisely about our experiences from audits. Those who prefer watching video to reading text can watch the recording of the lecture (the content is similar) (recording is czech language only âšď¸).
What Surprised Me the Most during Audits
We divided the audits into two parts. Theoretical and practical. In the morning, we discussed the networkâs form, measures, security, and settings theoretically. In the afternoon, we sat down to a computer, and everything discussed before was shown to us.
Even last year, I wouldnât have thought how different these two things could be. Itâs basically the rule that the actual network settings differ from the described state. Why is that? I have no idea.đ¤ˇââď¸ Nor do I feel that IT wanted to lie to us or give an inaccurate description deliberately. Maybe there is so much work and constant changes that itâs not easy to remember everything.
I think being a technician is a significant advantage during audits. I can look at the network settings and read their actual form. During cybersecurity incidents, itâs not about how we think the network is secured but how it is actually secured.
What We Focused on during Audits
We can check a lot during an audit and then ponder over each setting. However, we didnât want to get into arguments about whether antivirus X or Y is better and whether it is necessary to have it set in aggressive mode, or normal sensitivity is sufficient. Many of these things depend on context, and the right solution can be discussed.
When creating the audit outline, we tried to bring as much value as possible in the least amount of time (and money). The list of things that can be checked is quite long. However:
- there isnât time to check everything,
- each item has a different probability of exploitation,
- and the potential damage from exploitation varies from item to item.
Therefore, we decided to focus on things that we know are exploited by attackers. Audits are another area where we benefit from our experience in dealing with cyber incidents (ransomware, compromised systems). I have written about our experiences on the blog several times. The last time I think in the article: âInsights from Interventions during a Ransomware Attackâ.
Problems We Frequently Encountered
And now finally, the problematic areas we encountered the most.
Backups
Yes, I know, itâs a dull topic that has been discussed for years and years. However, even though it is a dull topic, backups are crucial and need to be in top condition. Look at it this way: any additional security measures we take are to prevent compromise. However, networks are extensive and complex. Everyone adds that 100% security does not exist. So, even if you take 90% of other measures but attackers still get through, backups are the last and only measure that will save you. So, even though they are boring, spend time on them and make sure they are well done.
We encounter the following problems with backups:
- They are not resistant to hackers. If attackers take over the network, they can delete all backups, especially quickly if the backup server is a member of the production AD. In such a case, attackers immediately control both production data and backups.
- They do not contain all data. The amount of production data is still growing until there is nowhere to back it up. Then, only some servers begin to be backed up. Some data is found missing in the backups only when you want to restore from them because the production is encrypted (100% stressful situation đ¤Ż).
- Insufficient recovery speed. Compared to the previous points, this is just cosmetic. However, it is good to know, when the management asks when the company will be operational again, that restoring all the data will take 4 days. It is ideal to attempt a complete recovery. In a worse case, restore at least one medium server and calculate the rest (server size [TB] / recovery time [h] * total size of backed-up data [TB] = estimated data recovery time in hours).
Missing Antivirus on Servers
Often servers did not have antivirus installed. Administrators either did not feel the need to have it there (only they access the server), or they had some performance or functional problems in the past.
Antivirus (or rather EDR) with central management serves two purposes:
- It makes the job difficult for attackers. It can sometimes stop them, but mostly it just delays them, giving us (the defenders) time to react.
- It alerts us when something is happening somewhere. This is perhaps the more important function. Although it doesnât stop attackers, it gives us information that something is happening somewhere, and we have the opportunity to investigate the situation. We thus have the opportunity to stop the attack before the damages multiply.
Recently, we have been dealing with several attacked companies where attackers had been present on the servers for over a year without the defenders noticing. If there had been at least some antivirus on the servers, they would have noticed it (the attackers used known malware/backdoors).
Of course, instead of antivirus, you can monitor servers using other technology. Itâs just about having some mechanism to recognize that something illegitimate is happening on the server. I mention antivirus because, in my opinion, it is the cheapest and simplest way to achieve this.
It is also important to mention that the antivirus should be connected to central management, which someone monitors and evaluates the outputs. At Cyber Days 2022, I gave a whole lecture on this topic (itâs more complicated than it seems at first glance).
Outdated Operating Systems
As is written everywhere, it is necessary to update the operating system and applications. This requirement can be beautifully and simply expressed in one sentence. However, administrators know that its fulfillment is significantly harder (the larger the network they manage).
I am a realist, and it doesnât surprise me that many companies operate unsupported operating systems (Windows 7, Windows Server 200-2008 â I have only once seen a customer with a paid ESU program) and update servers once a year. In most cases, usually, only the given server or workstation is at risk.
However, it is necessary to realize that some servers have a significant position in the network. Their compromise usually means âgame overâ because attackers through them can control the whole network. These servers include domain controllers, backup servers, MS Exchange servers, AD FS servers, AD CS, servers with AD Connect, and any other server where a stored account with Domain Admin rights can be found. If you, therefore, chronically fail to update everything, the mentioned servers are a critical minimum.
Defending a domain where there is some Windows Server 2008 (R2) controller is indeed a challenge. I would mention, for example, CVE-2020-1472 (Zerologon), CVE-2021-42278+CVE-2021-42287 (sAMAccountName Spoofing), and CVE-2021â34527 (PrintNightmare). All are RCE, actively exploited, and without a free patch for the said OS đ¤ˇââď¸.
The âAdministratorâ Account Everywhere You Look
This involves several problems at once, but I donât feel like writing a separate chapter for each. đ Letâs look at them:
- Local (enabled) administrator account on servers and stations with the same password. In case of compromise of a station/server, attackers almost always extract the password hash of the local user âAdministratorâ. This can be used to control any station/server (with Windows), where this user is enabled and has the same password (âMITRE: T1550.002 Use Alternate Authentication Material: Pass the Hashâ). The solution is to deploy LAPS, disable local users, or enforce UAC for built-in admins with RemoteUAC.
- Management of stations and servers by a domain administrator. Administrators often use one user account, the domain administrator, to manage the entire network. The problem is, if they log in with it to a compromised station/server, attackers can steal it, and the entire network is compromised. This is perhaps the most common reason why hacking an ordinary server in AD (Active Directory) results in the compromise of the entire AD. Attackers find a logged-in domain administrator on the compromised server, steal their identity, and control the entire network. I have already written about this issue in the article âNetwork Security: Tiering and PAW.â
- Too many privileged users. While browsing AD, we also look at how many privileged users (user accounts with excessively high permissions) exist. A mistake/compromise of each such user results in significant damage. If these are accounts used to operate some application, they extend the list of âcritical serversâ (as I mentioned above). This user account is stored on the server (where the application runs), and if someone compromises the server, they control the entire network again.
The goal is thus to look at the members of the following groups: Administrators, Domain Admins, Enterprise Admins, Schema Admins, Account Operators, Backup Operators, DnsAdmins, Exchange Windows Permissions, Organization Management [Exchange], Server Operators, Print Operators, Cert Publishers, Cert Operators, Remote Management Users. For each of these groups, you should be able to say why each individual member is there and whether they need to be there.
Members of the mentioned groups can, with a bit of effort, become domain administrators. For some inspiration:
Conclusion
What do you think about this? Did anything I wrote surprise you? I know that for many of you this is not new and you have it covered. As I wrote about updates, as long as we talk/write about these things, they sound simple. Implementing measures and maintaining order is much harder. So, if you have everything covered, you deserve acknowledgment and praise. đ
We often encounter situations in companies where everyone knows what needs to be done and how it should be set up. But it fails to get finalized for months or years. As an external observer, I would say too many hares are being chased in the field at once, and none are caught. Itâs necessary to reduce the amount of ongoing work and tackle task by task (constantly switching between tasks is exhausting).
May your networks stay secure,
Martin