Welcome back!
We’re back again with a monthly-ish blog reflecting on major goings on in the security world. I am writing the first draft on the rarest of holidays, leap day1! February gets extended by a day every four years, and if we had to guess Ivanti wishes this month would end. After we published last month detailing the kerfuffle over the ConnectSecure vuln, a few more things happened: more zero-days were disclosed, it was discovered that the Pulse Secure appliance firmware was running an 11 year old version of Linux, and CISA (along with other law enforcement and national cybersecurity organizations) published a long advisory detailing the attacks against and problems defending Ivanti instances, including the fact that Ivanti’s Integrity Checking Tool (ICT) could be bypassed. They ended the summary with this incredibly (for them) strong wording
The authoring organizations strongly urge all organizations to consider the significant risk of adversary access to, and persistence on, Ivanti Connect Secure and Ivanti Policy Secure gateways when determining whether to continue operating these devices in an enterprise environment.”
Yikes. But let’s not live in the distant past2. Let’s live in the more recent past, and talk about what caught our attention this month as members of the cybersecurity rumination vanguard3. At the beginning of February, the Internet tubes were all abuzz (literally) with a story (original link to Swiss German tech publication) of 3 million smart toothbrushes being used in a DDoS attack. Unfortunately, this is the kind of report that reminds us of the old journalist saying “some stories are too good to check”. This saying proved true here, and it turned out that smart toothbrushes were not used in a DDoS, but rather “could be”.
Of course, many “smart” devices run a paired down version of linux, and it can be a little difficult to update the firmware on your toothbrush. This means that many of the “things” on the Internet of Things are going to be vulnerable in some way or another and can and have been used in attacks. Anyone remember Mirai? In fact Bitsight has done some work on finding cameras exposed to the Internet. We don’t want to say we are prescient, but that blog post did talk about the threats to organizational security these types of open cameras might expose. Lo and behold there was evidence last month that Russian attackers may have used exposed cameras to target attacks. Unlike the toothbrush attack, this is an all too real, and scary realization of experts reading the tea leaves and predicting an attack.
LockBit and ConnectWise
What’s forefront on our mind at the moment is the ongoing incidents with Lockbit ransomware and its leveraging of the ConnectWise ScreenConnect Remote Desktop Software. This has been a compelling story because, unlike imaginary toothbrush attacks, this one is very real and very damaging. Before we get to some data analysis4, if you’ll allow me a little digression to talk about why LockBit as a ransomware gang is so darn interesting.
LockBit has been around since late 2019 (originally named ABCD Ransomware after the file extension used by malware). At the time, it was somewhat innovative in that it leveraged automated network discovery once inside a network to find assets to ransom. Since those early days, they ransomware gang has been prolific and infamous:
- They managed to infiltrate the Royal Mail
- Infected a children’s hospital, then apologized and gave the hospital the decryptor for free
- They have a penchant for attacking industrial control systems
- In recent versions (3.0), they released a “bug bounty” program, the first of its kind in the ransomware world.
All in all, CISA estimates they are responsible for 1,700 attacks totaling more than $91M. As the good folks at Wired have pointed out, this type of high profile in a ransomware gang, tends to attract the attention of law enforcement, and it did in this case, it’s happened several times. In November of 2022, several members were arrested by the US Department of Justice, other members were charged in July 2023, and on February 19, 2024, an international group of law enforcement agencies seized control of LockBit infrastructure on the darkweb.
This latest takedown wasn’t the end of LockBit however; a good deal of their infrastructure soon reappeared along with a lengthy statement about the ineffectiveness of the takedown, some self-flagellation about the negligence of their own security that lead to the takedown, and threats to Fulton County, GA, which was experiencing a disruption for which the group claimed credit.
So after ~300 words and 12 references to “briefly” describe LockBit, we can actually get to what information Bitsight might have on this particular group. Well, it turns out that recent attacks that have been leveraging LockBit ransomware have been exploiting a vulnerability (CVE-2024-1709) in ConnectWise ScreenConnect.
So why is ConnectWise ScreenConnect such an attractive target? ScreenConnect is a remote desktop software that operates by having a central server connect to endpoints running an agent. An administrator logs into the server, selects the machine they’d like to administer and ScreenConnect facilitates the connection. The security minded among you are probably seeing how this central point of access could pose a pretty big risk. If an attacker can compromise a ScreenConnect server, then it’s open season to install ransomware on whatever machines that server administers.
The first step for any attacker is to find an asset that has an exploitable vulnerability. But how hard is it to discover ScreenConnect servers with this particular vulnerability? Turns out it’s really easy. Scan an IP and, if ScreenConnect server is running an on-prem instance, it will happily tell you so and even let you know what version it’s running.
Figure 1 Remarkably easy to find ScreenConnect Servers.
When examining a new vulnerability like this, two questions come to mind: How widespread is ScreenConnect and how quickly did folks leap to fix broken versions? From January 1, 2024 until March 1, 2024, our scanning infrastructure found 6,502 IP addresses responding that they were running on-prem versions of ScreenConnect. Of course, part of the power of Bitsight is we have a pretty good idea of what IP addresses belong to which organizations, so we can ask interesting questions like “Which industries are responsible for those 6,502 IPs”, unsurprisingly, it’s the Telecoms and Technology companies, as these are likely to be service providers who would be well served by having this kind of remote administration software.
Figure 2 Bubble chart of distribution of industry sectors and IPs discovered to have ScreenConnect5. The colors don’t actually mean anything, they are just brand colors to break up the riot of circles. Not everything has to mean something, sometimes things can just be pretty.
If one of the larger risks of having a vulnerable ScreenConnect server running is the ability to install ransomware on all the machines that ScreenConnect server administers, then it follows that larger organizations may be more at risk than smaller ones. Those larger organizations are likely to have more assets, network infrastructure, and diverse technology stacks too. So we examined the prevalence of ScreenConnect broken down by organization size in Figure 3.
Figure 3 Screen Connect prevalence in the first two months of 2024 by company size.
Figure 3 indicates that larger companies are more likely to have internet facing detectable instances of ScreenConnect running, though we’d note that there are fewer companies at the larger end so the error bars get a little larger. As is always the case with the risk posed by vulnerabilities like this, exposure is one thing, but remediation is another. So the next thing I looked at was how many of the found instances were vulnerable over time (Figure 4).
Figure 4 Percent of vulnerable instances6 of ScreenConnect from Feb 14, 2024 through March 1, 2024. Prior to Feb 15, 2024. The line represents a 1 week backwards looking rolling average.
We note that every scanned instance of ScreenConnect was vulnerable, and we started to see versions updated to 23.9.8 before the official security alert, was issued. After the security alert things declined to around 30% of instances and will probably continue to do so in the next month (maybe we’ll check back and let you know at the beginning of April). It’s worth it to break down the remediation curves by industry, just because we can in Figure 5.
Figure 5 Percent of vulnerable instances of ScreenConnect from Feb 14, 2024 through March 1, 2024. Prior to Feb 15, 2024, by sector. The line represents a 1 week backwards looking rolling average. The point at the end is the average over the prior week (February 23-March 1st).
There is quite a bit of variation in the chart, Retail struggling to quickly remediate, but Education and Healthcare/Wellness getting down to below 10% relatively quickly. But what about company size? If we take the measure of remediation average of the prior week as of March 1st and compare that to the overall prevalence we saw in Figure 3, we get Figure 6.
Figure 6 Percent of ScreenConnect servers with vulnerable version strings vs the overall percentage of organizations with a particular size that have screen connect.
What’s interesting here is that large organizations are more likely to have ScreenConnect running and seem to be slower to fix things. Given that ScreenConnect is a crucial tool for IT administrators it may be harder for them to actually just “update” things without breaking other dependent infrastructure. But they are running a higher risk here if those ScreenConnect instances are administering a large portion of their internal assets.
Finally, this particular vulnerability gives us the opportunity to examine how running legacy software might affect an organization’s ability to upgrade in the face of new vulnerabilities. In particular, the ScreenConnect versions are easy to parse and seem to closely follow Semantic Versioning practices. And also in this case the upgrade path was a multistep one, with ConnectWise indicating that upgrades needed to happen in sequential order to be successful (2.1 → 2.5 → 3.1 → 4.4 → 5.4 → 19.2 → 22.8 → 23.3 → 23.9). This means we can ask questions like “Were organizations that were way behind in updating ScreenConnect less likely to be able to update to a newer version?” The answer in Figure 7 seems to be a resounding “yes”.
Figure 7 % of IP addresses with ScreenConnect version strings vulnerable to CVE-2024-1709 broken out by what the major version of the software was when it was last discovered to be vulnerable.
The general trend in Figure 7 is that if an IP was at 23.x before this vulnerability came out they were much more likely to be able to fix things up and get to a non-vulnerable version of 23.x. This is followed closely by 22.x versions and then everyone else is not doing so great. We can even dig a little deeper and ask “what upgrade path did folks take?” and when we do we get some interesting results in Figure 8.
Figure 8 Upgrade path for ScreenConnect7
What’s fascinating here is that most folks were up to date to begin with and were able to quickly patch to the new version. Perhaps surprising is that a number of the servers with vulnerable version strings actually had upgraded or patched to still vulnerable versions, including to major versions (22.x and 23.x) that have patched releases available!
Parting Words
This (not so) little study of this vulnerability telegraphs some interesting overarching conclusions. First, those useful remote administration tools can pose a huge danger when in the hands of an attacker. It’s probably best to have multiple defenses for these types of tools, and the risks of putting them on the open Internet should be weighed very carefully. Second, at least in this case, it appears remediation is easier if you keep things up to date as you go. That software that’s three versions behind may still be supported and might be relatively free from security vulnerabilities, but an event like this could make the path to remediation that much more difficult.
We’ll be back next month to celebrate St. Patrick’s Day and Easter and whatever security goings-on catch attention. Cheers.
1It’ll get published a bit later, but I had to use this as a setup.
2If you haven’t unplugged your Ivanti Connect Secure VPN, thoroughly power washed everything that it was possibly connected to, and updated it, maybe keep living in the past until you get that done.
3Read “thought leadership”
4My true raison d'être.
5Is this chart a little messy? Yeah totally. Could it have been a bar chart? Absolutely. But if you make bar charts all day, even really nice ones, you get bored. So how about a bubble chart for some fun? If you hate it, I’m sorry.
6Hope you are ready for an extra long footnote on the messiness of security data and the challenge of deciding what versions of a particular piece of software are vulnerable to a particular vulnerability. In this case ConnectWise indicates that “ScreenConnect 23.9.7 and prior” are affected. However, NVD indicates that only 23.8.4 through 23.9.7 are vulnerable. In our data, we found servers running versions as far back as 3.x. It’s possible that some versions are so ancient that this particular vuln doesn’t exist for them. To complicate matters more ConnectWise released patches for many prior 22.x versions of their software. So how did we proceed? We parsed the version strings and unless the version was running 23.9.8 or greater OR the version string matched one of the patched versions released on February 22, we are calling it “vulnerable”.
7Note that the numbers in Figure 8 don’t quite align with the ones we saw in Figure 4. Don’t fret! Figure 8 reflects IP addresses that we could examine both before and after the security alert was issued. Some IPs simply didn’t show up as having ScreeenConnect in later scans, indicating they may have been taken offline completely or placed behind a VPN. Additionally some IPs show up with ScreenConnect only after the security alert was issued. There is probably some more research to be done on these disappearing and IPS, but this blogpost is already nearly 2,000 words, 7 footnotes, and 8 figures long, so we’ll stop soon.