Rapid Response to the Log4j CVE & Important System Design to Limit Potential Impact
Article By George Nychis
Response By Engineering, SRE, and Customer Success
- 7 February 2022
- 5 minute read
Introduction
Though many of our readers will already be aware of this topic and the severity of it, we present Soroco’s response with its engineering team on the Log4j vulnerability. In this post we will describe how we learned of the vulnerability and provided rapid response to it within 24 hours and patched it across our production environments. Additionally, how our system architecture ensured that any impact of the vulnerability would be isolated due to container technology. After significant analysis of our product’s usage, we were not able to find any known impact of the Log4j vulnerability.
Background
On December 9th, 2021 the Log4j vulnerability was announced and recorded as CVE-2021-4422 with the highest score of 10. It allowed remote code execution which meant that an attacker could potentially execute a series of commands on a server where information was processed with the Log4j library. Those commands could, for example, read information on the server, modify information, delete data, or even attempt to extract sensitive information from it. If the reader wants a technical explanation of the vulnerability, we recommend the following post.
What made this announced vulnerability so severe and the reaction to it so substantial was that it was “zero-day” and used so widely across the industry. Zero-day referred to attackers being aware of it before or at the same time as security researchers without a known patch. Its wide use across the industry included all major cloud services such as Apple, Google, Microsoft, Cloudflare, and Twitter. Additionally, around 32% of the Fortune 500 is reported to use ElasticSearch which leverages Log4j which likely contributed to the large response.
How Soroco’s System Architecture Minimized Any Potential Impact
Soroco’s use of Log4j was not direct, but rather indirect through our use of ElasticSearch and the ELK stack, like many others in the industry as descried above. This meant that our response was to patch a 3rd party technology’s use of the Log4j library rather than direct use of Log4j any of our products.
What was most beneficial to minimize the potential impact of the vulnerability, was Soroco’s use of isolating its software systems and services with software containers. The way that Soroco deploys the ELK stack is through the modern use of software containers. Containers run individual software systems in isolation by packaging the software system and all other dependencies needed to run the software. Aside from the simplicity containers provide in running software, their isolation properties also provide security benefits since what is running inside the container is not (by default) given access to other software systems running in other containers. Even if remote code were executed using the Log4j vulnerability (which we have found no evidence of at Soroco), it could not access anything other than what was in the ELK container. At Soroco, this deployment model meant that the vulnerable ELK stack would not have access to other parts of our infrastructure such as a production database.
Responding in 24 Hours and Patching the Product within 48 Hours
Immediately with our knowledge of the vulnerability and understanding our use of it in our products, we began proactively notifying our customers of our acknowledgement of the published vulnerability and our expected response to it. This was received positively by our customers, as they were trying to reach out to other vendors while having received a proactive acknowledgement from us.
After notification we began patching. By deploying the ELK stack through Elastic’s officially published containers for them, patching the vulnerability also became simple and would not require any substantial changes to Soroco’s product technology. We only needed to ensure our products could still run with small periods of ELK downtime while we patched them.
Once these two changes were made, we immediately began updating these two containers across our production environments to remove the vulnerable functionality. Though again, our use of container technology helped isolate the vulnerable technology from major portions of our technology stack and production databases.
Learning from Soroco’s Response
There are several things that we stressed to our teams globally from the announcement of the Log4j vulnerability and our response to it. First, we worked with our customer success team to ensure that we proactively acknowledged the Log4j CVE to all of our customers before they wrote to us looking for a response. This showed Soroco’s dedication to product security with our customers and assured them that we are continually monitoring systems for announced vulnerabilities. Second, Soroco’s engineering and site reliability engineering teams worked immediately on finding the known workarounds and patching the product within 48 hours. This showed our ability to quickly patch our product and in particular how the use of modern container technology made it simple. Lastly, Soroco’s use of container technology also limited the potential exposure from vulnerabilities by isolating the information and systems that they have access to. All of these together ensured our product and its use was safe for our customers.
Like this article? Spread the word