HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload
A vulnerability dubbed HollowByte allows unauthenticated attackers to trigger a denial-of-service (DoS) condition on OpenSSL servers with a malicious payload of just 11 bytes. The OpenSSL team has silently fixed the vulnerability and backported the patch to older releases.
Intelligence analysis by Llama
A 11-byte malicious payload can trigger a denial-of-service (DoS) condition on OpenSSL servers, causing the server to allocate considerable amounts of memory. The OpenSSL team has fixed the vulnerability and backported the patch to older releases.
Imagine you're at a restaurant and you order a small plate of food. But the waiter brings you a huge plate of food instead, and you have to wait for a long time for the waiter to bring you the correct plate. This is similar to what happens when a hacker uses the HollowByte vulnerability to make an OpenSSL server allocate a lot of memory. The server gets confused and can't do its job properly.
Analysis
A Critical OpenSSL Vulnerability
A vulnerability dubbed HollowByte allows unauthenticated attackers to trigger a denial-of-service (DoS) condition on OpenSSL servers with a malicious payload of just 11 bytes. The OpenSSL team has silently fixed the vulnerability (no identifier assigned) and backported the patch to older releases.
How the Vulnerability Works
In a TLS handshake, each message has a 4-byte header for declaring the size of the incoming message. However, vulnerable OpenSSL versions allocate the declared length before receiving the payload and checking its size. Every TLS handshake message begins with a 4-byte handshake header, where a three-byte length field discloses the size of the handshake data that should follow. Without validating the payload, the server trusts the packet's claims and allocates the indicated memory.
The Impact of the Vulnerability
Okta researchers note that while OpenSSL frees the buffers when a connection drops, the GNU C Library (glibc) has a different way to handle memory and "does not immediately return small-to-medium allocations to the operating system; it keeps them for potential reuse." "By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing those freed chunks," Okta says. "The heap fragments heavily, causing the server's Resident Set Size (RSS) to climb continuously. Even after the attacker disconnects, the server remains permanently bloated." The only way to fully reclaim the space is by restarting the process.
Fixes and Recommendations
The open-source OpenSSL library is embedded in popular software projects such as NGINX and Apache web servers, language runtimes (e.g., Node.js, Python, Ruby, PHP), and databases (MySQL, PostgreSQL). It comes pre-installed on most Linux distributions for TLS encryption and certificate handling. In Okta's tests on NGINX showed that low-capacity environments can be easily depleted of memory using HollowByte, while higher-spec servers may lose up to 25% of their memory while the attack bandwidth remains below security alerting thresholds. Although DoS flaws are considered less severe than vulnerabilities that enable data theft or code execution, they can cause operational disruptions and reputational damage. The HollowByte DoS issue has been fixed in OpenSSL 4.0.1 and backported to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21, which now grow the buffer only when the data arrives, ignoring header claims. Despite being addressed as a "hardening fix" and not a security vulnerability, Okta recommends "upgrading your distribution's OpenSSL packages immediately."
Key points
- A vulnerability dubbed HollowByte allows unauthenticated attackers to trigger a denial-of-service (DoS) condition on OpenSSL servers with a malicious payload of just 11 bytes.
- The OpenSSL team has silently fixed the vulnerability and backported the patch to older releases.
- The vulnerability affects popular software projects such as NGINX and Apache web servers, language runtimes, and databases.
- Organizations should prioritize switching to a fixed version of the OpenSSL library to prevent operational disruptions and reputational damage.
Organizations can prevent operational disruptions and reputational damage by prioritizing the switch to a fixed version of the OpenSSL library. The OpenSSL team has fixed the vulnerability and backported the patch to older releases, making it easier for organizations to stay secure.
If organizations fail to switch to a fixed version of the OpenSSL library, they may experience operational disruptions and reputational damage. The HollowByte DoS issue can cause the server to allocate considerable amounts of memory, leading to a permanent bloating of the server's Resident Set Size (RSS).
