How an old bug in Lighttpd gained new life in AMI BMC, including Lenovo and Intel products

Executive summary

The software supply chain is complicated, and all the issues associated with it are something we haven't dealt with before and require a different mindset and approach. The vulnerability in Lighttpd was discovered and fixed back in 2018, but a CVE was not assigned to this vulnerability, and a fix was delivered silently by project maintainers. Frequently, the software that uses the open-sourced components does not consume every single update coming from OSS maintainers and only watches the critical changes or important security fixes to apply. In reality, it's hard to track every change for security issues without specific security advisories and CVE assigned. 

‍

‍

In the firmware world, the situation is even worse, and the OSS components that are used have not been updated for years. That's exactly what happened with Lighttpd when developers of AMI MegaRAC BMC didn't consume security fixes for the remotely exploitable vulnerability from 2018 when maintainers of Lighttpd upstreamed it. This particular vulnerability was discovered in many different devices because many different device vendors, including Intel and Lenovo use the AMI MegaRAC. 

The Binarly REsearch team led the coordinated disclosure of this vulnerability to both Intel and Lenovo PSIRTs. They both declined to fix or acknowledge the vulnerability report because the associated products recently hit end-of-life status and will no longer receive security fixes.

We call this the forever bugs that will haunt the software supply chain for a very long time. We decided to document this software supply chain security failure to help the ecosystem recover from these repeatable firmware security failures. 

We have assigned three Binarly identifiers BRLY-2024-002, BRLY-2024-003, BRLY-2024-004.

Here's what happened 

A few weeks ago,  as a continuation of our work on researching BMC security and improving capabilities in our flagship Binarly Transparency Platform, we found a security issue that perfectly encapsulates the complex - and insecure nature - of firmware and software supply chains.

The vulnerability, a Heap Out-of-bounds read (CWE-125) exists in the lighttpd module used in one of Intel Server System devices. We assigned the BRLY-2024-002 identifier and immediately notified Intel’s PSIRT of an unfixed issue in one of their products. 

Intel’s response was that the affected device is currently end-of-life, meaning it will no longer receive firmware updates or security patches.

This is yet another vulnerability that will remain unfixed forever in some products and will present high-impact risk to the industry for a very long time.

While Intel’s PSIRT response is somewhat expected (it may be unreasonable to expect a company to support such a product over the decades), the reality is that this problem is almost six years old!  It was even present and fixed a few years before this Server System was launched. 

All these years, it was present inside the firmware and nobody cared to update one of the third-party components used to build this firmware image. Including the latest firmware version:

This is another perfect example of inconsistencies in the firmware supply chain. A very outdated third-party component present in the latest version of firmware, creating additional risk for end users.  Are there more systems that use the vulnerable version of lighttpd across the industry?

This was later confirmed by our research team: Lenovo BMC firmware for HX3710, HX3710-F, HX2710-E servers are also vulnerable to the very same security issues. We have assigned BRLY-2024-003 and sent notifications to Lenovo PSIRT.

Like Intel, Lenovo’s response was that these servers were also end-of-life and will no longer be supported with security patches.

A very important part of this story is the silent fix that was pushed by lighttpd developers without details in an advisory or without a CVE identifier to help defenders with patch management processes.

Instead, there’s just a commit pushed into the lighttpd repository:

‍https://github.com/lighttpd/lighttpd1.4/commit/df8e4f95614e476276a55e34da2aa8b00b1148e9 

The absence of prompt and important information about security fixes prevents proper handling of these fixes down both the firmware and software supply chains.

In the end we decided to assign the BRLY-2024-002, BRLY-2024-003 identifiers to the affected Intel and Lenovo BMC firmware and BRLY-2024-004 for vulnerable lighttpd builds in general.

How this vulnerability works

Lighttpd web server processes folded HTTP request headers values, which means they can be provided in the following form:

GET / HTTP/1.1\r\n
Host: test\r\n
If-Modified-Since: PartOne\r\n\
\tPartTwo\r\n
\r\n

In such a case, PartTwo (stored in variable value) will be appended to the buffer that contains PartOne (ds->value) with the buffer_append_string function call:

if (in_folding) {
	buffer *key_b;
	…
	key_b = buffer_init();
	buffer_copy_string_len(key_b, key, key_len);

	if (NULL != (ds = (data_string *)array_get_element(con->request.headers, key_b->ptr))) {
		buffer_append_string(ds->value, value);
	}

Then buffer_string_prepare_append function is called, which takes as inputs the same buffer (ds->value) and the length of the string that should be appended to this buffer: 

void buffer_append_string_len(buffer *b, const char *s, size_t s_len) {
	char *target_buf;

	force_assert(NULL != b);
	force_assert(NULL != s || s_len == 0);

	target_buf = buffer_string_prepare_append(b, s_len);

	if (0 == s_len) return; /* nothing to append */

	memcpy(target_buf, s, s_len);

	buffer_commit(b, s_len);
}

‍buffer_string_prepare_append and next called buffer_realloc functions are responsible for checking that the size of the provided buffer is enough for appending the additional value. If this condition is not met, a new memory region with required size is requested using the realloc call, and the address of this memory region is assigned to the buffer, so it can be safely extended:

char* buffer_string_prepare_append(buffer *b, size_t size) {
	force_assert(NULL !=  b);

	if (buffer_string_is_empty(b)) {
		return buffer_string_prepare_copy(b, size);
	} else {
		size_t req_size = b->used + size;

		/* not empty, b->used already includes a terminating 0 */
		force_assert(req_size >= b->used);

		/* check for overflow: unsigned overflow is defined to wrap around */
		force_assert(req_size >= b->used);

		buffer_realloc(b, req_size);

		return b->ptr + b->used - 1;
	}
}

‍

static void buffer_realloc(buffer *b, size_t size) {
	force_assert(NULL != b);
	if (0 == size) size = 1;

	if (size <= b->size) return;

	b->size = buffer_align_size(size);
	b->ptr = realloc(b->ptr, b->size);

	force_assert(NULL != b->ptr);
}

On the other side, if the request contains multiple If-Modified-Since HTTP headers, web server checks if their values are equal with the strcasecmp function call. Here con->request.http_if_modified_since contains the value of the previously provided header value and ds->value->ptr the current one. Crucially, if "folding” was used for the previously provided header and buffer pointer was reassigned with aforementioned realloc call, memory region where con->request.http_if_modified_since points is not updated before this strcasecmp call, which means that current provided header value is compared with freed memory. Also, if the content of these two memory regions are not equal – 400 error code will be returned from the web server. It means that an attacker can start guessing the content of the target memory region based on response status codes:

} else if (cmp > 0 && 0 == (cmp = buffer_caseless_compare(CONST_BUF_LEN(ds->key), CONST_STR_LEN("If-Modified-Since")))) {
	/* Proxies sometimes send dup headers
	 * if they are the same we ignore the second
	 * if not, we raise an error */
	if (!con->request.http_if_modified_since) {
		con->request.http_if_modified_since = ds->value->ptr;
	} else if (0 == strcasecmp(con->request.http_if_modified_since,
				ds->value->ptr)) {
		/* ignore it if they are the same */

		ds->free((data_unset *)ds);
		ds = NULL;
	} else {
		con->http_status = 400;
		con->keep_alive = 0;

Using this OOB read vulnerability, it is potentially possible to exfiltrate sensitive data, such as process memory addresses, which can be then used to bypass security mechanisms such as ASLR.

How did Binarly detect the BMC firmware vulnerabilities?

One of the steps from the recently CISA released BMC hardening guidance is that BMC firmware scanning tools should be used to ensure its security. We strongly believe that this is one of the most important rules to ensure your BMC systems are not vulnerable and not exploited by attackers. Simple firmware integrity checks will not help in all of the cases, so this was the reason to introduce Binarly's comprehensive firmware scanning engine, which is capable of detecting all vulnerabilities we identified, as well as other known and unknown issues related to Baseboard Management Controllers:

Appendix:‍

The Binarly research team is currently tracking a massive number of vulnerable and publicly available BMC devices that are out of support and will remain vulnerable to this issue forever. 

To track Intel devices go to this Shodan search.

At the moment, no publicly available affected Lenovo devices have been identified using Shodan.