.webp)
The Binarly Research Team demonstrates critical security risks affecting the BMC firmware supply chain ecosystem that could lead to full remote control of the Server system.
Earlier this year, the Binarly REsearch team uncovered multiple vulnerabilities in the Supermicro IPMI product – their implementation of BMC (Baseboard Management Controller). These critical and high-severity vulnerabilities allow a remote attacker to completely compromise not only the BMC system itself, but also gain access to the target host and move laterally through the network.
In this blog post, we provide more details about the architecture of BMCs, the unique challenges we encountered during our research and how we addressed them, the risks these devices pose and how they can be mitigated.
Image shows BMC chip on a Supermicro motherboard
The BMC is a separate specialized microcontroller on a server’s motherboard. It has a highly privileged and isolated execution environment, independent from the main server CPU and operating system. As a rule, it implements the IPMI standard, which introduces functionality to remotely administrate, monitor, control and manipulate the target server. It makes this task much easier, because you don’t have to visit the server room every time you have some issues, for example, when the server was accidentally turned off or its main operating system was broken. In these scenarios, you can use BMC to remotely wake it up or reinstall the OS.
BMCs use a SoC (software on chip) concept popular in the IOT/XIOT world, where the capabilities of both hardware and software are combined in a single unit. It has its own CPU, RAM, FLASH memory, network and other interfaces, and at the same time, it has channels to communicate with the main processor and various sensors, allowing it to obtain different readings such as temperature or fan speeds.
Layout of a common BMC architecture
In addition to dedicated hardware, BMC also has its own firmware, which is usually Linux-based OS with limited functionality. It provides several common interfaces for communication with the remote part and the host, which typically is accessible via the following protocols and ports:1. SMASH (TCP 22);2. IPMI (UDP 623);3. Web (TCP 80, 443);4. WSMAN (TCP 5985);5. IKVM (TCP 5900).
Supermicro IPMI’s web interface
SMASH uses the same TCP 22 port as the more famous SSH protocol, and typically also provides a remote shell, but in this case it only allows to run a limited number of commands related to server monitoring and management. The IPMI (Intelligent Platform Management interface) focuses on low-level hardware control of the server, and allows to manipulate remote systems with ipmitool utility, while the web Interface permits to perform similar tasks using the web browser. The WSMAN and IKVM protocols also provide remote control functionality: the former allows to run commands with HTTP requests, while the latter gives access to the server GUI and provides remote keyboard and mouse functionalities.
Crucially, if the implementation of any of these protocols has vulnerabilities, it puts at risk not only the BMC component, but, as you can already guess, the entire server, allowing an attacker to perform any actions as if he were in the same room with the server.
We couldn't pass up the opportunity to research such a powerful attack vector.
One of the biggest problems with the BMCs research is that there are dozens of server vendors, and each of them has hundreds of various server models, and BMC firmware for each of them can be different. It is almost impossible to buy all of these servers and test locally, so we decided to try emulating BMC firmware.
Fortunately for us, most vendors provide firmware images for download at official support websites. The BMC underlying system is typically 32-bit ARM, which shouldn’t be difficult to emulate using QEMU. We initially tried to emulate individual binaries using qemu-arm-static or the entire filesystem with the chroot trick, but we quickly realized that this was not sufficient since different services and firmware components depend on each other. For example, the web server can’t function properly without PAM (Pluggable Authentication Modules) or the IPMI service doesn’t start without a specific kernel feature enabled.
Then we started investigating options for the full system emulation, and found quite a few! QEMU supports Aspeed AST2400/AST2500/AST2600 SoC machines with the ability to customize underlying hardware, such as size of RAM, serial flash and SPI model, configure network devices and so on. For example, to emulate BMC image for Supermicro X11SSM-F, we can use the QEMU supermicrox11-bmc image with the following configuration:
qemu-system-arm -m 128 -M supermicrox11-bmc -nographic \
-drive file=./BMC_X11AST2400-3101MS_20230214_1.66_STDsp.bin,format=raw,if=mtd \
-net nic \
-net user,hostfwd=::2443-:443,hostname=qemu
Here we have specified to use 128 megabytes of RAM, as on a real device, and also instructed to forward port 443 of the web server to the host network. And after some time, we have a working system!
Definitely, QEMU can’t provide the emulated hardware that will be 100% identical to the real device. And as a result, from the beginning we have problems with the network, because QEMU can’t provide two network interfaces that the BMC OS requires. However, this issue can be easily solved by manually setting up the network:
ip link set eth0 addr 4A:0A:AB:7C:96:2F
ifconfig eth0 10.0.2.15
ifconfig eth0 netmask 255.255.255.0
ifconfig eth0 broadcast 10.0.2.255
ifconfig eth0 up
ip route add 0.0.0.0/0.0.0.0 via 10.0.2.2Another problem is that the BMC system network driver behaves differently in QEMU, which causes kernel panics when making network requests:
Looking at the linux kernel source code, we discovered that this exception occurs inside the skb_put function call which is used to extend the size of the buffer. The kernel panic happens when the extended size exceeds the total buffer size:
This problem can be resolved by attaching the GDB to QEMU and bypassing the problematic condition with a such script:
target remote localhost:1234
break *0xc01e3918
commands
silent
set $pc=0xc01e391c
continue
end
continueAs a result, we get a working system that we can use for research:
The most dangerous bugs are those that can be exploited remotely, and the BMC system provides several protocols described in the previous section that can help with this. From our enumeration, the most promising is the web interface, since system administrators often expose that to the public, contrary to all security recommendations and guidelines. For this reason, we chose it as target number one and started testing with both static tools and dynamic analysis frameworks.
Typically a web server consists of a frontend and a backend parts, and we have covered both. As a result, we have identified and reported to the vendor four (4) critical and three (3) high severity vulnerabilities, which we thoroughly described in this blog post:
NOTE: There are multiple additional vulnerabilities currently part of an ongoing coordinated disclosure process. As patches are not yet available, Binarly is withholding details on these issues.
In short, an attacker could first exploit one of the described DOM-based XSS vulnerabilities to obtain an account with administrative privileges in the BMC web console and then use it to completely compromise the underlying BMC OS via command injection. At that stage, nothing stops the attacker from compromising the target host and moving laterally across the network:
In the one of our previous blogs, The Firmware Supply-Chain Security is broken: Can we fix it?, we looked at the supply chain problem with developing and distribution of UEFI firmware:
Supply chain issues arising when vulnerabilities in the end product come from different suppliers. The world of BMCs is no exception. Many parts of BMC firmware are created not by OEMs (original equipment manufacturers), but are outsourced to third-party vendors. If a vulnerability exists in the code base, it could affect multiple different vendors, each with hundreds of server models.
Another problem is that the vendor can’t just fix the vulnerability, it has to wait for another hub in the supply chain – the reference code supplier:
Image shows the affected BMC supply chain vendor ecosystem
In our research project, the vulnerabilities came from ATEN firmware developers, and fortunately we haven’t identified any other vendors using the same codebase except Supermicro. But as you can see, vulnerable code coming from proprietary sources can affect multiple OEMs.
We believe that the OpenBMC project is one of the solutions to this problem for device vendors. Initiated as a collaboration of Google, Intel, Microsoft and IBM, now it is a Linux Foundation project. Its code is open source, making it transparent not only to device manufacturers, but also to end users. It supports most of the modern features required for remote server controlling and management. Inevitably, it is also susceptible to security vulnerabilities, including critical ones such as CVE-2021-39296, but we strongly believe that a centralized solving of the problem will allow for faster and more frequent patching of affected systems, so you don’t have to wait for the 3rd party company when they release an update for the Critical security issue that is already actively exploited in the wild.
Baseboard Management Controllers provide critical functionality that allows access to full control of remote systems, while at the same time opening a wide surface for attackers. We see that for the BMCs, it is still possible to find classes of vulnerabilities known from the early 2000s, and also in this case, possible mitigations that might be introduced by developers like stack canaries, ASLR or non-executable stack are simply not applicable, so comprehensive code review and continuous static and dynamic analysis are also required. We also strongly encourage system administrators and security teams to follow BMC hardening guidelines, especially mitigating risk around exposed instances and weak credentials.
One of the steps from CISA’s new BMC hardening guidance is that BMC firmware scanning tools should be used to ensure its security. We strongly believe that this is one of the most important rules to ensure your BMC systems are not vulnerable and not exploited by attackers. Simple firmware integrity checks will not help in all of the cases, so we built a comprehensive Binarly firmware scanning engine capable of detecting all vulnerabilities we identified, as well as other known and unknown issues related to Baseboard Management Controllers.
Our solution allows to not only determine whether a certain component of the firmware contains a vulnerability, but also shows the specific piece of the code where this vulnerability is located. Moreover, our detection rules provide comprehensive annotations to findings, allowing developers and security teams to quickly identify the issues and reliably mitigate them:
Image shows the Binarly Transparency Platform detection capabilities
Reach out for a demo of these capabilities in the Binarly Transparency Platform.
Are you interested in learning more about Binarly Transparency Platform or other solutions? Don't hesitate to contact us at fwhunt@binarly.io.
.svg)
