Second of a two part series
In part one of this series, I invited guest blogger HD Moore to outline the vulnerabilities and risks inherent in baseboard management controllers. HD is best known as the founder of Metasploit, the foremost open source exploit development platform. He has spent the last 20 years auditing software, writing exploits, building products, and helping organizations secure their critical infrastructure. In his current venture, Special Circumstances, LLC, HD continues his mission to help organizations succeed through business advisory services, software development, security research, and penetration testing.
In this post, he will provide detail on the scope of BMC vulnerabilities and I will explain strategies that help mitigate the risks inherent in IPMI. Moreover, I will explain how the Avocent Universal Management Gateway can help you secure access to your infrastructure and streamline systems management.
By: HD Moore, Special Circumstances, LLC
Open doors are convenient but they can be dangerous
Baseboard Management Controllers (BMCs) can be found across all kinds of corporate networks, from mid-sized offices to massive data centers. Almost all major server manufacturers include BMCs on their mid-range and high-end servers, with most of these exposing the BMC to the connected network by default. In addition to standard IT equipment, BMCs are often enabled on third-party appliances, including those provided by security vendors. In most cases, the BMCs are vulnerable to compromise using off-the-shelf tools, providing an intruder with long-term access to the corporate environment from the heart of a trusted system.
As of June 2016, over 165,000 BMCs were found directly connected to the public internet. In the United States, the vast majority of these systems belong to hosting providers. Hosting providers struggle with BMC security more than most, as their business model depends on providing customers with access to servers, and BMCs are a convenient way for a customer to diagnose problems and reinstall a broken server remotely.
I examined the security measures taken by a popular hosting provider to secure access to the BMCs of their customers’ servers. This provider required each customer to access the BMC over a VPN and only provided access to a non-administrative user account. This didn’t help; any attacker with $60 to spend could reconfigure the BMC from the server OS, downgrade the firmware to a version vulnerable to a remote root exploit, and use that root access to read the clear-text administrative password, supposedly only known to the provider. This password was shared across all customer BMCs, allowing a complete compromise of every single customer server, along with permanent access to the physical hardware. This provider was one of the better ones that I looked at and this was just one of the attacks available.
Although hosting providers account for the most internet-connected BMCs in terms of volume, over 2,000 different organizations in the US have at least one BMC connected to the public internet. These include over 100 different educational institutions, approximately 10 public utilities, and a wide range of public and private enterprises. These are BMCs attached to servers that may be responsible for anything from payroll to voting systems.
On a positive note, there are a number of free tools available for identifying exposed BMCs, and most vulnerability scanning platforms should also make identification easy. In addition to the tools, Project Sonar provides a public dataset of all IP addresses identified with IPMI exposed. These resources can be used to quickly identify and take action for any internet-exposed BMCs.
Address the dangers with a well-crafted strategy
The challenge, however, is what to do about these issues. One issues identified by Dan Farmer, the RAKP+ password hash exposure, cannot be fixed and is part of the protocol itself. The other issues can sometimes be resolved depending on the vendor, but even if a solution is available, it may not reduce the risk by enough to make much of a difference.
Getting a handle around BMC security requires three things; disabling BMCs when they are not necessary, isolating the BMC interfaces to dedicated VLANs or interfaces, and monitoring corporate traffic to record IPMI activity. Keep in mind that once a server or a BMC is compromised, the other is too, and both network interfaces (the host and the BMCs) are available to the intruder.
Although a common recommendation is to move all BMCs to a management network, this doesn’t go far enough in terms of isolation. All it would take is a single server compromise to expose the entire BMC network. A more complex strategy involves creating a new VLAN for each server you are monitoring and tagging the BMC interface with these VLANs. On the backend, route a port to a management server, that has virtual interfaces for each VLAN. This host will be able to manage each server using standard tools, while still isolating each BMC from the network and from each other. The Avocent UMG product follows this model; it has physical ports available each BMC that you want to manage, and handles both the device management and the isolation aspect.
Create a Master Key: Secure and manage your infrastructure with the Avocent Universal Management Gateway
Over these two posts, HD has provided a thorough analysis of an important issue that is on the mind of most data center managers. He has also highlighted how the design of the UMG can help isolate each BMC, giving users greater control. But isolation is only one benefit of the UMG.
The Avocent Universal Management Gateway (UMG) is a valuable tool for managing BMCs, which are also called embedded service processors by most IT vendors. Each vendor provides a custom set of capabilities that can be accessed through the service processor and the UMG can access and manage them all, right out of the box.
The UMG can be configured in many ways. The most secure configuration is to directly attach each service processor to the UMG, creating an out-of-band network. This out-of-band network is protected by a hardened Linux distribution with complete access control (authentication and logging). The UMG aggregates all service processors, identifies the specific vendor implementation and presents the available functions (console, power control, sensors, etc.) in a user-friendly single pane of glass view. Up to 40 service processors can be connected to each appliance.
The second configuration option enables management of up to 1024 service processors per UMG appliance. This configuration automatically detects service processors on your network and populates the UMG database with the required management information. Once connected, administrators can update firmware, change passwords and initiate any service processor capability right from the centralized UMG management interface.
Know the risks and reap the rewards
Baseboard Management Controllers present a security challenge that is unique in terms of co-dependence and the difficulty in recovering from a successful attack. Be careful with used and refurbished servers because they may include BMCs that have been permanently backdoored. Segmentation and isolation is helpful, but needs to be implemented correctly to prevent an attacker from compromising the entire BMC segment. The Universal Management Gateway is one solution to help you minimize the risks and leverage all the features provided through BMCs.
Good luck and stay safe!
Download the Article: Service Processor Management-With Power Comes Responsibility and Unified Infrastructure Monitoring, Acess and Control White Paper
