Heartbleed has caused network administrators the world over to investigate their security infrastructure for vulnerabilities. While this review is a good thing on balance, some administrators are crippling their backup systems in the process, according to a recent blog post by out-of-band management firm, Opengear (News - Alert).
Running a Heartbleed vulnerability scan against Hewlett-Packard’s proprietary lights-out out-of-band management server technology (iLO), for instance, can cause a complete lock-up of the backup system.
Since the iLO itself controls power to the server, if the iLO goes down then the only way to restore management access is to physically remove power from the chassis so that the iLO can restart cold.
Basically: Firms that rely on the HP out-of-band management solution could find that the backup doesn’t work when it comes time to use it.
This highlights the need for following best practices when it comes to out-of-band management.
Chief among these best practices is having backups to the backups. Backup systems exist for when things go wrong, but sometimes these systems themselves also go down in a perfect storm scenario. A hardened system will use a remotely-switchable protocol data unit (PDU) that itself can be controlled by out-of-band such as a serial console.
It also is important to ensure that a company has out-of-band remote access to the management network itself through redundant WAN and/or PSTN dial-in and/or 3G/4G LTE (News - Alert) cellular.
Second, it is important to monitor the management network to ensure that it is ready for use when needed. For those who did a Heartbleed scan of their HP iLO, there could be a nasty surprise in a year or two when they go to use their iLO and discover it is not operational.
“An out-of-band management appliance can help here, by monitoring that consoles are plugged in and lights-out cards are up, and alerting you directly or via your corporate NMS should your contingency plan need attention,” noted the blog by Opengear.
Third, this particular Heartbleed matter brings to light the issue that some firms have their management interfaces connected to the main corporate network, when in fact these interfaces need to be separate.
That’s because many lights-out systems and PDUs lack the security systems of more public-facing servers. They’re maintained less, they sometimes have default passwords that have not been changed, and they often are powered by basic microcontrollers that can easily be manipulated by denial of service attacks and other threats. They need to be off the grid to avoid being generally accessible and therefore become a security vulnerability.
There’s nothing worse than a crisis caused by a crisis management system, so network administrators should learn from this scanning vulnerability with the HP iLO.
Edited by Rory J. Thompson