It was exactly one month ago, that we published our findings of the first real-world rootkit infecting HP iLO firmware. Along with the detailed report about the malware, its modules, and technical capabilities, we noted a simple method to detect the malware relying on the firmware UI’s appearance.

As there was no tool for detecting the malware, we promised then to publish a tool for scanning the iLO firmware, so that everyone can check their servers for infection.

Today, we’re pleased to announce Padvish iLO Scanner, an open-source tool (the first one) which can be used to dump and check the integrity of HP iLO firmware. This tool takes a whitelist approach, so it can not only detect iLOBleed, but also any kind of infection inside the firmware. It also provides the first step for any researcher who wants to analyze malware inside iLO, by providing the dump file.

With the tool currently published only for use in Linux, we hope to see versions for other operating systems being developed by enthusiasts.

We’d love to hear back from you if you’ve got any experience with the tool. You can contact us at apt@amnpardaz.com for this matter.

Also, in the meantime, we noticed some questions and confusions around iLOBleed, and how to protect against it. So we decided to put a FAQ about it with all the most important information we think is necessary for any admin to know.

What is ILOBleed?

iLOBleed (a.k.a. “Implant.ARM.iLOBleed.a”) is the name of the first malware (a rootkit) discovered in HP iLO firmware. It is a malware that can hide completely from detection by any security tool, run even when the system is turned off, persist even after formatting and clean-installing the OS, has unprecedented access to all server’s hardware beyond anything available to the OS, and do more.

What is HP iLO?

iLO is a management chip in HPE Servers that allows local and remote administration of the server. It lets server administrators remotely turn on the server, configure the hardware, install the OS, etc.

As soon as you connect the power cable, the iLO will boot up. Even when the server is itself powered off, the iLO is never turned off. It provides multiple services over the network over HTTPS, SSH, SNMP, etc.

iLO runs on a low-power CPU separate from the server’s main CPU that the OS runs on, and can communicate and control the main CPU, hardware, and the OS.

How can iLOBleed persist after formatting and reinstalling the OS?

iLO is a different OS, with its own CPU, persistent memory, network card, etc. The malware infects the iLO firmware, so it can persist even after formatting the hard disks and installing a new OS. And it can control and infect the OS or wipe it completely, right from the firmware. The firmware has complete control over the system.

What is the level of access iLOBleed has? How much damage can it do?

The malware “Implant.ARM.iLOBleed.a” has direct access to all the hardware, so it can do anything it wants, without any intervention from the OS or security products installed on the system. The current version is capable of wiping disk data, but in theory, it could do any other damage it wanted.

What do you mean that iLOBleed can run when the server is off?

HP iLO is designed to be capable of managing the server remotely, including the ability to turn the server on. To implement this feature, the iLO will start right after the server’s power is connected, and the malware will start with it.

Is this a new threat?

Yes.

This is the first time a real-world rootkit was found in HP iLO firmware.

The possibility of writing such malware had been theorized and studied in previous research, but real malware was never discovered before. In fact, this was the first time malware was reported in any kind of BMC. (iLO is the HP-special kind of BMC or Baseboard Management Controller)

Should I be worried?

Yes.

This kind of attack is supremely stealthy and very persistent. It cannot be found via conventional tools and can go unnoticed for years. It is very powerful, too.

The “Implant.ARM.iLOBleed.a” was found in iLO 4, but there are similar attacks applicable to the newest iLO firmware if a non-default configuration is not activated.

Am I safe? I’ve installed the latest patches from HP.

No.

We’ve seen FULLY-PATCHED systems infected with this malware.

If you’re using the HP’s G9 or lower series of servers, and even if you’ve upgraded the firmware, you’re still at risk. The attackers can easily downgrade the firmware to a vulnerable version, infect it, and then upgrade it for you.

In fact, without using an iLO scanner, you don’t know whether you’ve been infected previously or not. (which in that case, upgrading would have no benefit)

If you’re using the G10 series, then no persistent malware has been found for your server yet. But even these servers allow the downgrade process by default, so the attacker can brick the server’s firmware, rendering it completely unusable, without any means to boot or use the server. In this case, the server cannot be repaired without doing special hardware-level repair.

That’s why you should check your server with an iLO scanner too.

As there’s no iLO firmware scanner publicly available, Amnpardaz Software has published the Padvish iLO Scanner as open-source. (You can find the link at the end)

Am I safe? This malware only wipes disk data, but I have backups.

No.

Having backups is always a good plan. But this malware is not written to only wipe data, it is written to maintain ultimate persistence during long periods. Wiping data is one of the malicious functionalities the malware has.

Note that the attackers can also brick the server’s firmware and render it completely unbootable, and unrecoverable, without special hardware-level repair.

How can I get infected?

That’s a good question, which needs a little elaboration.

The iLO provides two categories of communication channels: Network, and local.

The network communication is provided through a special network adapter assigned to iLO, and is available using different protocols such as HTTP(S), SSH, SNMP, …

Segmenting the iLO network, upgrading the iLO firmware, and using strong passwords can increase its security against threats over the network.

The local communication is available directly to the host operating system installed on the server. This channel requires no authentication, and having the root/admin privileges on the server is sufficient to send commands directly to iLO.

Both methods can be used to infect the server’s iLO firmware with malware. This means infection of the host OS is sufficient for attackers to penetrate and infect the iLO firmware, to remain under the radar for extended periods. This can occur, even if you’ve detached the iLO network port completely.

How can I find out whether I’m infected or not?

Amnpardaz has published an open-source HP iLO scanner utility named Padvish iLO Scanner, which can be run on the host OS of the server and determine whether it is infected or not.

The tool directly reads the ILO’s flash memory and saves a dump of the firmware. It also compares the dumped firmware to known ILO’s published by HP and can warn you if any kind of infection or modification is done to the firmware, not just iLOBleed.

You can visit the tools repository and its guide at: https://github.com/padvish/iLOScanner

Padvish iLO Scanner reports that I’m infected, now what to do?

If you see the “UNKNOWN FIRMWARE” message, you may be infected, or it may be the case that the tools’ whitelist was not comprehensive.

Either way, you’re advised to contact us (apt@amnpardaz.com) to investigate the case.

If it is in fact an infection, you’ll be provided help fixing the issue, but there are other concerns you’ll have to find out. (How, when, and why you’d got infected)

How can I prevent being infected?

• First of all, please take the time to investigate whether your servers are currently infected or not. It is the first thing you should do.
• On G10 servers, you should turn on the “Permanently Disallow Downgrades” policy. This option is available inside iLO’s configuration. You should consult the server’s documents.
• On G9 and older servers, unfortunately, there is not much protection you can put in place. It’s always possible for the attacker to downgrade your firmware and infect it secretly. So you have to rely on mitigation methods to lower the risk of infection and periodically scan for the malware.

In either case, you can use the following tips to harden your servers and lower the risk:

• Do not connect the iLO network interface to the operating network and improvise a completely separate network. (Preferably using a totally separate switch, cabling, and management stations)
• Periodically update the iLO firmware version to the latest official release from the vendor
• Use defense-in-depth strategies to reduce risk and detect intrusions before reaching the iLO
• Periodically use the iLO Scanner tool to detect any infection or malware in the iLO Server firmware.
• Configure the iLO to send logs to another place, in order to analyze and save the logs.