New tool seeks to block rootkits by protecting their targets

In recent years, malware authors have developed increasingly sophisticated rootkits that burrow into the operating system itself, modifying basic filesystem and process management code in a way that ensures they are essentially invisible to anyone using the machine: no files visible, no processes apparent. While some progress has been made in detecting when a rootkit has compromised a system, preemptively blocking an attack has remained challenging, since the malware relies on important system functions. A team of computer scientists have now described a tool, called Hook Safe, that uses virtualization to preempt rootkits by moving and protecting the kernel functions that they target.

Rootkits burrow their way into an operating system’s kernel using a process called hooking. The services provided by a kernel—file system and hardware access, memory management, etc.—are accessible through callable functions. The kernel keeps track of where the functions reside in memory using pointers, which contain the address in memory of the function. Hooking involves replacing a legitimate function pointer with one provided by malware. So, for example, the malware might replace (or hook) a file system function with one that behaves perfectly normally except when it comes to the areas of the filesystem where the malware lives; in that case, it returns information that suggests the files aren’t there. Any software that uses the kernel for filesystem access will never know the rootkit is present.

Obviously, the simplest way of blocking a rootkit would be to prohibit this process by marking kernel memory as read-only. But there are two problems with this approach. For starters, the ability to perform a kernel hook has many legitimate uses, such as when a new input device hooks into the portions of the kernel that handle mouse or keyboard input. The other problem is that the function pointers are scattered around the kernel’s memory footprint, and are sometimes created and destroyed as the kernel creates new objects, like networking sockets. Locking the entire kernel down as read-only would cripple the operating system.

(The authors call this problem the “protection granularity gap.” It’s possible to lock down a page of memory that contains function pointers but, in doing so, you invariably lock down some dynamic data, which causes problems. There is currently no technology that provides a fine enough granularity to lock down the parts of a memory page that contain the pointers.)

The authors tackle these two problems separately. To identify legitimate kernel hooks, the authors run a clean version of the operating system (in this case, Ubuntu 8.04) under a modified version of the QEMU emulator. Their modified version of QEMU tracks all the kernel hooks that take place in the course of normal operations, and generates a unique signature for each of them. That allows this activity to be recognized and allowed during normal operations.

They solve the granularity gap by creating a shadow copy of every kernel hook they’ve identified, all collected in contiguous memory pages that can be marked as read-only and protected by a custom version of the Xen hypervisor. Their original location in the kernel gets replaced by a jump statement that shifts execution to what they call a trampoline, which bounces execution to the safe shadow copy, and then returns it to the original execution point. The variable-sized x86 instructions make this a bit more challenging than it might otherwise be, but the authors manage to compensate.

If the hook is used simply to execute the function, everything should take place as normal. If it’s used to replace the hook, the protected memory page invokes the Xen hypervisor. Their modified version checks the signature of the action, comparing it against the list generated when the kernel ran under QEMU. If the activity is recognized, it’s allowed to go forward. If not, the shadow copy of kernel hooks is kept unmodified.