IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 1
Detecting Kernel-level Rootkits
using Data Structure Invariants
Arati Baliga, Vinod Ganapathy, and Liviu Iftode
Abstract—Rootkits affect system security by modifying kernel data structures to achieve a variety of malicious goals. While early rootkits
modified control data structures, such as the system call table and values of function pointers, recent work has demonstrated rootkits that
maliciously modify non-control data. Most prior techniques for rootkit detection have focused solely on detecting control data modifications
and, therefore, fail to detect such rootkits.
This article presents a novel technique to detect rootkits that modify both control and non-control data. The main idea is to externally
observe the execution of the kernel during an inference phase and hypothesize invariants on kernel data structures. A rootkit detection
phase uses these invariants as specifications of data structure integrity. During this phase, violation of invariants indicates an infection. We
have implemented Gibraltar, a prototype tool that infers kernel data structure invariants and uses them to detect rootkits. Experiments show
that Gibraltar can effectively detect previously-known rootkits, including those that modify non-control data structures.
Index Terms—Kernel-level rootkits, non-control data attacks, invariant inference, static and dynamic program analysis.
!
1 I performance by modifying memory management meta data
and those that affect the output of the pseudo random number
K rnel-level rootkits are a form of malicious software that
compromise the integrity of the operating system. Such
rootkits stealthily modify kernel data structures to achieve
generator by contaminating entropy pools. Non-control data
presents a much larger attack surface than control data, and
these rootkits demonstrate the ease with which attackers can
a variety of malicious goals, which may include hiding
subtly modify non-control data structures to subvert the kernel.
malicious user space objects, installing backdoors, logging
To counter rootkits that modify non-control data, Petroni et
keystrokes and disabling firewalls. Recent studies have shown
al. [34] proposed a detection architecture in which kernel data
a phenomenal increase in malware that use stealth techniques
structures are periodically compared against a set of integrity
commonly employed by rootkits. For example, MacAfee Avert
specifications. These specifications codify semantic properties
Labs [5] reported a 600% increase in the number of rootkits
of kernel data structures; the detection architecture uses spec-
in the three year period from 2004-2006. The most recent list
ification violation as an indicator of rootkit behavior. While
of threat predictions [6], also by MacAfee, contains several
this approach has the advantage of detecting sophisticated
recent examples of trojan horses that were used to commit
rootkits, it also poses a new challenge—that of developing
bank fraud. These trojan horses used stealth techniques to
integrity specifications. High-quality specifications can possi-
hide on a victim’s system, disable anti-virus software, prevent
bly be supplied by a team of experts who have a detailed
signature updates, or include the victim’s system into a botnet.
understanding of kernel data structure semantics. However,
The increase in the number and complexity of rootkits
commodity operating system kernels typically maintain several
can be attributed to the large and complex attack surface
hundred, complex data structures. Consequently, specification
that the kernel presents. The kernel manages several hundred
writers could either fail to supply certain integrity specifica-
heterogeneous data structures, most of which are critical to
tions (e.g., because they are unaware of certain specifications)
its correct operation. A rootkit can subvert kernel integrity by
or fail to realize how a rootkit could exploit them.
subtly modifying any of these data structures. In particular,
We propose a novel approach that automatically generates
kernel data structures that hold control data, such as the
kernel data structure integrity specifications. In our approach,
system call table, jump tables and function pointers, have
these integrity specifications take the form of data structure
long been a popular target for attack by rootkits. However,
invariants—properties that must hold for the lifetime of a data
recent work has demonstrated rootkits that achieve a variety
structure. The key idea is to monitor the values of kernel data
of malicious goals by modifying non-control data in the kernel.
structures during an inference phase in order to hypothesize
For example, Petroni et al. [34] demonstrated a rootkit that hid
invariants that are satisfied by these data structures. These
malicious user space processes by manipulating linked lists
invariants can encompass both control and non-control data
used by the kernel for bookkeeping. In our previous work,
structures. For example, an invariant could state that the values
we also have demonstrated rootkits that alter non-control data
of elements of the system call table are a constant (an example
in the kernel [9], such as rootkits that degrade application
of a control data invariant). Similarly, an invariant could state
that all the elements of the running-tasks linked list (used
• All three authors are with the Department of Computer Science, Rutgers by the Linux kernel for process scheduling) are also elements
University. E-mail: {aratib,vinodg,iftode}@cs.rutgers.edu
of the all-tasks linked list that is used by the Linux kernel
This article is a revised and expanded version of work published in the 24th for process accounting (an example of a non-control data
Annual Computer Security Applications Conference [8].
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 2
invariant) [34]. These invariants are then checked during a administrator. Such rootkits can be detected using a number of
rootkit detection phase, in which violation of an invariant is prior tools, such as Tripwire [25] and Strider Ghostbuster [10],
assumed to indicate the presence of a rootkit. and several commercial tools (e.g., [1, 14, 45]). Most of
To evaluate the proposed approach, we built Gibraltar, a these tools operate by comparing the kernel’s view of user-
rootkit detection tool that automatically infers invariants on space object (e.g., a cryptographic hash of system utilities)
kernel data structures. Gibraltar periodically captures snap- against known values. Any inconsistencies are indicative of
shots of kernel memory via an external PCI card to reconstruct rootkits. Recent work [11] has aimed to prevent such rootkits
kernel data structures. It adapts Daikon [19], an invariant by modifying hard disk drives to disallow modifications to
inference tool for user-space application programs, to infer critical system utilities unless such changes are authorized,
invariants on kernel data structures. In experiments with 23 e.g., by physically connecting an authentication token to the
rootkits, including those that modify non-control data, we hard disk.
found that Gibraltar detected all rootkits while imposing a In contrast to these techniques, which focus on protecting
runtime monitoring overhead of under 0.5%. user-space objects, Gibraltar focuses on rootkits that operate
Our experiments demonstrate the feasibility of automated by modifying the kernel. These techniques may be used
generation of integrity specifications for kernel data structures. together with Gibraltar to provide comprehensive protection.
The invariants inferred by our approach can serve as the start- Protecting kernel code and critical data. Modern rootkits
ing point for a team of kernel experts, who can further refine operate by directly infecting the operating system kernel. For
these specifications. However, we found that the automatically- example, a rootkit can modify kernel code or the system call
generated invariants were quite precise. For example, during a table to instead execute malicious code. Prior work to detect
42 minute rootkit detection phase, we observed only 82 spuri- such rootkits falls under three broad categories:
ous invariants out of a total of 236,444 automatically-inferred • Kernel module validators. Rootkits often spread as kernel
invariants. Nevertheless, the semantic quality of automatically- modules that affect kernel code and data after they have been
generated invariants remains unknown. For example, it may be loaded. Such kernel modules can also use techniques such
possible for a kernel expert to provide a small set of invariants as polymorphism to evade detection by virus scanners. Static
that encompasses several hundred invariants inferred by our analysis of kernel modules (e.g., using symbolic execution [26,
approach. Future work can further explore how the output of 49]) can detect these rootkits, but conservative approximations
our automatic approach compares against invariants written by made by static analysis may result in false alarms. Instead
a kernel expert. of checking kernel modules, Gibraltar focuses on observing
In summary, this article makes the following contributions: and validating data modifications to kernel memory, including
• Rootkit detection via invariant inference. It proposes a those made by kernel modules.
novel approach that detects rootkits by identifying violations • Hardware-based integrity monitors. The recent addition
of automatically-inferred kernel data structure invariants. Sec- of trusted platform modules (TPM) to commodity hardware
tion 3 presents an overview of our approach, and presents has allowed the development of protocols to verify the in-
examples of both control and non-control data attacks that tegrity of the software stack executing on a remote machine
were detected in our experiments. (e.g., [21, 38, 39, 41]). Such techniques can detect certain
• Design and implementation of Gibraltar. Section 4 kinds of rootkits, in particular, those that modify kernel code
presents the design and implementation of Gibraltar, a pro- and critical data structures. However, we are not aware of
totype tool that uses the above approach for rootkit detection. the use of trusted computing to detect rootkits that modify
• Evaluation on real-world rootkits. Section 5 presents a arbitrary kernel data structures.
comprehensive evaluation of Gibraltar on 23 rootkits, which Secure co-processors allow remote monitoring of physical
affect both control and non-control data structures. Gibraltar memory, and have been used to build rootkit detectors [33, 51].
can detect all of them with negligible monitoring overhead. For example, Co-Pilot uses a co-processor [33] to periodically
fetch and ensure the integrity of physical memory pages
that contain kernel code and critical data. Gibraltar uses a
2 R W
similar technique (a PCI-card) to periodically fetch snapshots
The evolution of rootkits and the techniques to detect them of kernel memory from a target machine, reconstruct data
has traditionally been an arms race between attackers and structures in these snapshots and check invariants.
defenders. Early rootkits operated by replacing system binaries • Virtual machine introspection. Virtual machine monitors
on disk with infected versions. Rootkits have more recently offer an alternative technique to monitor the integrity of
evolved to infecting the system by modifying kernel code, memory pages that contain kernel code and data. In this
control data, and non-control data. Gibraltar is the latest in a technique, called virtual machine introspection [7, 22, 27], the
long list of rootkit detection tools, but is one of only two virtual machine monitor fetches and forwards memory pages
techniques that can detect malicious modifications of non- from a guest operating system to a rootkit detector, which can
control data. analyze the integrity of these pages. This technique has also
Defending system utilities. Rootkits can hide malicious user- been used to prevent rootkits from modifying kernel code [36].
space objects by replacing key system utilities with infected Gibraltar can potentially be adapted to use virtual machine
versions. For example, a rootkit can replace the ps utility introspection.
with a version that hides a malicious process from a system
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 3
Attack Example invariants violated by the attack
(a) Entropy pool contamination (§3.1) poolinfo.tap1 ∈ {26, 103}, poolinfo.tap2 ∈ {20, 76}, poolinfo.tap3 ∈ {14, 51}, poolinfo.tap4 ∈ {7, 25},
poolinfo.tap5 == 1
(b) Process hiding (§3.2) run-list ⊆ all-tasks
(c) Adding binary formats (§3.3) L(formats) == 2
(d) Resource wastage (§3.4) zone table[1].pages min==255, zone table[1].pages low==510, zone table[1].pages high==765
(e) Intrinsic denial of service (§3.5) max threads == 14, 336
(f) Disabling firewalls (§3.6) nf hooks[2][1].next.hook == 0xc03295b0
(g) Disabling PRNG (§3.7) random fops.read == 0xc028bd48, urandom fops.read == 0xc028bda8
(h) Altering real-time clock (§3.8) rtc fops->ioctl == 0xc01a39e0
(i) Defeating signature scans (§3.9) kmem fops->read == 0xc0186e00, mem fops->read == 0xc0186c40
Fig. 1. Summary of the attacks discussed in Section 3 and kernel data structure invariants violated by these attacks.
Protecting kernel hooks. Rootkits have recently evolved to attacks that employ stealth techniques. These attacks either
hijacking control of the kernel by modifying kernel control modify non-control kernel data (cf. Attacks 1-5) or modify
data, such as function pointers. This attack technique, called kernel control data (cf. Attacks 6-9). Although these attacks
hooking, has spurred research on tools to detect and protect were implemented using the Linux kernel, similar attacks
kernel hooks [35, 46, 47, 50]. For example, SBCFI [35] peri- should also be applicable to other operating systems. Gibraltar
odically scans hooks in kernel memory and ensures that they successfully detects each of the attacks discussed in this
point to pre-approved locations, e.g., addresses of exported section. Where applicable, we discuss existing tools that can
kernel functions. As discussed in Section 1, Gibraltar infers detect each attack.
invariants over both control and non-control data, and can For each of the attacks presented in this section, we also
therefore detect rootkits that use hooking to hijack kernel describe a data structure invariant (automatically inferred by
control flow. Gibraltar by observing the execution of an uncompromised
Protecting non-control data. Recent research has demon- kernel) that is violated by the attack (see Figure 1). In addition,
strated rootkits that affect system security by modifying ar- we also describe the semantic meaning of each invariant,
bitrary kernel data [9, 34]. In contrast to rootkits that hijack i.e., the reason why a data structure satisfies the property
kernel control flow, these rootkits operate by modifying kernel specified by the invariant in an uncompromised kernel. The
data structures to hide user-space processes [34], affect appli- invariants listed in this section are examples drawn from
cation performance, or affect the output of the kernel’s pseudo several thousand invariants that are automatically inferred by
random number generator [9]. The rootkit detection tools Gibraltar. Particularly noteworthy in the examples below is
discussed above do not monitor non-control data structures the heterogeneity of the data structures over which Gibraltar
and therefore cannot detect such rootkits. infers invariants. Although these invariants can be examined,
Petroni et al. [34] first proposed an architecture to detect interpreted and refined by a security expert, Gibraltar, by
rootkits that affect non-control data structures. Their archi- default, automatically uses these invariants as specifications
tecture periodically compares kernel data structures against of data structure integrity.
integrity specifications that describe semantic properties of
kernel data structures. These specifications must hold during 3.1 Attack 1: Entropy Pool Contamination
normal execution of the kernel. Violation of any of these
specifications indicates the presence of a rootkit. Their paper The kernel uses the pseudo random number generator (PRNG)
demonstrated this approach by using two sets of specifications to obtain randomness needed to seed several other security-
(developed manually) to detect two rootkits. critical applications. The goal of the entropy pool contamina-
Gibraltar extends the approach developed by Petroni et al. tion attack [9] is to contaminate entropy pools and associated
by automating the extraction of integrity specifications. It does polynomials used by the PRNG, so as to degrade the quality
so by applying automated techniques that observe kernel data of random numbers that it generates.
structures over a period of time and hypothesize invariants, Attack. The PRNG uses the primary and secondary entropy
violations of which are then used in an anomaly detection pools to generate random numbers. The primary pool derives
phase to identify rootkits. In this respect, Gibraltar closely entropy from external events such as keystrokes, mouse move-
resembles prior work on software engineering aids that use a ments, disk and network activity. As a request arrives for a
similar approach to detect programming errors [13, 16, 24]. random number, the kernel extracts bytes from the primary
Recovery. Finally, recent work has proposed techniques to pool and moves them to the secondary pool. Bytes extracted
automatically recover from system infections and errors, in- from the secondary pool are in turn used to provide random
cluding rootkits [17, 20, 32, 42]. In contrast to these tech- numbers to kernel functions and user-level applications.1
niques, Gibraltar only performs detection and cannot recover To ensure that the numbers generated by the PRNG are
from rootkit infections. pseudo random, the contents of the pools are updated using a
stirring function each time bytes are extracted from the pools.
The stirring function uses a polynomial whose coefficients are
3 R D I I
1. For ease of presentation, we have simplified some details of the attacks
This section motivates the use and effectiveness of data presented in Section 3. For full details of each attack, please consult the
structure invariants at detecting rootkits by presenting nine original references.
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 4
specified in five integer fields of a struct poolinfo data Attack. This rootkit adds a malicious handler to support
structure, namely tap1, tap2, tap3, tap4 and tap5. This a new binary format. The binary formats supported by a
attack zeroes the coefficients of the polynomial, which renders system are maintained by the kernel in a global linked list
ineffective part of the algorithm used to extract bytes from the called formats. The binary handler, specific to a given binary
pool. It also writes zeroes constantly into the entropy pools. format, is also supplied when a new format is registered.
Consequently, the numbers generated by the PRNG are no When a new process is created on the system, the kernel
longer random. creates the process address space, sets up credentials and in
Invariants. Figure 1(a) shows the invariants that Gibral- turn calls the function search binary handler, which is
tar identifies for the coefficients of the polynomial that responsible for loading the binary image of the process from
is used to stir entropy pools in an uncompromised kernel the executable file. This function iterates through the formats
(the poolinfo data structure shown in this figure is repre- list to look for an appropriate handler for the binary that it is
sented in the kernel by one of random state->poolinfo attempting to load. As it traverses this list, it invokes each
or sec random state->poolinfo). The coefficients are handler in it. If a handler returns an error code E, the
initialized upon system startup, and must never be changed kernel considers the next handler on the list; it continues to
during the execution of the kernel. The attack violates these do so until it finds a handler that returns the code S.
invariants when it zeroes the coefficients of the polynomial. This attack works by inserting a new binary format in the
Gibraltar detects this attack when the invariants are violated. formats list and supplying the kernel with a malicious handler
that returns the error code E each time it is invoked.
3.2 Attack 2: Process Hiding Since the new handler is inserted at the head of the formats
The goal of this attack is to hide a malicious user-space process list, the malicious handler is executed each time a new process
from the system utilities, such as ps. The attack operates is executed.
by modifying the contents of the kernel linked lists used for Invariants. Gibraltar infers the invariant shown in Figure 1(c)
process accounting and scheduling [2, 34]. on the formats list on our system, which has two registered
Attack. This attack relies on the fact that process account- binary formats. The size of the list is constant after the system
ing utilities, such as ps, and the kernel’s task scheduler starts, and changes only when a new binary format is installed.
consult different process lists. The process descriptors of all As this attack inserts a new binary format it changes the length
tasks running on a system belong to a linked list called the of the formats list violating the invariant in Figure 1(c);
all-tasks list (represented in the kernel by the data struc- consequently, Gibraltar detects this attack.
ture init tasks->next task). This list contains process
descriptors headed by the first process created on the system. 3.4 Attack 4: Resource Wastage
The all-tasks list is used by process accounting utilities. In This attack creates artificial pressure on the memory subsys-
contrast, the scheduler uses a second list, called the run-list tem [9], thereby forcing the memory management algorithms
(represented in the kernel by run queue head->next), to to constantly free memory by swapping pages to disk. In spite
schedule processes for execution. of the availability of free memory, this memory is not used
The process hiding attack removes the process descriptor of either by the kernel or by user-space applications. Continuous
a malicious user-space process from the all-tasks list (but swapping to disk also affects the performance of the system.
not from the run-list list). This ensures that the process is Attack. The kernel’s memory management unit ensures that
not visible to process accounting utilities, but that it will still there are always free pages in memory to fulfill allocation re-
be scheduled for execution. quests made both from the kernel and user-space applications.
Invariant. Figure 1(b) presents the invariant automatically To do so, it employs memory balancing algorithms, which
discovered by Gibraltar, which states that all elements of use three watermarks to estimate memory pressure, namely,
run-list must also be elements of the all-tasks list. When the fields pages min, pages low and pages high, of a
a rootkit attempts to remove a task from the all-tasks struct zone struct data structure (zone table[1], in
list, this invariant is violated, and is therefore detected by Figure 1(d)). When the number of free pages in the system
Gibraltar. This attack was previously described by Petroni et drops below the pages low watermark, the kernel asyn-
al. [34] as an example of a non-control data attack. They chronously swaps unused pages to disk. This process continues
also describe an invariant enforcement tool to detect such until the number of pages reaches the pages high water-
attacks. However, in their approach, data structure invariants mark. In contrast, if the number of free pages available drops
were supplied manually by a security expert. Gibraltar extends below the pages min watermark, the kernel synchronously
Petroni et al.’s work by developing an automated approach to swaps pages to disk.
produce data structure invariants. This attack manipulates the three watermarks and sets their
values close to the number of free pages in the system.
3.3 Attack 3: Adding Binary Formats Consequently, the number of free pages frequently drops
The goal of this attack is to invoke malicious code each time below the pages min and pages low watermarks, forcing
a new process is created on the system [44]. While rootkits the kernel to continuously swap pages to disk, thereby creating
typically achieve this form of hooking by modifying kernel synthetic memory pressure in the system.
control data, such as the system call table, this attack works Invariants. Gibraltar identifies the invariants shown in Fig-
by inserting a new binary format into the system. ure 1(d) for the three watermarks. These values are initialized
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 5
upon system startup, and typically do not change in an uncom- enforces firewall rules through the netfilter framework.
promised kernel. The attack sets the pages min, pages low Pointers to the netfilter hooks are stored in a global table
and pages high watermarks to 210, 000, 215, 000 and called nf hooks. This attack overwrites the hooks for the IP
220, 000 respectively. The values of these watermarks are close protocol, and instead sets them to point to the attack function,
to 225,280, which is the total number of pages available on thereby effectively disabling the firewall. The table where the
our system. Gibraltar detects this attack because the invariants firewall rules are stored is unaltered and therefore displayed
shown in Figure 1(d) are violated. by iptables when the user manually inspects the firewall.
Invariants. Gibraltar inferred the invariant shown in Fig-
3.5 Attack 5: Intrinsic Denial of Service ure 1(f) for netfilter. The attack overwrites the hook with
The goal of this attack is to degrade application performance the attack function, thereby violating the invariant that function
by throttling the number of processing threads that an ap- pointer nf hooks[2][1].next.hook is a constant.
plication can create to perform tasks in parallel. It works As this attack modifies kernel function pointers, it can also
by corrupting data used by the clone system call in Linux. be detected by SBCFI [35], which automatically extracts and
As a result, tis attack stealthily causes a measured degree of enforces kernel control flow integrity. In fact, function pointer
denial of service because resources beyond a certain threshold invariants inferred by Gibraltar implicitly determine a control
become temporarily unavailable to applications, which in turn flow integrity policy that is equivalent to SBCFI. However,
experience a slowdown. in contrast to SBCFI, Gibraltar can also detect non-control
Attack. The kernel relies on the process creation mechanisms attacks (e.g., Attacks 1-5).
to satisfy user requests. In particular, server applications are
designed to be multi-process or multi-threaded; they constantly 3.7 Attack 7: Disabling the PRNG
create new processes/threads to service requests obtained from
This attack overwrites the addresses of the functions registered
clients. This attack changes the max threads variable used
with the virtual file system layer by the PRNG [9]. The
by the clone system call. This variable is used to check if
overwritten values point to functions that always return zero or
the total number of processes created on the system exceeds
an attacker-defined sequence when random bytes are requested
the total number of processes that can possibly be created.
from the PRNG; the PRNG’s functions are never executed.
This check within the clone system call is incorporated to
Attack. The kernel provides two devices /dev/random and
curtail fork bombs. The max threads variable is in the
/dev/urandom from which random numbers can be read.
static kernel has a default value of 14, 336, which is therefore
The data structures used to register the device functions
an upper limit on the total number of processes that can
are random fops and urandom fops, both of which are
exist on a system. During our attack, the total number of
variables of type struct file operations. These data
processes existing at the time of attack was 33. The attack
structures have function pointers to the various functions pro-
changes the value of max threads to 40, thereby severely
vided by the PRNG. The attack replaces the genuine function
limiting the number of new processes that can be created
pointers for the read function within these data structures.
on the system. Once the number of processes exceeds 40,
After the attack has infected the kernel, every byte read from
subsequent system calls to create new processes receive an
the two devices simply returns a zero. The original PRNG
error message, indicating the temporary unavailability of the
functions are never called.
resource. Although applications are typically programmed to
Invariants. The invariants inferred by Gibraltar on our system
handle this error code, they experience a slowdown because
for the random fops and urandom fops are shown in Fig-
this attacks limits their ability to create new processes. This
ure 1(g). The attack code changes the values of the above two
attack resembles an intrinsic denial of service attack, where
function pointers, violating the invariants. As with Attack 6,
the service is unable to function at its full capacity.
this attack can also be detected using SBCFI.
Invariants. The invariant inferred by Gibraltar on the
max threads variable is shown in Figure 1(e). The attack
modifies the value of this variable and violates the invariant 3.8 Attack 8: Altering Real-time Clock Behavior
and is therefore detected by Gibraltar. The real-time clock (RTC) on a system provides the system
time and features, such as setting an alarm clock, for scheduled
3.6 Attack 6: Disabling Firewalls execution of applications at later points in time. This attack
The goal of this attack is to stealthily disable firewalls installed alters the behavior of the real time clock so that alarms
on the system [9]; a user is unable to determine that fire- registered by certain applications, such as anti-virus software
walls have been disabled using the iptables utility. Instead, and other intrusion detection systems, are never triggered. This
iptables shows the firewall rules that were created for the disables scheduled virus scans and other defense activities
system, and the firewall appears to be enabled. carried out on the system, thus making it vulnerable to attacks.
Attack. This attack overwrites hooks in the Linux netfilter Attack description. The real time clock in a computer system
framework, which is a packet filtering framework in the is powered by a small battery or accumulator and continues to
Linux kernel. It provides hooks at multiple points in the tick even when the system is turned off. It can be programmed
networking stack, and was designed for kernel modules to to issue periodic interrupts or to issue an interrupt when the
register callbacks for packet filtering, packet mangling and net- clock reaches a certain value. The Linux kernel uses the RTC
work address translation. The iptables command line utility to retrieve the date and time. The RTC driver provides the
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 6
device /dev/rtc to applications, which they can use to access
the RTC. The system time can be set by an administrator using
the clock utility.
The RTC is used by applications that rely on periodic
execution of tasks. A classic example of such an application
is an anti-virus software. A user typically schedules complete
disk scans for viruses and worms when the system is not in
use because it is a time-consuming process. For example, a
periodic scan of the system might be scheduled to run every
Sunday morning at 3:00am. The anti-virus program relies on
the RTC to issue an interrupt when the clock reaches this time.
The goal of this attack is to disable the scheduled execution
of the anti-virus program. Applications set such an alarm
by using the ioctl system call on the /dev/rtc device.
This attack works by overwriting the function pointer for the
ioctl system call, which is stored within the data structure Fig. 2. Design of Gibraltar. Boxes with solid lines show
rtc fops. The malicious function can selectively disable components of Gibraltar, while boxes with dashed lines show
alarms only for certain applications of interest, such as the the input or output of different components.
anti-virus software, thereby other regular applications function
flawlessly. The system continues to be vulnerable to attacks
because anti-virus and intrusion detection systems do not run Invariants. The /dev/mem and /dev/kmem devices register
at their scheduled times. This attack makes the system more their respective read function handlers with the Linux VFS
vulnerable to future attacks. layer. The invariants inferred by Gibraltar for the respective
Invariants. The invariant inferred by Gibraltar on the data structures are shown in Figure 1(i). When the attack
rtc fops->ioctl variable is shown in Figure 1(h). Since the replaces these function handlers to point to its own malicious
attack installs its own ioctl handler, this invariant is violated functions, the invariants shown in Figure 1(i) are violated and
and the attack is detected. As with Attack 6, this attack can hence the attack is detected.
also be detected by SBCFI.
4 D I G
3.9 Attack 9: Defeating In-memory Signature Scans Gibraltar must be physically isolated from the monitored sys-
This attack defeats malware detectors that use in-memory tem because it aims to detect kernel-level rootkits. In our im-
signature scans by providing them with a fake view of memory. plementation, Gibraltar executes on a separate machine, called
The attack achieves this goal by installing malicious read func- the observer, and monitors the execution of the target machine.
tions for the /dev/kmem and the /dev/mem devices, which Both the observer and the target are interconnected via a
provide interfaces for reading and writing to the kernel virtual secure back-end network using the Myrinet PCI intelligent
address space and the system physical memory, respectively. network cards [3].2 The back end network allows Gibraltar
Attack description. /dev/mem and /dev/kmem are character to remotely access the target kernel’s physical memory, from
devices provided by a Linux system that allow read and write which it infers data structure invariants. Both coprocessor
access to system memory. Only privileged users are allowed to and VMM-based monitors use similar techniques to read the
read or write to these devices. The device /dev/kmem accesses target’s memory. Consequently, Gibraltar can be easily adapted
data from the kernel virtual memory, while /dev/mem reads to work with either infrastructure.
data from the system physical memory. Reads from these Gibraltar operates in two modes, namely, an inference mode
devices return the memory contents existing at the respec- and a detection mode. In the inference mode, Gibraltar infers
tive memory locations, while writes allow patching memory invariants on data structures of the target’s kernel. Inference
with supplied data. Rootkits have also used the /dev/kmem happens in a controlled environment on an uncompromised
interface to patch the running kernel (e.g., SucKIT [40]). target, e.g., a fresh installation of the kernel on the target
The high-level objective of this attack is similar to that of the machine. In the detection mode, Gibraltar checks whether
Shadow Walker rootkit [43], which utilizes the split TLB ar- the data structures on the target’s kernel satisfy the inferred
chitecture of the Intel Pentium processor to modify the kernel’s invariants.
page-fault handler to return fake memory contents. However, Gibraltar consists of four key components, as shown in
this attack achieves the same objective by overwriting function Figure 2. The page fetcher responds to requests by the data
pointers registered by the /dev/mem and /dev/kmem devices.
These function pointers are stored in the virtual file system 2. Prior work [37] shows that both PCI and co-processor-based techniques
to read the contents of main memory can be bypassed for AMD processors.
layer in the data structure kmem fops and mem fops, of type These attacks operate by inserting illegal entries into the memory map of
struct file operations. The malicious handlers for the the memory controller (i.e., the northbridge). In the worst case, attempts to
read function can present a counterfeit view of the memory read the contents of memory on a compromised machine can return values
chosen by the attacker. Gibraltar can also operate with other techniques that
pages, thus thwarting all detection software that uses these can securely fetch memory pages from the target machine, e.g., VMM-based
interfaces to scan memory for malware signatures. monitors [22].
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 7
structure extractor to fetch kernel memory pages from the Input: (a) R: addresses of roots;
target. The data structure extractor, in turn, analyzes raw (b) Data structure definitions.
Output: Set of all data structures reachable from R.
physical memory pages and extracts values of data structures. 1. worklist = R;
To do so, it requires data type definitions of the target’s 2. visited = φ;
kernel and a set of root symbols that it uses to traverse the 3. snapshot = φ;
4. while worklist is not empty do
target’s kernel memory pages. Both these inputs are obtained 5. addr = remove an entry from worklist;
via an offline static analysis of the source code of the kernel 6. visited = visited ∪ {addr};
version executing on the target machine. The data structure 7. M = physical memory page containing addr;
8. obj = object at address addr in M;
extractor outputs a (partial) kernel snapshot, i.e., the set of 9. snapshot = snapshot ∪ value of obj;
kernel data structures in the physical memory pages obtained 10. foreach pointer p in obj do
from the target.3 The invariant generator processes snapshots 11. if p visited
12. worklist = worklist ∪ {p};
and hypothesizes likely invariants, which represent properties 13. return snapshot;
of individual data structures (i.e., memory objects), as well
Fig. 3. Algorithm used by the data structure extractor.
as collections of data structures. Examples of objects include
scalars, such as integer variables or array elements, and struct task struct {...
fields within aggregate data structures, such as C structs. struct list head
Container(struct task struct,run list) run list;
Collections of data structures represent objects of the same ...}
type grouped together e.g., linked lists. During detection, the
Fig. 4. An example showing the Container annotation. The
monitor uses the inferred invariants as specifications of kernel
field run list within the structure task struct points to
data structure integrity. The monitor raises an alert when
the run list field of another task struct.
a kernel data structure invariant is violated. The following
sections elaborate on the design of each of these components.
structures. We automatically extracted 1,292 type definitions
4.1 The Page Fetcher by analyzing the source code of the target’s Linux-2.4.20
Gibraltar executes on the observer, which is isolated from the kernel using a CIL module [30].
target system. Gibraltar’s page fetcher is a component that Figure 3 presents the algorithm that the data structure
takes a physical memory address as input, and obtains the extractor uses to locate data structures in physical memory.
corresponding memory page from the target. The target runs The extractor first adds the addresses of the roots to a worklist
a Myrinet PCI card with an enhanced version of the original and issues requests to the page fetcher for memory pages
firmware, which directly interprets and services requests from containing the roots. It extracts the values of the roots from
the page fetcher, without intervention from the target’s kernel. these pages, and uses their type definitions to identify pointers
For each request, the firmware on the target initiates a DMA to previously unseen data structures. For example, if a root
request for the requested page, and sends the contents of the is a C struct, the data structure extractor adds all pointer-
physical page to the observer upon completion of the DMA. valued fields of this struct to the worklist to locate more
data structures in the kernel’s physical memory. This process
continues in a recursive fashion until the traverser identifies
4.2 The Data Structure Extractor all the data structures target kernel’s memory reachable from
The data structure extractor analyzes raw physical memory its roots. A complete set of data structures reachable from
pages received from the page fetcher and outputs kernel snap- the roots is called a snapshot. The data structure extractor
shots. It has two key responsibilities: locating (Section 4.2.1) periodically probes the target and outputs snapshots.
and naming (Section 4.2.2) data structures. The data structure extractor may require assistance when
it encounters certain pointer-valued fields during traversal. A
4.2.1 Locating data structures particularly important case is when the traverser encounters
The extractor uses a set of root symbols and type definitions to pointers to linked lists. In the Linux kernel, linked lists are
locate data structures in raw physical memory pages received implemented using the list head data structure. Kernel
from the target. Root symbols denote kernel data structures objects that must be organized as a linked list simply include
whose physical memory locations are fixed. All data structures the list head data structure. Figure 4 presents an example
on the target’s heap are reachable from root symbols. In of a task struct, in which the field run list is of
our implementation for targets running Linux, we used the type list head. Objects of type task struct are linked
symbols in the System.map file of the target’s kernel as the together as a list using the next and prev fields, which are
set of roots. Second, it uses a set of type definitions of the members of the list head structure. The kernel provides
data structures in the target’s kernel. Type definitions are used functions to add, delete, and traverse list head data struc-
as described below to recursively identify all reachable data tures. To traverse a list of task struct structures, the kernel
locates the list head structures within the task struct
3. The snapshot is partial because our implementation of the data structure structure in the list and computes a pointer to the next
extractor currently ignores dynamically-allocated arrays, opaque pointers and task struct object in the list using pointer arithmetic. This
untagged unions (see Section 4.2). Moreover, the data structure extractor may
encounter inconsistent data structures views because it fetches memory pages is a commonly-used idiom in the Linux, and is codified in the
in an asynchronous fashion. container of macro.
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 8
We added code annotations to assist the data structure ex- useful during invariant inference, when it helps identify cases
tractor when it encounters pointer-valued fields, such as those where the same name may represent different data structures
in the list head structure. The Container annotation, in multiple snapshots. Such a scenario may arise because
shown in Figure 4, explicitly specifies the type of the container of deallocation and reallocation. For example, suppose that
data structure (struct task struct) and the field within the kernel deallocates (and reallocates, at a different ad-
this type (run list) that the list head pointers point to. dress) the head of the all-tasks linked list. As the name
The extractor uses this annotation when it encounters the init tasks->next task will be associated with different
run list field, and locates the container task struct data virtual memory addresses before and after allocation, it repre-
structure. The Container annotations therefore disambiguate sents different data structures.
the semantics of the list head pointer to the data structure Each of these naming schemes has distinct advantages.
extractor. In our experiments, we annotated all 163 occurrences Pathnames are useful to generate meaningful invariants for
of the list head data structure in the Linux-2.4.20 kernel. data structures whose paths persist across multiple snapshots.
Gibraltar may also require assistance to disambiguate opaque They have the important advantage of being portable across
pointers (void *), dynamically-allocated arrays and untagged machine reboots: the data structure can be identified using
unions. For example, the extractor would require the length of its path name. In contrast, names based on virtual memory
a dynamically-allocated array in order to traverse and locate addresses are not portable across machine reboots. However,
objects in the array. Our current prototype does not support virtual memory addresses can be used to express invariants
traversal of dynamic arrays, opaque pointers and untagged for both persistent and transient data structures; Section 4.5
unions. However, recent work has addressed this shortcoming discusses this issue in further detail.
using pointer analysis of the kernel [12].
The algorithm depicted in Figure 3 may encounter in- 4.3 The Invariant Generator
consistencies, such as pointers to non-existent objects, as it
During invariant inference, Gibraltar uses the output of the
processes raw physical memory pages. This is because the
data structure extractor to infer likely data structure invariants.
page fetcher obtains pages from the target asynchronously,
These invariants are used as specifications of data integrity.
i.e., without halting the target. Consequently, operations such
We adapted the Daikon [19] tool to infer likely data structure
as deallocation may invalidate pointers. Such invalid pointers
invariants. Daikon is a software engineering aid that infers
are problematic because the data structure extractor will incor-
likely program invariants using dynamic program analysis.
rectly fetch and parse the (invalid) memory region referenced
Daikon’s front-end instruments programs to emit a trace file as
by the pointer. In turn, this memory region may contain values
it executes. Each trace file contains the values of variables at
that resemble pointers, which the data structure extractor
selected program points, such as the entry points and exits of
will recursively follow to identify more spurious objects. To
functions. Several trace files may be obtained by executing the
prevent our implementation from extracting a large number of
program on a test suite, and are then fed to Daikon’s inference
spurious objects, we placed an upper bound on the number
engine, which analyzes these traces to infer likely invariants—
of objects traversed by the extractor (as in prior work [35]).
properties of variables that hold across all executions of the
In our experiments, we found that on an idle system, the
program. Daikon generates invariants that conform to one
number of data structures in the kernel varies between 40,000
of several templates. For example, the template x == const
and 65,000 objects. We therefore placed an upper bound of
checks whether the value of a variable x equals a constant
150,000. The data structure extractor aborts the collection of
value const, where const represents a symbolic constant.
new objects when this threshold is reached. In our experiments,
Daikon also infers invariants over collections of objects. For
this threshold was reached only when the system was heavily
example, it may infer that the field bar of all objects of type
loaded. On average, the data structure extractor takes about
struct foo has the value 5.
100 seconds to gather a single kernel snapshot. Note that by
We developed a new front-end to convert kernel snapshots
placing an upper bound on the number of extracted objects,
into a format that can be processed by Daikon’s inference
the data structure extractor may fail to extract certain kernel
engine. Our front-end converts each snapshot into the equiv-
data structures. In turn, Gibraltar may fail to infer invariants
alent of a single Daikon trace file. The front-end outputs a
on these data structures (or detect invariant violations).
variable declaration and observed variable values for each
memory object in a snapshot. To record the values of complex
4.2.2 Naming data structures objects (such as C structs), we flatten the objects and
Gibraltar uses two schemes to name each data structure record the values of each of their fields. Figure 5 presents
extracted from the target’s memory. The first scheme as- a simplified example: Figure 5(a) shows the declaration of a
signs a pathname to each data structure, which reflects variable named proc fs type->next (using the pathname-
the path from one of the root symbols to the data struc- based naming scheme) of type file system type. This
ture. For example, Gibraltar represents the head of the declaration also contains the names and types of the fields
all-tasks linked list, described in Section 3.2, using the of the data structure; hashcode indicates that the type is a
name init tasks->next task (here, init tasks is a pointer. The type hashcode is intended for unique object
root symbol). The second scheme names a data structure using identifiers like memory addresses (pointers) or the return value
its virtual memory address in the physical memory pages of Java’s Object.hashcode method. Figure 5(b) shows the
received from the target. This naming scheme is particularly values of each of the fields observed in the snapshot. Daikon’s
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 9
DECLARE list:::struct task_struct
file_system_type::: \ file_system_type::: \ init_tasks->next_task
proc_fs_type->next proc_fs_type->next [4160716800 3250692096 3250675712 3250667520 3250987008
name hashcode name 3223787403 3250978816 3250888704 4153163776 4153663488 4155981824
fs_flags int fs_flags 16 4154613760 4154474496 4151255040 4150353920 4150247424
read_super hashcode read_super 3223233888 4149018624 4149002240 4148912128 4148871168 4148207616
owner hashcode owner 0 4147781632 4151115776 4147806208 4147773440 4147830784
next hashcode next 3223882668 4147757056 4147675136 4147642368 4147470336 4146806784
fs_supers0_next hashcode fs_supers0_next 3250705524 4152762368 3224125440]
fs_supers0_prev hashcode fs_supers0_prev 3250705524
init_tasks->prev_task
(a)Variable declaration. (b)Variable values. [4152762368 4146806784 4147470336 4147642368 4147675136
4147757056 4147830784 4147773440 4147806208 4151115776
Fig. 5. Declaration and values observed for an object of type 4147781632 4148207616 4148871168 4148912128 4149002240
file system type. 4149018624 4150247424 4150353920 4151255040 4154474496
4154613760 4155351040 4151402496 4155981824 4153663488
4153163776 3250888704 3250978816 3250987008 3250667520
inference engine hypothesizes likely invariants for this variable 3250675712 3250692096 4160716800 3224125440]
by reasoning about its value across multiple snapshots. We use
init_tasks->thread_group0->next
the term object invariants to refer to properties of individual [3250978816 3250987008 3250667520 3250675712 3224125440]
memory objects.
Our front-end can also convert snapshots into a format init_tasks->thread_group0->prev
[3250675712 3224125440 3250978816 3250987008 3250667520]
that allows Daikon to infer invariants over collections of
objects. We call such invariants collection invariants. To do Fig. 6. Output of the front-end to generate linked list invari-
so, the front-end converts each memory snapshot into the ants on objects of type task struct.
equivalent of a Daikon trace file that contains the equivalent
of one “program point” for each collection of objects. The NULL value (or not 0, if var is not a pointer).
front-end records the values of all objects belonging to that
collection. The inference engine hypothesizes invariants for (3) Bounds template (var ≥ const), (var ≤ const).
all objects in that collection by observing their values across This template corresponds to invariants that determine lower
multiple snapshots. In preliminary experiments with Gibraltar, and upper bounds of the values that var acquires.
we found that the number of objects in a collection may be The three example templates discussed above correspond
overly large, which may in turn cause the invariant inference to invariants over variables and fields of C struct data
engine to exhaust available memory on our machine. We structures. These invariants can be inferred over individual
avoided this problem by configuring the front-end to split a objects, as well as over collections of data structures, e.g., the
large collection of objects into smaller sub-collections. fields bar of all objects of type struct foo have value 5.
Daikon does not infer properties on linked lists. However, Invariants over collections describe a property that hold for all
linked lists are ubiquitous in the Linux kernel and can be members of that collection across all snapshots.
exploited subtly by rootkits, as demonstrated in Section 3. (4) Length template (Length(var) == const). This tem-
As done in prior work [18], our front-end therefore linearizes plate describes invariants over lengths of linked lists.
linked lists into arrays, over which Daikon infers invariants. (5) Subset template (coll1 ⊂ coll2 ). This template rep-
Figure 6 presents examples of four kernel linked lists of resents invariants that describe that the collection coll1 is
type task struct, linearized into arrays. In each of these a subset of collection coll2 . This is used, for instance, to
examples, addresses of the objects in each list are recorded represent invariants that describe that every element of one
as elements of the array. This representation suffices to infer linked list is also an element of another linked list.
a restricted family of invariants over linked lists, which we
discuss next. In the current prototype of Gibraltar, we restrict The last two example templates are used to describe prop-
ourselves to linked lists with heads in the static data region. erties of kernel linked lists. As reported in Section 5, in
our experiments, invariants that conformed to the Daikon
Invariants and Invariant Templates templates sufficed to detect all the control and non-control
data attacks that we tested. However, to accommodate for
Daikon infers invariants that conform to 75 different tem-
rootkits that only violate invariants that conform to other
plates [19]. In the discussion below, and in the experimental
kinds of templates, we may need to extend Gibraltar with
results reported in Section 5, we focus on five templates. In
more templates in the future. Daikon supports an extensible
the templates below, var denotes either a scalar variable or a
architecture that allows new templates to be supplied, thereby
field of a structure.
allowing Gibraltar to detect more attacks.
(1) Membership template (var ∈ {a, b, c}). This tem-
plate corresponds to invariants that state that var only acquires
a fixed set of values (in this case, a, b or c). If this set is a 4.4 The Monitor
singleton {a}, denoting that var is a constant, then Daikon During detection, the monitor ensures that the data structures
expresses the invariant as var == a. in the target’s memory satisfy the invariants obtained during
(2) Non-zero template (var != 0). The non-zero template inference. As with the invariant generator, the monitor obtains
corresponds to invariants that determine that a var is a non- snapshots from the data structure extractor, and checks the data
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 10
init fs->root->d sb->s dirty.next->i dentry.next Rootkit/Attack Data structures affected
->d child.prev->d inode->i fop.read == 0xeff9bf60 Rootkits from Packet Storm [4]
Adore-0.42, All-root, Kbd, Synapsys-0.4 System call table
Fig. 7. Example of a transient invariant. The name of the Linspy2, Modhide, Phide, Rial, Kis 0.9
Rkit 1.01, Shtroj2, THC Backdoor
variable changes across reboots.
Adore-ng Vfs hooks, udp recvmsg
Knark 2.4.3 System call table, proc hooks
Rootkits from research literature [9]
structures in each snapshot against the invariants. This ensures Disabling firewall (§3.6) Netfilter hooks
that any malicious modifications to kernel memory that cause Disabling PRNG (§3.7) Vfs hooks
the violation of an invariant are automatically detected. Altering real-time clock (§3.8) Vfs hooks
Defeating signature scans (§3.9) Vfs hooks
Fig. 8. Linux-based rootkits that modify control data. This
4.5 Persistent and Transient Invariants table shows the data structures modified by the rootkit.
The invariants inferred by Gibraltar can be categorized as Gibraltar successfully detects all the above rootkits. The
either persistent or transient. Persistent invariants represent invariants violated are all Object invariants, detected by the
properties that are valid across reboots of the target machine, Membership(constant) template.
provided that the target’s kernel is not reconfigured or recom-
piled between reboots. An invariant is persistent if and only if
• Detection accuracy. We tested the effectiveness of Gibraltar
the expression that references the object persists across reboots
by using it to detect both publicly-available rootkits as well as
and the property represented by the invariant holds across
those proposed in the research literature [9, 34, 44]. Gibraltar
reboots. All the examples presented in Section 3 are persistent
detected all these rootkits (Section 5.3).
invariants.
In contrast, a transient invariant either expresses a property • Spurious alerts. When operating in the detection mode,
of an object whose pathname does not persist across reboots Gibraltar raises an alert when it observes an invariant violation.
or represents a property that does not hold across reboots. An alert is spurious if the invariant violation was not as a result
For example, consider the invariant in Figure 7, which ex- of a malicious change to a data structure. In our experiments,
presses a property of a struct file operations object. we observed that 0.035% of the persistent invariants raised
This invariant is transient because it does not persist across spurious alerts (Section 5.4).
reboots. The expression that references this object changes • Performance. We measured Gibraltar’s performance and
across reboots as it appears at different locations in kernel found that it imposes a negligible monitoring overhead (Sec-
linked lists. Consequently, the number of next and prevs tion 5.5).
that appear in the expression differs across reboots. All the experiments reported in this section were performed
The distinction between persistent and transient invariants is on a target system with a Intel Xeon 2.80GHz processor
important because it determines the number of invariants that with 1GB RAM, running a Linux-2.4.20 kernel. Infrastructure
must be inferred each time the target machine is rebooted. To limitations prevented us from upgrading to the latest version
find the number of persistent invariants inferred by Gibraltar, of the Linux kernel. The observer also had an identical
we repeatedly rebooted the system and executed an inference configuration.
workload (Section 5.1 presents details of the workload) until
the number of persistent invariants remained constant. After
eight reboots, we found that the total number of persistent 5.1 Experimental Methodology
invariants reported as true by Daikon was 26,946. Invariants Our experiments with Gibraltar were conducted using three
inferred by Gibraltar over the kernel static area also persist workloads, as described below. We ran Gibraltar in the in-
across reboots, and total 209,498 invariants. In contrast, a ference mode and executed an inference workload, which
single run of the inference workload yielded a total of 718,940 emulated user behavior on the target system. We then used
invariants for data structures allocated dynamically on the Gibraltar’s invariant generator to hypothesize invariants using
kernel’s heap. the kernel snapshots collected during the execution of the
Even though the number of persistent invariants is much inference workload. The numbers reported in this article are
smaller than the number of transient ones, persistent ones have based upon invariants inferred over fifteen kernel snapshots
the advantage that they do not have to be re-learnt after the collected by Gibraltar. We also configured Gibraltar to collect
system is rebooted. Moreover, they sufficed to detect all the thirty kernel snapshots and inferred invariants over these
rootkits in our test suite. Therefore, the primary focus of our snapshots. However, we observed that fewer than 0.01% of
experiments is on persistent invariants. the invariants changed when we used a larger set of kernel
snapshots for invariant inference. We then configured Gibraltar
to run in the detection mode using the automatically-inferred
5 E R
invariants. We executed a malicious workload, comprising 23
This section presents the results of experiments to test the rootkits, and studied the effectiveness of Gibraltar at detecting
effectiveness and performance of Gibraltar at detecting rootkits these rootkits. Finally, we studied the number of spurious
that modify both control and non-control data structures. We alerts by executing a benign workload on the target. All three
focus on three concerns: workloads are described in detail below.
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 11
Attack Name Data Structures Affected Invariant Type Template
Entropy Pool Contamination (§3.1) struct poolinfo Collection Membership
Hidden Process (§3.2) all-tasks list Collection Subset
Linux Binfmt (§3.3) formats list Collection Length
Resource Wastage (§3.4) struct zone struct Object Membership (constant)
Intrinsic Denial of Service (§3.5) max threads Object Membership (constant)
Fig. 9. Rootkits that modify non-control data [9, 34, 44]. This table also shows the data structure modified by the attack, the
type of the invariant violated by the attack, and the template that this invariant conforms to.
(1) Inference workload. We chose Lmbench [29], a micro- • Detecting control data modifications. We used fourteen
benchmark suite used to measure operating system perfor- publicly-available rootkits [4] that modify kernel data struc-
mance, as our inference workload. Lmbench measures band- tures to test the effectiveness of Gibraltar. We also included
width and latency for common operations performed by ap- four rootkits that have been proposed in the research litera-
plications, such as copying to memory, reading cached files, ture [9] (Attacks 6-9 from Section 3); these rootkits modify
context switching, networking, file system operations, process kernel function pointers.
creation, signal handling and IPC operations. This benchmark Gibraltar was successfully able to detect all the above rootkits.
exercises several kernel subsystems and therefore modifies Each of these rootkits violated a persistent invariant that
several kernel data structures as it executes. conformed to the template var == constant. Because these
(2) Malicious workload. This workload comprised 23 rootk- rootkits modify kernel control flow, they can also be detected
its (see Figure 8 and Figure 9). We installed the rootkits one by SBCFI. However, as discussed in Section 3, the invariants
at a time, and determined whether Gibraltar could detect the on control data structures inferred by Gibraltar implicitly
infection. We uninstalled each rootkit after recording alerts determine a control flow integrity policy that is equivalent to
generated by Gibraltar, and then installed the next rootkit from SBCFI.
the workload. The most common form of invariant violated by publicly avail-
(3) Benign workload. To count the number of spurious alerts, able rootkits that modify control data is var == constant.
we designed a workload consisting of several benign appli- This is because these rootkits hijack control of the kernel by
cations: (1) copying the Linux kernel source code from one overwriting otherwise immutable function pointers. Gibraltar
directory to another; (2) editing a text document; (3) compiling can possibly detect all such rootkits by restricting invariants
the Linux kernel; (4) downloading eight video files from the to the var == constant template. Doing so can possibly
Internet; and (5) performing file system read/write and meta improve the performance and precision of Gibraltar. However,
data operations using the IOZone benchmark [31]. This work- detecting advanced rootkits that violate properties of non-
load ran for 42 minutes on the target, during which time we ran control data requires more powerful invariants, as discussed
Gibraltar in detection mode using the automatically-inferred at length in Section 3. Consequently, we currently use five
invariants. The benign workload contained real application templates during invariant inference.
tasks and was chosen to be completely different from the • Detecting non-control data modifications. We used five
inference workload. non-control data attacks discussed in prior work [9, 34, 44] to
test Gibraltar. These attacks and the invariants that they violate
were discussed in detail in Section 3. Figure 9 summarizes
5.2 Invariants these attacks; it shows the data structures modified by the
As discussed in Section 4, the invariants inferred by Gibraltar attack, the invariant type (collection/object) violated, and the
include both properties of both individual objects and col- template that classifies the invariant. Each of the invariants
lections of objects, e.g., all objects of the same type or all that was violated was a persistent invariant, which survives a
nodes belonging to a linked list. Gibraltar inferred a total of reboot of the target machine.
26,946 persistent invariants on individual objects as well as
on collections of objects on the kernel heap. These invariants 5.4 Spurious Alerts
conformed to the five templates discussed in the previous An invariant violation reported by Gibraltar is spurious if the
section; the length and subset invariants apply only to linked violation was not caused by a malicious data structure mod-
lists. Gibraltar inferred 209,498 invariants on the kernel static ification. Spurious alerts therefore identify invariants whose
data region, thereby yielding a total of 236,444 persistent precision must be improved using manual refinement or an
invariants. Gibraltar also inferred 428,046 transient invariants enhanced inference workload. To measure the number of spu-
on dynamically-allocated kernel data structures to yield a total rious alerts, we conducted an experiment in which we executed
of 718,940 invariants. Gibraltar in detection mode using a benign workload on the
target system. For this experiment, we configured Gibraltar to
detect violations of persistent invariants, numbering 236,444
5.3 Detection Accuracy in total. During the 42-minute duration of this workload, we
We tested Gibraltar’s detection accuracy using 23 rootkits, observed 85 spurious alerts on a total of 82 unique invariants.
listed in Figure 8. These included rootkits that modify both Note that we directly used the automatically-inferred invariants
control and non-control data. for detection. Despite this fact, only 0.035% of the persistent
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 12
invariants contributed to spurious alerts, thereby suggesting Func. Time (Monitoring OFF) Time (Monitoring ON) Overhead
Avg. Min Max Avg. Min Max
that the persistent invariants inferred by Gibraltar are of Copy 0.2260 0.2259 0.2271 0.2272 0.2269 0.2277 0.48%
relatively high precision. Future work on manual refinement Scale 0.2239 0.2237 0.2242 0.2251 0.2248 0.2254 0.49%
Add 0.3316 0.3313 0.3321 0.3329 0.3326 0.3336 0.39%
of invariants can further improve their precision and reduce Triad 0.3295 0.3292 0.3298 0.3308 0.3304 0.3314 0.37%
the number of spurious alerts.
We also measured the number of spurious alerts by config- Fig. 10. Results from the Stream microbenchmark. All num-
uring Gibraltar to use both persistent and transient invariants, bers reported are across 100 iterations of the benchmark.
numbering 718,940 in total. In this case, the number of
spurious warnings rose drastically. For the same 42-minute du- and turned on. Bandwidth measurements and time taken for
ration, we observed a total of 4,673 spurious alerts (i.e., 0.65% the four vector operations are shown. This figure shows the
of the invariants resulted in spurious alerts). This experiment maximum and minimum time taken for each operation, and the
suggests that the precision of transient invariants is lower than average over 100 executions. As this figure shows, Gibraltar
persistent invariants, and that future work is needed to improve imposes a negligible overhead of 0.49% on the operation of
their precision. the target system.
5.5 Performance 6 L
We measured three aspects of Gibraltar’s performance: (1) in- While we are encouraged by Gibraltar’s ability to detect
ference time, i.e., the time taken by Gibraltar to observe the rootkits, our current prototype has several limitations, which
target and infer invariants; (2) detection time, i.e., the time we plan to remedy in future work.
taken for an alert to be raised after the rootkit has been
installed; and (3) performance overhead, i.e., the overhead on • Inconsistent data structures. Gibraltar fetches pages from
the target system as a result of periodic page fetches via DMA. the target system in an asynchronous fashion. Consequently,
the data structure extractor may encounter inconsistent data
(1) Inference time is calculated as the cumulative time taken structures during traversal. For example, it may encounter a
by Gibraltar to gather kernel data structure snapshots and linked list in which the the next pointer of a node was updated
infer invariants when executing in inference mode. Overall, by the kernel but the prev pointer was not yet updated. In such
the process of gathering 15 snapshots of the target kernel’s cases, the data structure extractor will traverse stale pointer
memory required approximately 25 minutes, followed by 31 values and view the resulting data structure as valid. Such
minutes to infer invariants, resulting in a total of 56 minutes. inconsistencies introduce noise in the inference process and
Inference is currently a time-consuming process because our may result in spurious alerts during detection. In addition to
current prototype invokes Daikon to infer invariants after col- false positives, this noise may also prevent an important set of
lecting all the kernel snapshots. Inference time can potentially data structure invariants from being inferred, thereby causing
be reduced by invoking Daikon after obtaining each snapshot Gibraltar to miss attacks that violate those invariants.
so that the invariant set is built in parallel as Gibraltar fetches This limitation can potentially be remedied by a proxy driver
more snapshots. in the target kernel that notifies and blocks Gibraltar’s page
(2) Detection time is the interval between the installation fetcher if a data structure is in the process of being mod-
of the rootkit and Gibraltar detecting that an invariant has ified. For example, this proxy can identify program points
been violated. As Gibraltar traverses the data structures in (e.g., function entry points and exits) at which it is “safe”
a snapshot and checks invariants over each data structure, to fetch pages, acquire locks on behalf of Gibraltar, and
detection time is proportional to the number of objects in allow the page to be transferred to the observer. It can
each snapshot and the order in which they are encountered subsequently release the lock after Gibraltar has acquired a
by the traversal algorithm. Gibraltar’s detection time varied complete snapshot of the target’s memory.
from a minimum of fifteen seconds, when there were 41,254 • Portability of transient invariants. Transient invariants
objects in the snapshot, to a maximum of 132 seconds, when inferred by Gibraltar are not portable, i.e., invariants must be
there were 150,000 objects in the snapshot. On average, we inferred for each target system to be protected by Gibraltar,
observed a detection time of approximately 20 seconds. and must also be inferred each time the system is booted.
(3) Monitoring overhead. The Myrinet PCI card fetches raw Two reasons contribute to this limitation. First, the names
physical memory pages from the target using DMA. Because used to identify objects are not portable. As discussed using
DMA increases contention on the memory bus, the target’s the example in Figure 7, several data structures in dynamic
performance will potentially be affected. We measured this regions do not have portable names because the path to these
overhead using the Stream benchmark [28], a synthetic bench- data structures constantly changes as a result of allocations
mark that measures sustainable memory bandwidth. Measure- and deallocations. Second, invariants may contain absolute
ment is performed over four vector operations, namely, copy, values, such as memory addresses, that may not be the same
scale, add and triad and averaged over 100 executions. The across different machines or after a reboot. This limitation
vectors were chosen so that they clear the last-level cache in can possibly be overcome by naming kernel data structures
the system, forcing data to be fetched from main memory. differently and improving the invariant specification language.
Figure 10 presents the bandwidth measurements for these four For example, kernel allocators can possibly be annotated to
vector operations, both with Gibraltar’s monitoring turned off, assign object names, which in turn can be used to uniquely
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 13
identify objects. Similarly, constant values, such as memory the feasibility of automatic invariant inference for rootkit
addresses, can be encoded symbolically rather than using their detection.
actual values, thereby ensuring that invariants inferred on one There are a number of avenues to improve Gibraltar. Fore-
system can be applied to other systems. most among these are techniques to address the limitations
• Soundness and completeness of invariants. As Gibraltar outlined in Section 6. Additional directions include:
uses a dynamic approach to infer invariants, the resulting • Attack analysis. While Gibraltar successfully detected all
invariants are neither sound nor complete. That is, false in- 23 rootkits that we tested it against, it may be possible to
variants may potentially be inferred, e.g., because the inference design a rootkit that compromises the kernel without vio-
workloads fail to discover all possible values of a data member, lating any of Gibraltar’s automatically-inferred data structure
leading to spurious alerts during detection. Similarly, it is invariants. Such a rootkit could exploit deficiencies in the
challenging to infer all possible invariants. This is because the coverage provided by Gibraltar’s invariants. Future work could
invariant templates used by Daikon may not suffice to capture develop techniques to estimate the difficulty of creating such
an important data structure property. While it is challeng- rootkits by measuring how much of the kernel’s attack surface
ing to overcome these limitations, prior work has developed is covered by Gibraltar’s invariants. An alternative approach
techniques to produce inference workloads that improve the could be to employ a “red team” to design rootkits that bypass
accuracy of invariants [23]. Future work can possibly adapt Gibraltar’s invariants, e.g., as done by Perkins et al. [32].
these techniques to improve the kernel data structure invariants • Studying the quality of invariants. Our study left unex-
used by Gibraltar. plored the semantic quality of automatically-inferred invari-
• Need for source code. Gibraltar’s data structure extractor ants. For example, it may be possible for a kernel expert to
critically relies on knowing the layout of kernel data structures. craft a single, high-quality invariant that subsumes thousands
We currently obtain this information using static analysis of of automatically-inferred invariants. Future work could com-
the kernel’s source code. Consequently, Gibraltar cannot be pare the merits of a manual approach against an automated
used with closed-source operating systems that lack debug approach to infer kernel data structure invariants.
symbols. This limitation can possibly be overcome using static • Incremental invariant updates. As presented, the design
analysis of binary executables, or using recent techniques to of our system requires a new set of invariants to be inferred
infer data structures from memory layouts [15]. upon each update to the operating system, e.g., via security
• Inability to detect transient attacks. Gibraltar’s approach patches. However, future enhancements to Gibraltar may make
of inferring and enforcing invariants restricts it to detecting it possible to incrementally update the set of invariants by
persistent modifications to kernel data. A rootkit might violate performing a change-impact analysis on how the updates
an invariant for a short period of time and revert it between affects kernel data structures.
two consecutive snapshots that Gibraltar collects [48]. Such Acknowledgments. We thank Joe Kilian for insightful discus-
rootkits can possibly be detected by reducing the time required sions about this work. We also thank the anonymous reviewers
to collect a single snapshot and increasing the frequency at of this article for their insightful comments. This work has
which Gibraltar samples the target’s memory. been supported in part by the NSF under grants 0728937,
0831268, 0915394 and 0931992 and award from the Rutgers
DIMACS center.
7 C F W
R
Several recent rootkits and research papers have demon- [1] Chkrootkit: Locally checks for rootkits. http://www.chkrootkit.org.
strated that attacks against control and non-control data in [2] F-Secure rootkit information pages: Fu rootkit. http://www.f-secure.
dynamically-allocated kernel memory are a realistic and grow- com/v-descs/fu.shtml.
[3] Myricom: Pioneering high performance computing. http://www.myri.
ing threat. Such rootkits are challenging to detect because the com.
vast number and heterogeneity of kernel data structures makes [4] Packet storm. http://packetstormsecurity.org/UNIX/penetration/rootkits.
writing correctness specifications for these data structures a [5] Rootkits, part 1 of 3: A growing threat, April 2006. MacAfee AVERT
Labs Whitepaper.
challenging exercise. Motivated by this challenge, we sought [6] 2010 threat predictions, December 2009. MacAfee AVERT Labs
to develop an automated approach to detect such rootkits. Whitepaper.
In this quest, we developed Gibraltar, a tool that detects [7] A. Baliga, X. Chen, and L. Iftode. Automated containment of rootkit
attacks. Elsevier Computers and Security Journal, 27:323–334, 2008.
kernel-level rootkits using data structure invariants. Gibraltar [8] A. Baliga, V. Ganapathy, and L. Iftode. Automatic inference and
uses an anomaly detection-based approach: it uses an auto- enforcement of kernel data structure invariants. In Annual Computer
matic technique to infer invariants on kernel data structures, Security Appl. Conf., 2008.
[9] A. Baliga, P. Kamat, and L. Iftode. Lurking in the shadows: Identifying
including both control and non-control data structures, and systemic threats to kernel data. In IEEE Symp. on Security and Privacy,
uses these invariants to detect rootkits. Gibraltar was able to 2007.
detect all 23 rootkits that we tested it against, while imposing [10] D. Beck, B. Vo, and C. Verbowski. Detecting stealth software with
Strider GhostBuster. In Intl. Conf. on Dependable Systems and Networks,
a runtime monitoring overhead of under 0.5%. We also found 2005.
that the automatically-inferred invariants were quite precise. [11] K. Butler, S. McLaughlin, and P. McDaniel. Rootkit-resistant disks. In
For example, we observed spurious warnings on only 0.035% ACM Conf. on Computer and Communications Security, 2008.
[12] M. Carbone, W. Cui, L. Lu, W. Lee, M. Peinado, and X. Jiang. Mapping
of the persistent invariants during a 42 minute testing phase. kernel objects to enable systematic integrity checking. In ACM Conf.
The approach described in this article therefore demonstrates on Computer and Communications Security, 2009.
IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. V, NO. N, TBD 14
[13] T. Chilimbi and V. Ganapathy. HeapMD: Identifying heap-based bugs [42] S. Sidiroglou, O. Laadan, N. Viennot, C. Perez, A. Keromytis, and
using anomaly detection. In Intl. Conf. on Arch. Support for Prog. Lang. J. Neih. ASSURE: Automatic software self-healing using rescue points.
and Operating Systems, 2006. In Intl. Conf. on Arch. Support for Prog. Lang. and Operating Systems,
[14] B. Cogswell and M. Russinovich. RootkitRevealer v1.71: Rootkit 2009.
detection tool by Microsoft. [43] S. Sparks and J. Butler. Shadow walker. Phrack Magazine, #63, Jan
[15] A. Cozzie, F. Stratton, H. Xue, and S. King. Digging for data structures. 2005.
In ACM/USENIX Symp. Operating System Design and Impl., 2008. [44] Shellcode Security Research Team. Registration weakness in Linux
[16] C. Csallner and Y. Smaragdakis. DSD-Crasher: A hybrid analysis tool kernel’s binary formats: Polluting sys execve in kernel space without
for bug finding. In Intl. Symp. on Software Testing and Analysis, 2006. depending on the sys call table. http://goodfellas.shellcode.com.
[17] B. Demsky, M. Ernst, P. Guo, S. McCamant, J. Perkins, and M. Rinard. ar/own/binfmt-en.pdf, 2006.
Inference and enforcement of data structure consistency specifications. [45] Y. Wang, R. Roussev, C. Verbowski, A. Johnson, M. Wu, Y. Huang, and
In Intl. Symp. on Software Testing and Analysis, 2006. S. Kuo. Gatekeeper: Monitoring auto-start extensibility points (ASEPs)
[18] M. D. Ernst, W. G. Griswold, Y. Kataoka, and D. Notkin. Dynamically for spyware management. In USENIX Conf. on System Administration,
discovering program invariants involving collections. Technical Report 2004.
UW-CSE-99-11-02, University of Washington, 2000. [46] Z. Wang, X. Jiang, W. Cui, and P. Ning. Countering kernel rootkits
[19] M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. with lightweight hook protection. In ACM Conf. on Computer and
Tschantz, and C. Xiao. The Daikon system for dynamic detection of Communications Security, 2009.
likely invariants. Sci. of Comp. Prog., 2006. [47] Z. Wang, X. Jiang, W. Cui, and X. Wang. Countering persistent kernel
[20] T. Fraser, M. R. Evenson, and W. A. Arbaugh. VICI: Virtual machine rootkits through systematic hook discovery. In Intl. Symp. on Recent
introspection for cognitive immunity. In Annual Computer Security Appl. Advances in Intrusion Detection, 2008.
Conf., 2008. [48] J. Wei, B. Payne, J. Giffin, and C. Pu. Soft-timer driven transient kernel
[21] T. Garfinkel, B. Pfaff, J. Chow, M. Rosenblum, and D. Boneh. Terra: A control flow attacks and defense. In Annual Computer Security Appl.
virtual machine-based platform for trusted computing. In ACM Symp. Conf., 2008.
on Operating System Principles, 2003. [49] J. Wilhelm and T. Chiueh. A forced sampled execution approach to
[22] T. Garfinkel and M. Rosenblum. A virtual machine introspection based kernel rootkit identification. In Intl. Symp. Recent Advances in Intrusion
architecture for intrusion detection. In Network and Distr. Systems Detection, 2007.
Security Symp., 2003. [50] H. Yin, Z. Liang, and D. Song. Hookfinder: Identifying and understand-
[23] N. Gupta. Generating test data for dynamically discovering likely ing malware hooking behaviors. In Network and Distr. System Security
program invariants. In Intl. Workshop on Dynamic Analysis, 2003. Symp., 2008.
[24] S. Hangal and M. S. Lam. Tracking down software bugs using automatic [51] X. Zhang, L. van Doorn, T. Jaeger, R. Perez, and R. Sailer. Secure
anomaly detection. In Intl. Conf. on Software Engineering, 2002. coprocessor-based intrusion detection. In 10th ACM SIGOPS European
[25] G. H. Kim and E. H. Spafford. The design and implementation of workshop: beyond the PC, 2002.
Tripwire: A file system integrity checker. In ACM Conf. on Computer All URLs were last verified on February 3, 2010.
and Communications Security, 1994.
[26] C. Kruegel, W. Robertson, and G. Vigna. Detecting kernel-level rootkits
through binary analysis. In Annual Computer Security Appl. Conf., 2004.
[27] L. Litty and D. Lie. Manitou: a layer-below approach to fighting
malware. In Arch. and System Support for Improving Software De-
pendability, 2006. Arati Baliga is a research scientist at the wire-
[28] J. D. McCalpin. Memory bandwidth and machine balance in current high less information network laboratory (WINLAB) at
performance computers. In IEEE Technical Committee on Computer Rutgers University. She obtained her Ph.D. degree
Architecture, 1995. in Computer Science from Rutgers University in
[29] L. McVoy and C. Staelin. lmbench: Portable tools for performance 2009. Her research interests broadly lie in sys-
analysis. In USENIX Annual Technical Conf., 1996. tems security and security in emerging wireless
[30] G. C. Necula, S. McPeak, S. Rahul, and W. Weimer. Cil: Intermediate networks.
language and tools for analysis and transformation of C programs. In
Intl. Conf. on Compiler Construction, 2002.
[31] W. Norcott. IOzone filesystem benchmark. http://www.iozone.org, 2001.
[32] J. Perkins, S. Kim, S. Larsen, S. Amarasinghe, J. Bachrach, M. Carbin,
C. Pacheco, F. Sherwood, S. Sidiroglou, G. Sullivan, W. Wong, Y. Zibin,
M. D. Ernst, and M. Rinard. Automatically patching errors in deployed
software. In ACM Symp. on Operating System Principles, 2009.
[33] N. L. Petroni, T. Fraser, J. Molina, and W. A. Arbaugh. Copilot: a
Vinod Ganapathy is an assistant professor of
coprocessor-based kernel runtime integrity monitor. In USENIX Security
Computer Science at Rutgers University. He
Symp., 2004.
earned a B.Tech. degree in Computer Science &
[34] N. L. Petroni, T. Fraser, A. Walters, and W. A. Arbaugh. An architecture
Engineering from IIT Bombay in 2001 and a Ph.D.
for specification-based detection of semantic integrity violations in
degree in Computer Science from the University of
kernel dynamic data. In USENIX Security Symp., 2006.
Wisconsin-Madison in 2007. His research interests
[35] N. L. Petroni and M. Hicks. Automated detection of persistent kernel
are in computer security and reliability.
control-flow attacks. In ACM Conf. on Computer and Communications
Security, 2007.
[36] R. Riley, X. Jiang, and D. Xu. Guest-transparent prevention of kernel
rootkits with VMM-based memory shadowing. In Intl. Symp. on Recent
Advances in Intrusion Detection, 2008.
[37] J. Rutkowska. Beyond the CPU: Defeating hardware based RAM
acquisition, part I: AMD case. In Blackhat Conf., 2007.
[38] R. Sailer, T. Jaeger, X. Zhang, and L. van Doorn. Attestation-based
policy enforcement for remote access. In ACM Conf. on Computer and Liviu Iftode is a professor of Computer Science
Communications Security, 2004. at Rutgers University. He earned a Ph.D. degree
[39] R. Sailer, X. Zhang, T. Jaeger, and L. van Doorn. Design and in Computer Science from Princeton University in
implementation of a TCG-based integrity measurement architecture. In 1998. His research interests are in operating sys-
USENIX Security Symp., 2004. tems, distributed systems, systems security, mobile
[40] “sd” and “devik”. Linux on-the-fly kernel patching without LKM: and pervasive computing and vehicular computing
SucKIT rootkit. Phrack Magazine, #58, Dec 2001. and networking.
[41] E. Shi, A. Perrig, and L. van Doorn. Bind: A fine-grained attestation
service for secure distributed systems. In IEEE Symp. on Security and
Privacy, 2005.