Get documents about "Distributed incident analysis and detection in IP networks Ondrej
Document Sample
Czech Technical University in Prague
Faculty of Electrical Engineering
ˇ ˇı ˇu
CVUT FEL katedra poc´tac ˚
Master thesis
Distributed incident analysis and detection in IP networks
r
Ondˇej Bezucha
ın a
Supervisor: Ing. Anton´ Kr´l
Study program: Electrical Engineering and Information Technology
Study branch: Computer Engineering
May 2007
ii
Thanks
I would like to thank to Antonin Kral for his valuable suggestions and his supervision and to
Lubomir Pinkava for his comments and feedback.
iii
iv
as ı
Prohl´ˇen´
s z a e z
Prohlaˇuji, ˇe jsem svou diplomovou pr´ci vypracoval samostatnˇ a pouˇil jsem pouze podklady
e r z e
uveden´ v pˇiloˇen´m seznamu.
Nem´m z´vaˇn´ d˚vod proti uˇit´ tohoto ˇkoln´ d´ ve smyslu §60 Z´kona ˇ. 121/2000
a a z y u z ı s ıho ıla a c
a e a
Sb., o pr´vu autorsk´m, o pr´vech souvisej´ ıch s pr´vem autorsk´m a o zmˇnˇ nˇkter´ch
ıc´ a y e e e y
a u y a
z´kon˚ (autorsk´ z´kon).
V Praze dne 25. 5. 2007 .............................................................
v
vi
Abstract
This thesis is a proof of concept implementation of Netflow based network intrusion detection
system. Incident detection in this system is based on analysis of data flow records, which are
exported using the Netfow technology from many routers distributed in the network – this is
different approach to centralised IDS deployment. The system implements several techniques,
that were described in two articles [6] written by Yiming Gong. It is able to read natively
Netflow version 5 exports in the flow-tools file format, it is also able to distribute information
about arised incidents to many destinations using several protocols like BSD syslog or SMTP
and it can be integrated with monitoring systems – currently Nagios support has been imple-
mented. The system is fully modular, so it can be freely extended with additional input, output
or method modules.
Abstrakt
a ı ı u e
Tato pr´ce je pilotn´ implementac´ Netflow IDS. Detekce incident˚ v tomto syst´mu je zaloˇenaz
y y u y
na anal´ze datov´ch tok˚ exportovan´ch technologi´ Netflow, kter´ pˇ´
ı a ı e
e rımo proch´z´ smˇrovaˇi, c
y ıti. a
rozdistribuovan´mi v s´ Jedn´ se o pˇ´ s y u c ı
rıstup zcela odliˇn´ od detekce incident˚ tradniˇn´ ces-
tou centralizovan´ch IDS, syst´m vyuˇ´ a techniky popsan´ v pr´ci Yiming Gonga [6]. Syst´m
y e zıv´ e a e
y e cıst a az
je schopn´ nativnˇ ˇ´ export Netflow verze 5 ve form´tu flow-tools, dok´ˇe distribuovat in-
se y ıce ı
formace o zjiˇtˇn´ch incidentech na v´ destinac´ protokoly jako BSD syslog nebo SMTP a lze
y e c e e a ı
jej integrovat s dohledov´mi syst´my. V souˇasn´ dobˇ je implementov´na nativn´ podpora
e e e a ı uz y e er e sr a
syst´mu Nagios. Syst´m je plnˇ modul´rn´ a m˚ˇe b´t t´mˇˇ neomezenˇ rozˇiˇov´n o podporu
sıch ıch y ıch au sı
dalˇ´ vstupn´ nebo v´stupn´ form´t˚ nebo o dalˇ´ metody detekce incident˚. u
vii
viii
Index
List of figures xiii
List of tables xv
1 Introduction 1
1.1 Intrusion Detection Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Netflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 History of Netflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Flows, flow cache and Netflow Data Export . . . . . . . . . . . . . . . . 3
1.2.3 Netflow versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Netflow based network IDS 8
2.1 Netflow based analysis methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.1 Top N and Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.2 Pattern Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.3 Detecting network scans . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.3.1 Detecting TCP scans . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.3.2 Detecting UDP scans . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Netflow IDS architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3 NNIDS design and implementation 13
3.1 Existing components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.1 Netflow probes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.2 Netflow Collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2 NNIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2.1 Main logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2.2 Program life-cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.3 Implemented input modules . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.3.1 Flowtools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2.3.2 cflowd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2.4 Implemented output modules . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2.4.1 Stderr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2.4.2 Syslog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2.4.3 Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2.4.4 Nagios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2.5 Implemented method modules . . . . . . . . . . . . . . . . . . . . . . . 25
ix
3.2.5.1 TopN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2.5.2 Scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.5.3 PatternIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.5.4 PatternIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.5.5 DoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.6 Method modules in a real network . . . . . . . . . . . . . . . . . . . . . 28
4 Testing and improvements 29
4.1 Performance testing and improvements . . . . . . . . . . . . . . . . . . . . . . . 29
4.2 Operation in real networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5 Conclusion 32
5.1 Finished work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6 Bibliography 33
Appendix A Installation and configuration 35
A.1 Netflow infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
A.1.1 Netflow probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
A.1.1.1 Configuring NDE on Cisco router . . . . . . . . . . . . . . . . 35
A.1.1.2 Configuring NDE on a Unix box . . . . . . . . . . . . . . . . . 36
A.1.2 Netflow collector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
A.1.2.1 Configuring flow-capture . . . . . . . . . . . . . . . . . . . . . 37
A.2 NNIDS installation and configuration . . . . . . . . . . . . . . . . . . . . . . . . 38
A.2.1 Installation requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 38
A.2.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
A.2.3 NNIDS configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
A.2.3.1 Global section . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
A.2.3.2 Flowtools input module . . . . . . . . . . . . . . . . . . . . . . 42
A.2.3.3 cflowd input module . . . . . . . . . . . . . . . . . . . . . . . . 43
A.2.3.4 TopN method module . . . . . . . . . . . . . . . . . . . . . . . 43
A.2.3.5 Scan method module . . . . . . . . . . . . . . . . . . . . . . . 44
A.2.3.6 PatternIP method module . . . . . . . . . . . . . . . . . . . . 44
A.2.3.7 PatternPort method module . . . . . . . . . . . . . . . . . . . 44
A.2.3.8 DoS method module . . . . . . . . . . . . . . . . . . . . . . . . 45
A.2.3.9 Stderr output module . . . . . . . . . . . . . . . . . . . . . . . 45
A.2.3.10 Syslog output module . . . . . . . . . . . . . . . . . . . . . . . 45
x
A.2.3.11 Mail output module . . . . . . . . . . . . . . . . . . . . . . . . 46
A.2.3.12 Nagios output module . . . . . . . . . . . . . . . . . . . . . . . 46
Appendix B Manual page 49
Appendix C Contents of enclosed CD-ROM 58
xi
xii
List of figures
1.1 Network with Netflow infrastructure . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Netflow Version 9 Packet Example . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1 TCP session example. Server (host on the right side) is closing the connection. 10
3.1 NNIDS deployment scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2 NNIDS design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.3 Nagios user interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.4 Badness calculation in the TopN module (mind the logarithmic scale on x axis) 26
xiii
xiv
List of tables
1.1 Netflow Version 5 Header and Record Format . . . . . . . . . . . . . . . . . . . 5
3.1 Flow-tools file format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2 Flow-tools index version 1 format . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3 Flow-tools index version 3 format . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.4 Flow-tools type, length and value (TLV) triplets for index version 3 . . . . . . 20
3.5 Flow-tools flow record version 5 structure . . . . . . . . . . . . . . . . . . . . . 21
xv
xvi
CHAPTER 1. INTRODUCTION 1
1 Introduction
Internet and computer networks grow every day. Every day, more and more data are being
processed by network connected computers and every day grows the value of data stored in
these computers. What however grows as well, is a number of security threats emerging every
day.
According to Symantec’s Internet Security Threat Report[1] in 2005 there were 927 DoS
attacks detected every day in average. In 2007 it is 6110 attacks in average per day. In 2005
there was observed an average of 9163 bot-infected computers per day. In 2007 it is 63912
computers per day. The number of threats is increasing and it will continue to grow very likely
in the future.
To protect end-hosts from the increasing number of threats, whole security infrastructure is
usually being deployed in computer networks. End host can be running anti-virus software to
protect it from malware (software designed to infiltrate or damage a computer system without
the owner’s informed consent). Network switch can use 802.1X [2] or related protocols to protect
the network from unauthorized use and thus limit the number of anonymous attack from the
inside of the network. Firewalls are deployed to enforce and police traffic flows between internal
and external networks. But even computer running up to date operating system and anti-virus
software, connected to well policed network can become the subject of hackers activity or can
be infected by yet unknown worm/virus. For this reasons additional security infrastructure
element is usually being deployed in computer networks – an Intrusion Detection System,
shortly IDS.
1.1 Intrusion Detection Systems
Intrusion Detection System, as its name implies, detects computer attacks and computers
misuse [3]. If some sort of security incident has been detected, IDS triggers preconfigured
action. Action can be anything, starting with simple syslog entry, SNMP trap or email and
ending with complex scripts, which can do access list modifications on routers, etc. What action
can IDS do depends on IDS type, vendor and model. What is considered to be an ”security
incident” also differs and is usually configurable option.
IDS role in network security is often compared to burglar alarm system. When its properly
configured and deployed, it detects if the network is or is not under attack, but usually other
elements like firewall have to be deployed to prevent the intrusions.
There are several categories of IDS, divided by type, technique of incident detection, reac-
tivity or type of facture. Basically IDS’s are Host-Based or Network-Based:
• Host-Based IDS (HIDS) was the first type developed and implemented. HIDS is
usually a piece of software (also called “agent”) running on the end-host and collecting
data about particular services on this host. The agent can be monitoring network con-
nection attempts, audit trail events, system resources and other input data, which are
being correlated by the agent. If an security incident is detected, agent triggers alarm.
Nowadays HIDS are often sending alarms and events into central place, which can corre-
late events from many network places (hosts) and thus be more accurate in false-positive
reduction and intrusion detection. Examples of HIDS are Samhain[16] or Aide[17] in the
open source world, or Cisco Security Agent[18] and CA Host-Based Intrusion Prevention
System[19] in the commerce world.
2 CHAPTER 1. INTRODUCTION
• Network-Based IDS (NIDS) are nowadays more common. Securing large network
using HIDS can become very difficult, since every host should run software agent. Main-
taining HIDS on every end-host is labour-intensive and often impossible, since there are
many devices, that can become victim of security attacks, but are not able to run the
HIDS – network printers, simple routers or switches, wireless access points, embedded
systems, etc. NIDS listens data flowing in the network and analyzes them. If abnormal
or security violating data flow is detected, alarm is triggered. NIDS are usually placed at
strategic places in the network, where the biggest amount of traffic to/from the secured
network can be captured. Placement can be in the path of traffic flow, so every packet has
to pass it, or out of band, when packets are being delivered for analysis by other methods
(for example using switch port mirroring techniques like SPAN or by using network tap
devices). Examples of NIDS are Snort[20] in the open source world, or Cisco IDS 4200[21]
sensors in the commerce world.
• Combined Intrusion Detection is what the security experts would like to see deployed
in the network the most. Certain hosts are running HIDS, while NIDS for the whole
network are deployed. This method is the most accurate in detection and provides least
false-positives, since intrusions are being detected by many devices. Central correlating
points, which are receiving alerts from all hosts and triggering the master alert are a must
in this type of IDS. Examples of correlating software from the commerce world are Cisco
MARS[22] and CA Security Command Center[23].
IDS can be also divided by manner in which are security incidents being detected. These
types are: Anomaly detection, misuse or signature detection, target monitoring and stealth
probes.
• Anomaly Detection based IDS detects deviations of normal network/host/service oper-
ation behavior. This technique establishes a baseline of normal usage patterns. If serious
deviation from the baseline is detected, alarm is triggered. An example of this would be
if a user logs on and off of a machine 20 times a day instead of the normal 1 or 2. Another
example can be 100Mbit/s data flow to unknown Internet site from host, which normally
makes just a few kbit/s. Advantage of anomaly detection approach is, that no signature
pattern have to be updated to detect new threats. The same system can work for years
without any update.
• Signature Detection uses predefined misuse or security violation patterns, for which are
logging entries or flowing packets being searched. For example well known network worm
Code Red, which was exploiting a vulnerability in the indexing software distributed with
Microsoft IIS, was generating HTTP requests GET /default.ida?NNNNNNNNNNNNN ...
=a HTTP/1.0. Any host generating this form of request is almost surely infected by the
Code Red. Script which is searching log files for this pattern and notifying administrator
of infected hosts can be with a bit of good will called signature based IDS. Signature based
IDS has to be updated often, to stay accurate when detecting new threats. However it
is able to detect incidents with very little behavioral change, which anomaly detection
approach would not be able to detect.
• Target Monitoring looks for modifications made on monitored objects (mostly files)
and triggers alarm, when unauthorized modification was made. This type of detection is
mostly used in HIDS like Tripwire or Samhein, which create MD5 and SHA1 check-sums
of systems files and periodically check them. If a check-sum mismatch is detected, alarm
is triggered. Target monitoring is usually able to detect only specific types of threats,
and can be hardly used as general purpose IDS.
CHAPTER 1. INTRODUCTION 3
• Stealth Probes are mostly implemented in combined intrusion detection systems. They
collect inputs of many probes for a long time period and attempt to discover any corre-
lating attacks. This types require large data calculations on correlating systems, but are
able to detect sophisticated attacks and probes made over long periods of time.
For a long time IDS were just passive and their only activity was alerting the administrator.
Nowadays IDS are being used even in the actively defending part of security infrastructure, so
now we have passive and reactive IDS:
• Passive IDS is the traditional IDS. It detects, it alerts, but it does not do anything to
stop detected security incident.
• Reactive IDS is able to do proactive steps to stop or restrict the impact of detected
security incident. An example of reactive IDS is Intrusion Prevention System – IPS. In
a NIDS world, IPS is placed in the way of packet flow, similarly to the firewall. Normal
traffic is passing through, traffic that violates security policies is dropped.
1.2 Netflow
Netflow is an open technology developed by Darren Kerr and Barry Bruins at Cisco Systems
in 1996, which makes possible to export detailed information about packets forwarded by the
router to a Netflow collector for later analysis or accounting purposes. Only aggregated packet
headers without payload are exported to the collector.
1.2.1 History of Netflow
In IP unicast world, there are always 2 programs, which communicate with each other. The
data flow is split into packets for transport. Router’s basic role in the IP world is just to
transport IP packets to right next hop, according to packet’s destination IP address. Basic
router does not know anything about port numbers or flows, it works just on destination IP
address basis and routing table with L3 to L2 mapping table is its only switching base.
When the Cisco company was optimizing their algorithm for packet switching, later called
CEF - Cisco Express Forwarding, additional tables were added to speed up router’s next-hop
look-up process. In CEF Forwarding Information Base table is created and maintained, which
contains precalculated route entries (address, prefix and reference to the adjacency table).
Adjacency Table contains layer 2 address information for adjacent hops similarly to the ARP
table. Look-ups in these two tables are made in nearly constant time so forwarding decisions
are fast even with large number of route entries and adjacent hosts. Very often (it depends
on the particular platform used) another table is also being maintained by the CEF – the flow
cache.
1.2.2 Flows, flow cache and Netflow Data Export
A flow in the IP world is identified as unidirectional stream of packets between a given source
and destination. Both ends are defined by pairs source IP address, source transport-layer port
number and destination IP address, destination transport-layer port number and by protocol
type, type of service field and the input interface. These fields are called key fields, because
they can be used as unique key in the flow table.
4 CHAPTER 1. INTRODUCTION
Figure 1.1: Network with Netflow infrastructure
When a packet is being forwarded, flow cache is looked up. If no record of the flow has
been found, its created of contents of datagram headers and CEF tables. When the record or
appropriate flow has been found, entries like packet/octet counters or flow lifetime are updated.
Since the number of maximum flow cache entries is limited, flow records have to be deleted
sometimes.
Flow entry is deleted from the flow cache when it has been inactive for a long time (15 second
by default) or when RST/FIN flag has been sent in TCP session... whichever comes first. Flow
entries are being deleted by least recently used algorithm, when flow table fills. Flow entry is
also deleted when its life time is longer than the timeout (30 minutes by default).
Before the flow entry is deleted, information in the record can be exported from the flow
cache to a distant station and thus every session can be logged. The distant station is called
Netflow Data Collector and data are transported by the Netflow protocol. The process of
exporting deleted flow entries is called Netflow Data Export and is available on most of the
Cisco routers (series 800, 1700, 1800, 2800, 3800, 6500, 7300, 7600, 10000 and CRS-1), large
Cisco switches (series 4500, 6000, 6500), all of the Juniper routers and many platforms by
Nortel, Enterasys, Extreme Networks and other network vendors.
Netflow is de-facto standard for network traffic accounting and despite other newer tech-
nologies like sFlow, it is the most widely supported telemetrics technology in the IP networking
world.
1.2.3 Netflow versions
When the flow record is exported from the flow cache, its contents are transformed into Netflow
datagram. As the Netflow Data Export was being improved, many Netflow versions were
created. All of the versions are UDP based and are incompatible with each other. Until version
9, each Netflow datagram contains one header with version number and time and one or more
records, which contain main flow record entries.
CHAPTER 1. INTRODUCTION 5
Header
Bytes Content Description
0-3 version and count Netflow export format version number and number of
flows exported in this packet (1-30).
4-7 SysUptime Current time in milliseconds since router booted
8-11 unix secs Current seconds since 0000 UTC 1970.
12-15 unix nsecs Residual nanoseconds since 0000 UTC 1970.
16-19 flow sequence Sequence counter of total flows seen.
20-23 reserved Unused (zero) bytes.
Record
0-3 srcaddr Source IP address.
4-7 dstaddr Destination IP address.
8-11 nexthop Next hop router’s IP address.
12-15 input and output Input and output interface’s SNMP index.
16-19 dPkts Packets in the flow.
20-23 dOctets Total number of Layer 3 bytes in the flow’s packets.
24-27 First SysUptime at start of flow.
28-31 Last SysUptime at the time the last packet of flow was re-
ceived.
32-35 srcport and dstport TCP/UDP source and destination port number or
equivalent.
36-39 pad1, tcp flags, prot, and Unused (zero) byte, Cumulative OR of TCP flags,
tos IP protocol (for example, 6=TCP, 17=UDP), and IP
type-of-service.
40-43 src as and dst as AS of the source and destination, either origin or peer.
44-47 src mask, dst mask, and Source and destination address prefix mask bits, pad
pad2
Table 1.1: Netflow Version 5 Header and Record Format
Version 1 was the initial Netflow version released. Versions 2, 3 and 4 were only Cisco’s
internal development versions and haven’t ever been released to public.
Version 5 header and flow record format is specified in table 1.1. This version added
BGP’s source and destination AS and flow sequence number to every flow record. Purpose of
source/destination AS numbers is clear, but why the sequence numbers? Since Netflow is UDP
based protocol, packet loss can happen without any notice of transport layer. When packet
loss of Netflow datagrams occur, Netflow v1 collector has no way to detect this fact in any of
the ISO/OSI layers. Netflow 5’s sequence number is the number of total flows exported from
the router since the last export. Thus NDE packet loss can be easily detected, if the number
of flows received mismatch with the sequence number in the last received NDE packet.
Version 6 is used very rarely and is not implemented in new IOS releases. Version 7 is
similar to version 6 – it was designed for the old Catalyst 5000 series equipped with a NetFlow
feature card and is not compatible with Cisco routers.
The version 8 format adds router-based aggregation schemes. Previous versions provided
very little aggregation on data flows and thus can be unusable on very high speed saturated lines.
In version 8, five flow cache tables can be created on the router, where flows are aggregated.
Aggregation on the router side makes more information loose – the individual flows cannot be
seen on the collector and thus can be unusable for some kinds of flow analysis. This approach
but requires much less data to be transferred to the collector. The following aggregation tables
6 CHAPTER 1. INTRODUCTION
Figure 1.2: Netflow Version 9 Packet Example
can be created on the router, each with different aging timers, export destination IP address
and UDP port:
• Autonomous System Aggregation table – each entry represents packets with the
same source BGP autonomous system, destination BGP autonomous system, input in-
terface, and output interface.
• Destination Prefix Aggregation table – each entry represents packets with the same
destination prefix, destination prefix mask, destination BGP autonomous system, and
output interface.
• Prefix Aggregation table – each entry represents packets with the same source prefix,
destination prefix, source prefix mask, destination prefix mask, source BGP autonomous
system, destination BGP autonomous system, input interface, and output interface.
• Protocol Port Aggregation – each entry represents packets with the same IP protocol,
source port number, and destination port number when applicable.
• Source Prefix Aggregation – each entry represents packets with the same source prefix,
source prefix mask, source BGP autonomous system, and input interface.
Most recent version of Netflow is 9, introduced in October 2004 and described in RFC 3954 [8].
Version 9 is an enhancement to support different technologies such as Multicast, Internet Proto-
col Security (IPSec), and Multi Protocol Label Switching (MPLS). To accomplish this, template
CHAPTER 1. INTRODUCTION 7
flow sets had been introduced to replace the old fixed records. Every flow record set is preceded
by template with type/length pair for each field of the flow record. Version 9 is thus “future-
proofed” against new or developing protocols, because the version 9 format can be adapted to
provide support for them. An example of version 9 packet can be seen in figure 1.2.
Although version 9 has many enhancements, most widely used version these days is still
version 5. It contains all fields necessary for network management and accounting purposes, it
is well supported by management software and if the router supports Netflow Data Export, it
will very likely support just version 5.
8 CHAPTER 2. NETFLOW BASED NETWORK IDS
2 Netflow based network IDS
Placement of traditional IDS in a large network is a matter of compromise. We’d like to analyze
as much traffic as possible (and all the traffic would be the best), but IDS have limited number
of network interfaces and are quite expensive. Thus limited number of IDS is usually deployed
at strategic places of the network like network boundaries, data centers or central offices, where
most of the network traffic is being captured and most security threats are arising. Traditional
IDS can also be run on every router, but this option is not available on every platform, often
requires additional expensive software licenses and highly utilizes router’s CPU, so its used
rarely.
Netflow usually requires no additional license, can be run on nearly every router and requires
very little bandwidth and CPU time, so it could be desirable to use Netflow as an input for
specialized intrusion detection system.
Traditional network IDS usually does deep packet inspection. It looks into packet payload,
detects application protocol, scans for known threat patterns, classifies the packet and then
makes the decision. As was mentioned before, Netflow does not provide any application layer
information. Flow record has no payload field, only certain information from network and
transport layer headers can be gained. Thus traditional IDS will not be able to work with
just Netflow based information. Still many security threats exist, that can be detected just by
inspecting IP, TCP or UDP headers and appropriate performance counters. Netflow based IDS
thus can be very valuable tool for busy, high-speed network environments and in zero–day or
“mutant attack” detection in cases where signature–based intrusion detection systems would
fail. Additionally Netflow probes can be distributed in the whole network with no additional
cost and thus save expenses traditional IDS would require.
2.1 Netflow based analysis methods
Netflow based IDS is able to detect many security threats. To accomplish this, several anal-
ysis methods can be involved. Most of these methods have been described in Yiming Gong’s
article [6], which is also base of this master thesis. This chapter describes most useful of these
methods and common threats, that can be detected through this methods.
2.1.1 Top N and Baseline
A baseline is a model describing what “normal” network activity is according to some historical
traffic pattern; all traffic that falls outside the scope of this established traffic pattern will
be flagged as anomalous. Every Netflow record has 2 statistical counters that can establish
a baseline: total number of packets in the flow and total number of layer 3 bytes in the flow.
Since every flow represents at most one session, total number of sessions per time interval can
also be measured in per host basis. From these data following methods can be applied:
• Top N sessions. When a single hosts generates abnormal amount of sessions, the most
likely reason for these activities are presence of new worms, DoS/DDoS attacks, network
scans or certain kinds of network abuses. Normal clients connecting to the Internet should
keep a relatively normal connection frequency to the outside. But if a host is infected
with a worm, it will act differently. It will always launch a large number of connection
requests to the outside for its attempts to infect the next batch of victims, and as a result,
the connection request numbers sent out will be significantly high.
CHAPTER 2. NETFLOW BASED NETWORK IDS 9
• Top N bytes transferred. Group of top N hosts, that transfer traffic data to or from
the outside in an enterprise should be ranked into relatively fixed groups. If this pattern
changes, and a new host suddenly appears in the Top N hosts matrix, an alert should
be triggered. Abnormally large amount of data transferred can also be a sign of worm,
DoS/DDoS attack or use of P2P program, which can be forbidden by the network use
policy.
• Top N packets transferred. Group of top N packet generating hosts should be even
more fixed group, than group of top N byte consumers. Abnormally large amount of pack-
ets, when almost ordinary number bytes has been transferred is usually sign of network
scan. When abnormally large amount of bytes has been also detected, DoS/DDoS attack
or worm activity is the most likely reason. In either case an alert should be triggered.
2.1.2 Pattern Matching
Pattern matching is another method we can use to identify abnormal network activities when
doing flow-based analysis. With this method, the flow records will be searched and those hosts
associated with flow fields that seem suspicious based on our criteria will be flagged. Any flow
field can be used to do a pattern match, but most commonly used are:
• IP address. Source or destination IP address can be very good source for detecting
malicious activities. There are many IP address blocks, that should never appear in
normal Internet communication. Examples of these addresses are IANA unallocated IP
address ranges like 1.0.0.0/8 or 2.0.0.0/8, IANA special reserved ranges like 0.0.0.0/8 or
127.0.0.0/8, or private address ranges according to RFC 1918 in public Internet. Address
ranges that should never appear in normal communication are publicly maintained in
“Bogon list” and detection of packets to this destinations can be desirable.
In a stub (ie. non transit) network packet should have local network’s source or destination
IP address. Packets with non-local source and destination IP address are also sign of
malicious activity or configuration problem.
Lastly, many well known worms are trying to contact one or more fixed IP addresses on
fixed TCP/UDP ports. Detection of these worms by pattern matching is simple and very
effective.
• Transport layer port number. UDP or TCP port number matching can also help
in malicious activities detection, since many attacks are blindly target on specific port
numbers. Using this method alone brings many false positives.
Pattern matching is commonly used in groups – for example connection attempts to Unix
hosts on Microsoft domain services ports can be a sign of an attack, or in combination with
“top N” methods – for example abnormally high number of connection attempts to Microsoft
SQL Server port numbers can be a sign of worm activity.
2.1.3 Detecting network scans
Network scan is by itself mostly non-disruptive network activity, but can be an indication of
future attack. Since worms are trying to replicate and thus find as many victims as possible,
they are also doing some sort of network scan. Typically they send out hundreds or even
thousands of probes to large blocks of IP addresses in a very short period of time. If a worm
10 CHAPTER 2. NETFLOW BASED NETWORK IDS
Figure 2.1: TCP session example. Server (host on the right side) is closing the connection.
was designed to spread via TCP (as most of them are), during its propagation there will be
a lot of corresponding TCP SYN packets sent out as it seeks vulnerable services in other hosts.
2.1.3.1 Detecting TCP scans
Life-cycle of normal TCP session begins with three–way handshake. Client sends TCP segment
with SYN flag to the server. Server responds with SYN and ACK flags set. Then clients
responds with ACK flag set and data segments with ACK flags will follow. When session is
going to end, segment with FIN and ACK will appear (from both hosts), ACKed by both sides.
Then the session is closed and flow record deleted. TCP session example can be seen on figure
2.1.
When the remote host (server) is not listening on specified TCP port, client initiates the
session with SYN segment and the server closes the session immediately with RST and ACK
flagged segment.
When the remote host (server) is not even alive, no TCP response can be detected. The
client sends several SYN segments and gives up soon, when no reply segment is received. Many
TCP/IP stacks are using ICMP destination unreachable messages to inform the client that
destination IP address is not listening.
In Netflow version 5, every flow record contains list of flags, that have been set in any
segment of the flow at least once – its a cumulative logical OR of TCP flags set in the flow.
When detecting TCP scan we should capture a large number of flows with only SYN or SYN,
ACK and RST flags set.
2.1.3.2 Detecting UDP scans
UDP protocol has no flag field in its header. The only way how the server can tell the client,
that there is nothing listening on particular port is through the ICMP protocol.
CHAPTER 2. NETFLOW BASED NETWORK IDS 11
Life-cycle of normal UDP session generates no ICMP messages. When the remote host
(server) is not listening on specified UDP port, ICMP port unreachable message is generated
by the server and destinated to the client.
When the remote host (server) is not listening at all and thus no L3 to L2 mapping is
available or when intermediate router has no route to the destination host, ICMP destination
unreachable message is generated on the last router and destinated to the client. This manner
is common for all transport layer protocols.
If a host has an abnormal volume of ICMP port/host/network unreachable set of flow
records, that may indicate that the host is acting abnormal and is probably scanning the
network or spreading a worm.
2.2 Netflow IDS architecture
Even thought methods for detecting security incidents using Netflow Data Exports are well
described and documented in [6], there are very few Netflow based network IDS (NNIDS) im-
plemented. Gong’s article has the “Implementation guidelines” part, but its rather methodically
oriented and presents steps for hand work analysis using existing Netflow tools. As far as the
author knows no open-source NNIDS has been released and no architecture recommendation
or tips have been publicly presented and should be main part of this thesis.
When designing Netflow based IDS we should keep in mind several facts, that are re-
lated even to the ordinary network IDS, but are augmented in the Netflow based network IDS
(NNIDS):
• None of the methods mentioned above is able to provide fair detection results, when it
is used alone. Using single method for incident detection provides large number of false–
positives and has limited range of detectable incidents, since traffic in most of today’s
networks is multifarious, as well as symptoms of malicious activities. Traffic generated by
heavily used ftp server can be very similar to the traffic generated by a worm in a number
of sessions or amount of transferred data.
• Use of several methods together provides less false–positive results, but limits the range
of detectable incidents.
• Network traffic flows patterns are changing, speed and saturation of lines is increasing,
patterns of today’s worm will be different to tomorrow’s one. IDS has to adapt these
changes. Fixed configuration, fixed pattern or fixed method IDS will be sooner or later
unusable.
Recognition of security incidents in traffic patterns can be similar to spam detection in
electronic mail. A wide variety of testing methods exist, but mostly none of them can safely
decide if the mail is spam or it is not. In spam detection a wide variety of testing methods is
usually applied, each giving some amount of penalty (badness) when appropriate pattern was
found. When all the tests are applied, sum of penalty points is counted. If the sum is higher
than specified threshold, mail is considered spam.
The same approach can be used in detecting security incidents in Netflow exports. A wide
variety of testing methods is applied on every flow package. Each test can give a penalty to the
appropriate host IP. When all the tests are applied, sum of penalty points is counted. If the
sum is higher than specified threshold, then there is probably something wrong with the host.
12 CHAPTER 2. NETFLOW BASED NETWORK IDS
NNIDS should be extendable in methods, tests and policies applied to every flow package.
Modular architecture will assure that no design changes in future will have to be made in
order to extend it. Since the traffic patterns in nearly the same networks can be very different,
method, which works perfectly in one network can provide many false–positives or can be
completely unusable in the other network. User should be able to tune individual modules to
fit his/her needs or enable/disable certain modules completely.
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 13
3 NNIDS design and implementation
To analyze network traffic by Netflow certain components have to be deployed in the network
infrastructure. An example of network with deployed Netflow has been already shown in
figure 1.1. The same components are used for incident detection purposes.
At least one source of the Netflow data (i.e. Netflow probe) has to be deployed. The
Netflow probe collects short-time information about the forwarded traffic in flow records and
sends the records using Netflow Data Export to a Netflow Collector. One Netflow Collector
usually receives, processes and stores records from many probes. These records are taken as
an input for calculations for incident detection purposes. When security incident is detected,
alert should be triggered and administrator notified.
All of the components mentioned above already exist. Most of them already exist even in
open-source world and thus can be deployed without additional license fees. Netflow probe
functionality is in most cases already part of the router software or can be easily installed.
High quality Netflow Collectors exist in both commerce and open-source world, so it’s not
wise nor necessary to re-implement them as a part of NNIDS. Nearly every network is running
monitoring system, which is watching hosts/services and alerting prospective faults and the
network administrator would very likely prefer use of escalation schemes already configured in
this existing monitoring system instead of separate escalation scheme in the NNIDS. The only
component, that is still missing is the flow processing and incident detection software.
These fact were kept in mind, when designing the NNIDS architecture. The NNIDS does not
care about NDE collection, it should be fed from already deployed collection engines. NNIDS
also does not care about incident escalation. Generally, it could be possible to write a output
module with escalations, but no-one should need it. The output module should just format
the output data and send them for further processing to syslog server or network monitoring
system, which should deal with the escalations instead. We can imagine NNIDS simply as an
black box, which takes flow records on the input and puts alert messages on the output. An
example of NNIDS deployment can be seen in figure 3.1.
3.1 Existing components
Although the non-NNIDS components were not subject of this thesis’ implementation and ex-
isting solutions were chosen instead, they are tightly bound with it and their correct deployment
is crucial for NNIDS to work correctly. Short discussion about already existing components
will follow.
3.1.1 Netflow probes
Netflow capable hardware routers and switches are in most cases the deciding factor, if the
network is able to use NNIDS or not. In multilayer switched networks things are a little more
complicated, since Netflow is usually enabled only in high-end switches. In routed networks
the infrastructure will almost surely support Netflow Data Export. Same situation is in server-
routed networks, because there is a large number of both user space and kernel space Netflow
probes for Unix and Windows based operating systems.
NNIDS currently supports Netflow v5, so the probe should provide v5 export. But the
implementation is not bound to specific a Netflow version by it’s nature and can be extended
to any version in the future. These days version 5 export is the most common in all kinds
14 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
Figure 3.1: NNIDS deployment scheme
of devices and should provide the least trouble in deployment. Additional instructions about
Netflow probe configuration will be mentioned further.
When configuring Netflow in a complex infrastructure on many routers and interfaces, it
is important not to duplicate information about flows. The duplication can occur for example
on certain Netflow probes, when exporting flows on all interfaces of a single router in both
directions (inbound and outbound). Single flow is thus recorded twice – first time in a inbound
direction of input interface and second time in a outbound direction of output interface. Similar
duplications can occur on two routers each with Netflow enabled interface in the same direction.
Detection of these duplicates can be hard or even impossible to find and can make confusion
to NNIDS (but generally any other Netflow analyzing software). Duplication avoidance can be
assured with careful Netflow configuration. Recommended approach is to enable Netflow only
in inbound direction on the network border interfaces, while other directions and non-boundary
interfaces have Netflow disabled.
Successfully tested Netflow probes were Cisco IOS 12.2, 12.3 and 12.4 in hardware router
world and fprobe-ng in server-router world. Known non working probe is the old fprobe, since
it exports wrong time information, which makes confusion to the block scheduling algorithm
in NNIDS.
3.1.2 Netflow Collector
Netflow Collector is a place where the probes are sending their export to. Collector usually
stores the received flows locally, but is also able to replicate them (by unicast or multicast
groups) to other collectors, aggregate them or do simple reporting. For smaller or low traffic
networks only a single collector may be deployed. Since flow collection does intensive i/o
operations, very large and extremely high traffic networks may need to deploy several collectors
for reliable NDE capture.
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 15
Figure 3.2: NNIDS design
NNIDS has been tested and developed with the flow-capture collector, which is part of the
flow-tools package. Flow-tools is library and a collection of programs used to collect, send,
process, and generate reports from NetFlow data. It is one of the best Netflow suits in the
open-source world and has been even recommended by CAIDA1 , well known authority in global
Internet infrastructure analysis.
3.2 NNIDS
Since this thesis is some sort of proof of concept implementation of NNIDS, Perl programming
language has been chosen to speed up the development process. Perl provides fast enough
run-time environment, can be run on nearly any platform and the development cycle is much
faster, than development cycle of the same project in C programming language. If the concept
will prove good and Perl’s performance will become a bottleneck, project can be rewritten in
C programming language.
Design of the project is displayed in figure 3.2. Input data are read by one or more input
modules. Input can be anything that module can read – raw Netflow data, command output,
socket, etc. Input module transforms this data into separate flow records and distribute them
between one or more method modules. Method modules do the main penalty calculation for
1
Cooperative Association for Internet Data Analysis – provides tools and analyzes promoting the engineering
and maintenance of a robust, scalable global Internet infrastructure
16 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
every detected IP address. When all input data are read and penalties calculated in every
method module, data structures with IP address and penalty pairs are sent to the enumerator
module. This module makes sums of penalties from every method module for every host.
When the penalty is higher than specified threshold, proper output record (containing the
total penalty and list of modules, that gave non–zero penalty) is sent to every output module.
Output module takes the output record and transforms it into desirable form (standard error
output message, syslog entry, etc.).
Program is controlled by a .ini–style configuration file, commonly used in a wide variety
of software. INI configuration style has been preferred over the XML style, because NNIDS
configuration is simple, complex nature of XML would make the configuration file more com-
plicated to read, and lastly INI style and similar formats are more common in user editable
configurations in Unix world. Simple NNIDS configuration file can for example look like this:
[Global]
state file = /home/andrew/dp/nnids.state
critical badness = 10
warn badness = 5
[Input Flowtools]
data path = /home/andrew/dp/flow2
[Output Stderr]
[Method TopN]
max octets = 10000001
[Method Scan]
max ports = 64
Configuration is divided into sections, each section starts with declaration [Name of section].
Currently there are defined 4 types of sections. Section “Global” controls main NNIDS functions
(enumerator, state information, retention policies, etc.). Section “Input” (followed by the
module name) contains configuration parameters for particular input module and also instructs
the NNIDS to load this module automatically. Sections “Output” and “Method” are similar
and contains configuration for output or method modules respectively.
Program code is in a single file nnids, code of every module is in a separate Perl module
file Module name.pm. Path to the modules can be specified in a configuration file. Program is
supposed to be run from cron jobs, or similar scheduling system, or from endless loop scrips
(mainly on heavy loaded systems). NNIDS also accepts certain command line arguments,
mainly intended for debugging or configuration tuning purposes. Perl is not strictly object
oriented programming language, but with a little bit of good will can act like that. Every
module is thus written as a Perl object, while the NNIDS core is written in traditional way.
Object oriented module design eases module’s dynamic loading and data separation.
The whole system was developed in Perl 5.8.8 on NetBSD 4 operating system, but has been
tested on GNU/Linux and should work on any operating system, running Perl 5 or newer and
proper depended modules (Storable, Config::IniFiles, Getopt::Std).
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 17
3.2.1 Main logic
Network traffic is very bursty. Network scan, worm spread or even DoS attack must last
for certain time period to distinguish it from the “normal” network behavior just upon its
traffic pattern. More irregular the incident is, shorter time period is needed for distinguishing.
Therefore, when detecting incidents, the data should be measured over a period of time for
reasonable rate of successful detection.
NNIDS measures data in “time blocks”. Time blocks are small units of constant length
(15 minutes by default), where the flows are parceled into. On the first NNIDS run, when the
very first flow record is read, it’s time-stamp is considered time zero and new time block is
created. All subsequent flow records will be assigned to this time block. When the flow record
with time-stamp later than time block’s ending time is read, this time block is closed and new
one is created. New time block’s starting time is equal to previous block’s ending time. Last
read flow record and all subsequent ones are being assigned to this new time block. Block is
closed and new one is created again, when flow record with non fitting time-stamp is read. This
cycle repeats forever (or until last input record is read... whichever comes first).
When any time block is being closed, results and penalties are calculated by every method
module. Since “data should be measured over a period of time” and “time blocks are small
units”, results are calculated across several time blocks (last 8 by default). Specified number
of time blocks is always saved for next calculation, older time blocks are being deleted.
Way how the penalties are calculated is independent and can be freely implemented in any
method module. Recommended and probably the most common approach should be calculation
of penalties per source IP address. Output of method modules is hard-coded and must be
always based on the IP address. In the future, aggregator module can be implemented, which
can group the results into families of related hosts (for example per IP network grouping or per
AS grouping), if needed.
NNIDS is designed to exit, when the last record has been read. It does not wait until
next flow records will come, it simply prints out the actual results and exits. When reasonable
amount of flow records will arrive, NNIDS should be run again to measure the new records.
This approach has been chosen, because many Netflow collectors are buffering the captured
flow records in memory and provide them on the output only after a certain period of time.
This is not as bad approach as it might seem, since the buffering gives performance benefits
caused by the block nature of disk i/o operations and can also provide some post-processing
benefits in a form of flow compression or flow files meta-data for easy searching (which might
be hard to implement in on-the-fly storing method).
So we run the NNIDS, it will count up the results and exit. After 15 minutes, next flow
file will appear, NNIDS will be started again and since it has to use (by default) 8 time blocks
15 minutes each, it should again count up 7 previous and 1 new time blocks to provide the
results. Only the last one time block contains the new data, previous time blocks have been
already counted before and contain no new data. This would be very inefficient, hence NNIDS
uses state file.
State file provides a place for every input and method module, where it can store arbitrary
data structure. This structure is loaded when NNIDS starts and is saved when NNIDS exits.
In the state file are (or should be) stored results from the previous run, already counted time
blocks, last position in the input file, etc. Whatever the module might require in the next run
can be stored in the state file. The same file is used for both program core and modules. State
file is created using Perl’s Storable module, which is highly optimized and implemented in C,
so it should be fast enough for even large data structures.
18 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
Since the security incident might last for a long time and output messages are generated after
every time block completion, one incident can create many messages. To avoid that, retention
mechanism has been implemented. When retention is enabled, only the first occurrence of
security incident of the same severity is reported per one IP address. When the same severity
should be reported for the same IP in the next block (or run), no output message is generated.
If the severity has changed or when the severity goes back to the normal state, message is
generated. To remind the administrator after a while, maximum number of message retentions
can be specified (8 by default). When more than specified number of messages has been
suppressed, message is generated again.
3.2.2 Program life-cycle
When the program is started, it scans for configuration file. Since the configuration file contains
names of all used modules it must always be present and must always contain at least one input,
one output and one method module definition. If anything above is not present, program dies
with an error message. When the configuration file is parsed, every method module is loaded
and instantiated using their load() constructor. If loading of any module fails, again – program
dies with an error message.
After initialization of all modules, state from the previous run is loaded. When no state
file exists, default values are chosen instead. State information is passed to every module using
their loadstate() method.
The main cycle buffers flow records received from the input modules (by input module’s
getflow() method) and distributes them among all method modules, by calling their putflow()
method.
When the end of the time block has been detected (non-fitting flow records has been read
or there are no flow records left), all method modules are informed by their newblock() method.
Every method module is then probed for it’s results (getresults() and the resulting data struc-
tures are passed to the enumerator. Enumerator groups the results from all method modules
by their IP addresses. For every IP address, sum of all penalties is counted and according to
the configured thresholds, severity levels are assigned. If retention has been enabled, retention
counters are adjusted. As a last step, IP addresses with normal results, which are not subject
of retention recovery, are deleted from the structure. The result is a set of output records,
which is passed to every output module by their output() methods.
This cycle is repeated, until all flow records from all input module are processed. When
the last record has been processed, state information from all input and method modules is
gathered, state structure is built and saved into the state file. After that, program exits.
3.2.3 Implemented input modules
3.2.3.1 Flowtools
This input module reads flow records in a format, used by all utilities from the Flow-tools
package[10]. For NNIDS purposes, the most important utility will be Netflow Collector called
flow-capture.
When the module is being loaded, it creates a list of regular files in a specified paths. When
state information is available, it directly opens the last open file on the last used position.
When no state information is available, first (and oldest) file is chosen and open. For perfor-
mance reasons native file parsing has been reimplemented. Internal flow-tools format is not
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 19
Main header
Bytes Content Description
0-1 magic Magic number 0xCF10
2 endian Endian used (1 = little, 2 = big)
3 hdr ver Flow-tools header version (1 or 3)
Index (version 1 or version 3)2
Flow records (version 1 – 9)3
Table 3.1: Flow-tools file format
Bytes Content Description
0-1 d version Data version (1,5,7,8)
2-5 start Start time of flow capture
6-9 end End time of flow capture
9-12 flags Flags (complete/incomplete, compressed/not com-
pressed, multiple PDU/single PDU, streaming/file,
old format/new format, preloaded header)
13-16 rotation Rotation schedule
17-20 flows Number of flow records
21-24 pdu drops Number of dropped records
25-28 pdu misordered Number of misordered records
29-96 hostname 0 terminated name of capture device
97-352 comments 0 terminated ascii comments
Table 3.2: Flow-tools index version 1 format
documented (at least the author was unable to find any documentation related to it), so its
structure had to be learned from the source code of the flow-tools package.
Every flow-tools file begins with the 4-byte header. It contains magic numbers 0xCF, 0x10,
byte order (endianity) and header version. As the flow-tools had been developed, two header
versions has evolved, which differ in second part of the header – index. Index contains meta-
data about flows captured in the file, like oldest and newest flow’s time-stamp, number of flows
in the file, if the records had been compressed, if it’s a regular file or just a piped stream from
other utility, etc.
Index version 1 has fixed size and format (see table 3.2). Version 3 uses sequences of
type/length/value triplets instead. Currently there are only 13 out of 65535 TLV’s defined
(see table 3.4), other might be allocated in the future. There must always be 2 crucial items
specified in the header. First – what Netflow version records are stored in the file and second
– if the records are stored in the plain or compressed form. These two items describe what
format will the flow records part of the file have. Record’s format is based on the actual Netflow
version’s PDU format and is very similar to it. Flow records compression is something new
and let us save a disk space by using zlib compression on the flow records part of the file.
3
Index format depends on byte hdr ver in the main header. See table 3.2 or 3.3 for internal structure of both
versions.
3
Flow record’s format depends on byte d version (when version 1 index is used), or TLV type 0x2 (when
version 3 index is used). See table 3.5 for Netflow version 5 record format structure.
20 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
Bytes Content Description
0-3 hdr length Header length (main header + index)
0-1 type Type (int16)
2-3 length Length (int16)
4-... value Value (type dependent)
0-1 type Type (int16)
2-3 length Length (int16)
4-... value Value (type dependent)
...
padding Padding to align header size to a multiple of 32 bits
Table 3.3: Flow-tools index version 3 format
Type Length Value
0 0 Null type. Used for alignment only.
1 2 Vendor code – 1 = Cisco. (uint16)
2 2 Export version – 1,5,7,8,9. (uint16)
3 1 Aggregation version (uint8)
4 1 Aggregation method (uint8)
5 4 IP of exporter (uint32)
6 4 Capture start time (uint32)
7 4 Capture end time (uint32)
8 4 Flags4 (uint32)
9 4 Rotation schedule (uint32)
0xA 4 Number of flow records (uint32)
0xB 4 Number of dropped flow records (uint32)
0xC 4 Number of misordered records (uint32)
0xD 4 Number of corrupted packets (uint32)
0xE 4 Sequence reset (uint32)
0xF X 0 terminated name of capture device (string)
0x10 X 0 terminated ascii comments (string)
0x11 X Interface name (struct)
0x12 X Interface alias (struct)
Table 3.4: Flow-tools type, length and value (TLV) triplets for index version 3
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 21
Bytes Content Description
0-3 unix secs Current seconds since 0000 UTC 1970 (uint32)
4-7 unix nsecs Residual nanoseconds since 0000 UTC 1970 (uint32)
8-11 sysUpTime Current time in millisecs since router booted (uint32)
12-15 exaddr Exporter IP address (uint32)
16-19 srcaddr Source IP Address (uint32)
20-23 dstaddr Destination IP Address (uint32)
24-27 nexthop Next hop router’s IP Address (uint32)
28-29 input Input interface index (uint16)
30-31 output Output interface index (uint16)
32-35 dPkts Packets sent in Duration (uint32)
36-39 dOctets Octets sent in Duration. (uint32)
40-43 First SysUptime at start of flow (uint32)
44-47 Last and of last packet of flow (uint32)
48-49 srcport TCP/UDP source port number or equivalent (uint16)
50-51 dstport TCP/UDP destination port number or equiv (uint16)
52 prot IP protocol, e.g., 6=TCP, 17=UDP, ... (uint8)
53 tos IP Type-of-Service (uint8)
54 tcp flags OR of TCP header bits (uint8)
55 pad Zero byte (uint8)
56 engine type Type of flow switching engine (RP,VIP,etc.) (uint8)
57 engine id Slot number of the flow switching engine (uint8)
58 src mask mask length of source address (uint8)
59 dst mask mask length of destination address (uint8)
60-61 src as AS of source address (uint16)
62-63 dst as AS of destination address (uint16)
Table 3.5: Flow-tools flow record version 5 structure
A little special is the streaming format, which can be recognized by the 4th rightmost bit
in the “Flags” item of the index. Since many flow-tools utilities provide piped output of flow
records (for example flow-cat /flows/krc4 | flow-filter -P80 will provide only flows
destinated to port 80 on flow-filter’s standard output), file format has to reflect this. That’s
why “streaming” bit has been introduced. When application detects this bit set, it knows it
must not rely on the flow counters index items when reading flow records, but must do !feof()
loop to read them. Streaming files had no information about number of flows or starting/ending
times and cannot be compressed (since compression is being done on the whole records buffer,
when file is being saved).
NNIDS expects files created by the flow-capture, thus only Netflow version 5 format (both
compressed and uncompressed) support is implemented (see table 3.5), streaming files are not
supported. Version 1 index is considered obsolete, it’s support has not been implemented, but
might become a subject of future work.
3.2.3.2 cflowd
cflowd[9] is an old, but still popular flow analysis tool used for analyzing Cisco’s NetFlow en-
abled switching method. It was written by Daniel W. McRobb in 1998 and has been supported
by CAIDA for many years. These days it’s no longer supported and users were instructed to
4
See item “flags” in the index version 1 (table 3.2) for further description
22 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
migrate to flow-tools package by Mark Fullmer. Despite its age and stability problems, its still
being used in certain sites and some utilities still need its presence.
NNIDS is able to read the cflowd file format using supporting utilities flow-export and flow-
import. cflowd input module is much slower in reading records, than Flowtools module, but
still can be used in cases, where migration to flow-tools package is impossible.
3.2.4 Implemented output modules
Output modules are the simplest kind of modules used in NNIDS. Their task is just to format
properly the output records sent by the enumerator and encapsulate them in the desired format
or protocol. Since certain output modules might have difficulties with output retention (for
example statistic generating modules), output records are always passed to the output modules
including retent records. Every record has a retention flag, which indicates if the record should
be subject of retention or not. Then it’s every output module’s responsibility to respect this
flag and generate output messages in accordance with it.
3.2.4.1 Stderr
Stderr is the simplest output module. It just takes the output records, formats them and print
them out to the Unix standard error output. It’s intended for debugging or additional scripting
purposes. The typical output might look like this:
cofein:~/dp> ./nnids.pl -s
Sat Apr 28 21:10:14 2007: CRITICAL: 192.168.9.1 has badness 20:
TopN (10): Transferred 169256 bytes (4185 packets) in 4117 sessions
Scan (10): Host does a port scan against 192.168.9.4
Line begins with time block’s start time (Sat Apr 28 21:10:14 2007), continues with severity
level (CRITICAL), subject’s IP address (192.168.9.1) and badness total (20). Then every
module, that gave non-zero penalty is listed, with given penalty and description message.
3.2.4.2 Syslog
Syslog output module uses the standard BSD syslog protocol[11] to send messages to local or
remote host. Target host must be running properly configured syslog server. Messages can be
delivered to the server using 3 methods: local socket, udp network transport and tcp network
transport.
Local socket is used to deliver messages to the syslog server, running on the same host as
the NNIDS runs. This socket is placed in /dev/log on most unix platforms (or at least there’s
a symbolic link, that leads to the proper socket location) and is created on syslog server’s start.
UDP is the most common and these days the only IETF recommended way of transporting
syslog messages to the remote server. Messages are sent to the remote syslog server, listening on
the UDP port 514. TCP transport has not been standardized yet, but is already implemented
in certain syslog servers (syslog-ng for example). Since syslog transport over UDP has no
mechanisms to prevent or detect message replaying or dropping, TCP transport provides benefit
in addressing these issues, because of the reliable transport nature of the TCP protocol. On the
other side this brings another issues like more overhead or more possibility of denial of service
by opening many connections.
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 23
Syslog message received and formatted by the standard BSD syslog daemon may look in
the log file like this:
May 10 20:18:35 cofein nnids: Sat Apr 28 21:10:14 2007: CRITICAL: 192.168.9.1
has badness 20 (mods TopN,Scan).
The first date (May 10 20:18:35) is the date when the message was received. Hostname
(cofein) and name of the application (nnids) follow. Second date (Sat Apr 28 21:10:14 2007) is
the date when the security incident has been detected. Then severity of the incident (CRIT-
ICAL), subject’s IP address (192.168.9.1), badness total (20) and every module, that gave
non-zero penalty (TopN,Scan) are listed.
3.2.4.3 Mail
As the name implies, Mail output module uses Simple Mail Transport Protocol defined in
RFC 2821 [12] to notify administrator about detected incidents or incident recoveries. For mail
sending in Unix world, sendmail binary exists, which does this job. In traditional systems,
this binary reside in /usr/lib/sendmail and usually it is the main binary of the Sendmail mail
system. In newer systems, Sendmail was often replaced by other products (like Postfix, Exim
or Qmail), which place wrapper script in /usr/lib/sendmail to achieve interface compatibility
with the old Sendmail system.
Even the Mail output module utilizes this sendmail interface for sending notifications. While
other specified modules send notifications every time the output() method is invoked, Mail
buffers the messages and sends only one notification (mail), when NNIDS is going to quit
(in unload() method). This makes it little exceptional when compared to output modules
mentioned before. Resulting mail message looks like this:
Date: Thu, 10 May 2007 20:45:33 +0200 (CEST)
From: admin@somewhere.net
To: admin@somewhere.net
Subject: NNIDS events
Following events were detected by last run of NNIDS:
Sat Apr 28 21:10:14 2007: CRITICAL: 192.168.9.1 has badness 20:
TopN (10): Transferred 169256 bytes (4185 packets) in 4117 sessions
Scan (10): Host does a port scan against 192.168.9.4
Faithfully yours, etc.
The output format is very similar to the format used in the Stderr module and is fully
described in section 3.2.4.1 on page 22.
3.2.4.4 Nagios
Nagios is a system and network monitoring application [12]. It watches specified hosts and
services, alerts administrators when things go bad and when they get better. Nagios is run
in the background as a daemon and checks hosts and services in periodic intervals. Result of
the check can be OK, WARNING, CRITICAL or UNKNOWN. When the check returns state,
24 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
Figure 3.3: Nagios user interface
which is different to previous one, notification is executed. Notification can be any prescripted
action, usually it is mail or SMS message. Nagios has nice web graphical user interface and is
able to visualize service/host availability, trends, alert histograms, etc.
Nagios “check” can be also any prescripted action done usually by a Nagios plug-in. Plug-
ins are small pieces of software (small C program, shell scripts,...) developed and distributed
separately from the main Nagios system. Nowadays there are hundreds of plug-ins for various
purposes – checking health of disks, application response on specific protocol challenge, health
of SQL databases, etc.
There are 2 basic check types – active and passive. Active check is started upon Nagios’
request. For example Nagios knows, it should check local disk space every 10 minutes. It sets
the timer and when the timer expires, proper plug-in is executed. Plug-in checks the free spaces
and compares it with the specified thresholds. When available disk space is beyond specified
thresholds it returns result message CRITICAL to the Nagios. Nagios takes this message and
does a predefined action – for example it notifies the administrator, that disk is almost full.
When using passive check, Nagios does not care how it will be started. It does not execute
any plug-ins, just waits and when a message appears on its control socket, it assigns its result
and output to the proper service/host. Passive checks can be executed by cron, by administrator
or the messages can be directly generated by running system daemons. In passive checks Nagios
just passively waits, until check results will somehow appear on its control socket.
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 25
When we are checking health of local system, plug-ins can be simply executed locally. For
checking remote host, we have to run the proper plug-in on the remote host and put the result
to Nagios. For this purposes nrpe (Nagios Remote Plug-in Executor) suite has been developed.
It consist of the nrpe plug-in (client) and the nrpd daemon (server). nrpd is running on the
remote hosts and listens for incoming requests (5666/TCP by default). When Nagios wants
to run check on this host, it runs the nrpe client, which connects to this remote host on the
nrpd port and tells name of the desired check. nrpd then looks into its configuration file, where
proper plug-in command for the specified name is stated. nrpd runs the command and passes
the result to the Nagios.
Similarly to the nrpe plug-in for active checks, nsca plug-in (NetSaint Check Acceptor) for
the passive checks was developed. This time nsca server runs on the same host, where Nagios
is running. When the remote host want to submit its check results to the Nagios, it uses utility
send nsca. send nsca connects to the nsca server and submit the results. nsca takes this results
and passes them to the control socket of Nagios.
Active checks are used more often, since scheduling of checks can be controlled from a single
point – from the Nagios. Drawback of this approach is, that Nagios server must be able to reach
all the remote hosts on certain port. This may be hard to achieve for example when checking
many hosts behind NAT. Also, certain services are asynchronous in nature (like SNMP traps)
and thus cannot be actively checked. Passive checks might be a solution in these cases.
NNIDS supports both active and passive method of submitting results. If nrpd would
invoke NNIDS in active checks, NNIDS’s computation might take very long time and since
nrpd expects plug-in’s results in a specified timeout (60 seconds by default), this approach
might be unusable in many cases. Thus when active check is set, NNIDS puts the results in
a specified file. nrpd is invoking just a wrapper script check nnids.pl, which takes the results
from this file and passes them to Nagios. Passive checks are not affected by the timeout
problems, so results are passed directly to the send nsca.
3.2.5 Implemented method modules
Motivation and principles of methods used for security threats detection has been already
presented in chapter 2.1 on page 8. This chapter is a reference list of implemented methods
and describes implementation details of the particular modules.
3.2.5.1 TopN
The TopN method module is used for checking amount of traffic generated by every host.
No healthy and correctly acting host should send more than specified amount of data. Hosts
that violate this policy are probably doing something wrong (massive network scan, aggressive
download of data or computer virus infection).
Since every flow record represents traffic flowing in just one direction, two flow records are
always generated to represent one real bidirectional flow. TopN module thus measures data only
on the source IP address basis. For every detected source IP address 3 counters are maintained
– total number of bytes generated by the IP address, total number of packets generated by the
IP address and total number of flow records generated by the IP address.
These numbers are counted and summed within every time block (see 3.2.1 on page 17)
and are for every time block hold separately. When getresult() is called, precalculated sums
for every IP address are calculated across all time blocks – i.e. total of totals is calculated
for every IP address. These results are then compared to minimum and maximum thresholds
26 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
14
max octets
13
12
11
10 max badness
9
8
Badness
7
6
5
min octetes
4
3
2
1 min badness
0
10 101 102 103 104 105 106 107 108
Bytes transferred [bytes]
Figure 3.4: Badness calculation in the TopN module (mind the logarithmic scale on x axis)
and according to these thresholds, badness to every IP address is assigned. Badness is being
assigned from the range between the specified minimum and the specified maximum linearly,
depending on the actual amount of bytes/packets/flows transferred. In the config file, there
are always minimum and maximum thresholds specified for bytes, packets and flows. For every
source IP address, three badness numbers are calculated and the highest is reported to the
enumerator module. Example badness behavior for traffic in bytes is shown in figure 3.4. In
this example, min badness is set to 1, max badness to 10. min octets is set to 1000 bytes, max
octets to 1 000 000. When given IP generates less than 1000 bytes of traffic in all measured time
blocks, zero badness is assigned. When it generates more, than 1 000 bytes, badness is assigned
and its number depends on the actual bytes transferred. Maximum badness 10 is assigned when
host generates 1 000 000 bytes. When the host generates more traffic than 1’000’000 bytes,
badness will not raise any more – it will still be 10.
In the network, there are certain devices like file servers or network printers, that can legally
produce high amount of traffic, while other network devices are deeply below this number. For
these cases there is configuration directive ignore hosts, which contains networks and hosts,
that are not included in the calculations.
3.2.5.2 Scan
Scan method module is tracking how many destination IP addresses is the given source IP
address communicating to and how many ports with destination port number < 1024 is it
connecting to. Using these two methods port scan (destinated to single IP) or host scan
(destinated to many IP’s) originated from the source IP address can be identified.
Scan module stores every destination IP address per given source IP in a Perl hash. Hash is
a special type of array implemented in Perl, which uses arbitrary string or number as a unique
CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION 27
key. Searching in the keys is implemented deep in the Perl internals and is very fast. Both
IP addresses are used in the hash as a key. When the module is computing the number of
destination IP addresses, source IP address was talking to, only sub-hash size is counted.
Since we want to also detect port scan, which is destinated to a single IP address, destination
port numbers are also stored in the hash. Storing information about any destination port would
result in a very huge hash even on very low traffic networks. When doing port scan, attacker
is most interested in privileged ports, which are used for most of the system services like ssh or
httpd and can’t be used by normal (not privileged) users. Thus only destination port numbers
below 1024 are stored and counted when detecting port scan.
As when measuring amount of the generated traffic, also scan module might detect false
positives – hosts that communicate with large number of other hosts (name servers, heavily
used web-servers, etc.). These hosts should be ignored in statistics using configuration directive
ignore hosts.
3.2.5.3 PatternIP
Searching of certain IP address’s appearance in the traffic can also contribute to network
anomaly or misconfiguration detection. This module was originally intended as NNIDS’s im-
plementation of Team Cymru’s Bogon list [13]. When source IP from the specified range is
detected, badness is assigned.
Allocation of IP addresses in the public Internet is maintained by the Internet Assigned
Numbers Authority (IANA). This organization assigns large blocks of IP addresses (typi-
cally /8) to Regional Internet Registries (RIPE, ARIN, APNIC, LACNIC and AfriNIC), which
separate them into smaller blocks and assign these blocks to individual organizations.
There are many addresses, that can appear in the public Internet and there are many that
should not. To distinguish this groups, IANA’s IPv4 address space document [14], can be
used. Any block that is in “reserved” state should not appear in the public Internet. In private
networks, which are connected to the Internet through NAT, RFC 1918 addressing can be used.
In these cases the individual used ranges should be removed from the bogon list.
3.2.5.4 PatternIP
This module is intended mainly for smaller networks. When given source IP address is trying
to contact specified destination port, badness is assigned. This is useful for detecting packets
to well known services, which are not being used – for example use of port 1433/tcp does not
foretell anything good, when you have no MS SQL client or server in your network. UDP
and TCP port numbers are detected and configured separately. Any occurrence of given port
number will result in specified badness.
3.2.5.5 DoS
The most obvious symptom of Denial of Service or Distributed Denial of Service attacks in the
traffic pattern is enormous growth of traffic destinated to the victim IP address. TopN module
does traffic accounting, so it might be able to detect DoS, doesn’t it? Well, the TopN module
accounts traffic generated by the source IP, not destinated to the IP. DoS module thus has
been introduced, which does the same thing as TopN does, but it accounts information on per
destination IP address basis.
28 CHAPTER 3. NNIDS DESIGN AND IMPLEMENTATION
It’s principles, badness algorithm and configuration options are the same, as at the TopN
module. Minimal and maximum thresholds of octets, packets and flows, which assigned non zero
badness are specified. Badness is a linear monotonous function between these two thresholds
(see figure 3.4).
3.2.6 Method modules in a real network
Method modules do the “dirty work” in NNIDS and they are intended to be as much flexible as
possible. Since every network is different, method module parameters should be tuned carefully,
before NNIDS can be run with minimum number of false positives – small network will very
likely have absolutely different TopN thresholds to server housing provider.
Hosts, that legally generate high volume of traffic or high number of connections to a large
number of hosts, always make confusion to the method modules, since their traffic pattern is
absolutely different to traffic of other hosts and it’s character is often very close to the malicious
traffic. These hosts should be held in the ignore list to not affect thresholds of ordinary network
hosts. Tuning method modules to accept traffic generated by a single file server, when there are
100 clients with much lower traffic volume, would only degrade modules accuracy, it is better
to ignore it.
Networks with only several sources of high volume traffic or high number of connections
should be able to use all the method modules mentioned before with no serious problems and
it should be possible to tune them using techniques discussed above. But there are networks,
where these sources are densely spread across the network and it might be impossible or needless
to fit this network’s needs in specific module’s configuration. For example it’s very hard to
detect a host scan in a content delivery network. In these cases certain modules can be safely
disabled (removed from the configuration) with no high impact on the NNIDS functionality,
cause these modules will by their nature never work well in this environment.
CHAPTER 4. TESTING AND IMPROVEMENTS 29
4 Testing and improvements
4.1 Performance testing and improvements
During the development of this product, it has been tested on several sites and certain improve-
ments and changes has been made throughout this process. First performance issues arised,
when enumerator has been fully implemented. Processing of flow file with 153 391 record took
about 64 seconds of pure CPU time on Pentium M 800MHz, which was unexpected and very
poor result.
Profiler analysis showed, that 29.37 seconds of CPU time is spent in maincycle(), which
distributes flow data among all the modules, 17 seconds are spent in getflow(), which gathers
data from flow files, 8 seconds are spent each method’s module putflow() function and about
10 seconds are spent in Memoize functions, which were intended to speed up IP conversion
functions. Profiler output follows:
Total Elapsed Time = 81.29892 Seconds
User+System Time = 64.73892 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
45.3 29.37 72.746 1 29.378 72.745 Main::maincycle
26.3 17.05 27.617 153392 0.0001 0.0002 Input::Flowtools::getflow
12.7 8.263 8.263 153391 0.0001 0.0001 Method::TopN::putflow
11.7 7.625 7.635 306782 0.0000 0.0000 Memoize::_memoizer
11.5 7.483 7.483 153391 0.0000 0.0000 Method::Scan::putflow
3.89 2.521 10.155 306782 0.0000 0.0000 Memoize::__ANON__
0.59 0.384 0.411 970 0.0004 0.0004 Input::Flowtools::opennextfile
0.12 0.080 0.167 9 0.0088 0.0185 Main::BEGIN
0.08 0.050 0.069 4 0.0124 0.0171 Input::Flowtools::BEGIN
0.05 0.032 0.029 1 0.0321 0.0288 File::Find::_find_dir
0.03 0.021 0.021 1128 0.0000 0.0000 Main::debug
0.03 0.020 0.020 1 0.0200 0.0200 re::import
0.03 0.020 0.048 1 0.0199 0.0484 Input::Flowtools::load
0.03 0.020 0.020 4 0.0050 0.0050 Config::IniFiles::BEGIN
0.02 0.010 0.010 3 0.0033 0.0033 vars::BEGIN
There were certain implementation problems identified in the NNIDS by profiling analysis.
Whole data structures were gathered from input modules and passed to the method modules,
i.e. data structures were always duplicated. NNIDS has been thus rewritten to pass only hash
references (pointers in Perl) to the function. This made NNIDS about 20 % faster. Memoize
was removed from the sources, which caused about 15 % performance improvement.
After these changes NNIDS was still not fast enough, because it used flow-export utility to
convert flow files into ASCII representation, which has been parsed by the input module. flow-
export is very fast, but parsing input ASCII data into an array and conversion from strings
into numbers in the input module was slow. Thus, native flow-tools file support has been
implemented. To decompress flow files with compressed flow records Compress::Zlib module
written in C has been chosen. To parse the records unpack Perl function is used, which is again
implemented in pure C and thus very fast. This function parses the data into integer variables,
so no additional conversion is needed. After this rewrite, NNIDS was 4.5 times faster, than
before. Profiler output follows:
30 CHAPTER 4. TESTING AND IMPROVEMENTS
Total Elapsed Time = 16.67346 Seconds
User+System Time = 14.65346 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
40.4 5.927 18.261 1 5.9274 18.260 Main::maincycle
34.2 5.025 5.994 153392 0.0000 0.0000 Input::Flowtools::getflow
15.8 2.323 2.323 153391 0.0000 0.0000 Method::TopN::putflow
9.44 1.383 1.383 153391 0.0000 0.0000 Method::Scan::putflow
6.73 0.986 0.986 444 0.0022 0.0022 Method::Scan::mergestats
3.61 0.529 0.982 444 0.0012 0.0022 Method::TopN::getresults
2.16 0.316 0.316 444 0.0007 0.0007 Method::TopN::mergestats
2.09 0.306 0.448 444 0.0007 0.0010 Main::enumerate
1.93 0.283 0.954 968 0.0003 0.0010 Input::Flowtools::ftread
1.31 0.192 0.192 968 0.0002 0.0002 Compress::Raw::Zlib::inflateStre
am::inflate
0.78 0.114 0.114 13267 0.0000 0.0000 Main::getiptext
0.68 0.099 0.136 968 0.0001 0.0001 IO::Compress::Base::Parameters::
parse
0.61 0.090 0.321 968 0.0001 0.0003 Compress::Zlib::inflateInit
0.59 0.086 0.086 448 0.0002 0.0002 Storable::dclone
0.57 0.083 1.079 444 0.0002 0.0024 Method::Scan::getresults
Later small benchmark was made. NNIDS with TopN module enabled processes flow file
with 153 391 records for 5.499 seconds of pure CPU time on this system. Flow file with 2 804 142
records was processed in 75 seconds of CPU time. Times were measured using NetBSD time
utility on NetBSD 4.99.3 system with Perl 5.8.8. System had Pentium M 2.0GHz CPU and
1GB RAM.
4.2 Operation in real networks
For final NNIDS’s drive test two sites were used. The first one is small engineering company’s
network – about 50 hosts in 8 sub-networks, 3 servers and single router. This site generates
about 500 flow records per minute. NNIDS was configured with TopN, Scan, PatternIP and
PatternPort modules. Since the whole site has centrally managed anti-virus software deployed,
no worm or DoS attack has been detected by the NNIDS, nor the AV software. After tuning
the configuration, several incidents were detected, caused by excessive download of data from
the Internet. One false positive was also detected, when a company employee made a backup
of his notebook to the company server.
NNIDS operation has been tested on this site using network security scanner nmap:
gw:~# nmap -sS 192.168.9.4
Starting nmap 3.81 ( http://www.insecure.org/nmap/ ) at 2007-05-22 11:41 CEST
Interesting ports on xxxxxxxx.xxx.xxx (192.168.9.4):
(The 1660 ports scanned but not shown below are in state: filtered)
PORT STATE SERVICE
139/tcp open netbios-ssn
445/tcp open microsoft-ds
5900/tcp open vnc
CHAPTER 4. TESTING AND IMPROVEMENTS 31
MAC Address: 00:11:25:AB:BD:0F (IBM)
Nmap finished: 1 IP address (1 host up) scanned in 26.124 seconds
This incident was reported properly the NNIDS. Scan originating IP (192.168.9.1) had exces-
sive number of sessions, Scan module detected the scan properly and additionally PatternPort
has detected forbidden destination port 1433:
Tue May 22 11:40:14 2007: CRITICAL: 192.168.9.1 has badness 28:
TopN (8): Transferred 309197 bytes (5437 packets) in 3710 sessions
Scan (10): Host does a port scan against 192.168.9.4
PatternPort (10): Matching ports 1433 (TCP)
Second site tested is one of the largest Czech housing providers. It has thousands of hosts,
mostly servers. NNIDS was used on peering and transit lines, which are transporting about
2.5Gbit/s of traffic. These lines generate about 1 500 000 flow records per minute.
In this environment, Scan module was absolutely useless, since every server is sending data
to a large number of clients and thus generate a host scan in eyes of Scan module. PatternIP
module was also not needed, because ASBR routers were filtering bogon IP’s by themselves.
So the only modules used were TopN and PatternPort.
This site proved, that Perl might not be fast enough to process such a large amount of
data. After tuning the configuration, NNIDS produced no false-positives, but processing of
flows recorded in 140 seconds took about 80 seconds, which provides very little reserve, when
additional modules should be used or when significant traffic increase would arise.
(Un)fortunately no DoS/DDoS attack has been registered during the testing time, but since
DDoS attack in such a high traffic environment would produce extremely large amount of flow
records, it is very likely NNIDS won’t be able to count up resulting flow files in “real time”.
That is why NNIDS should be rewritten in C in future to be able to operate in such a high
traffic environment.
32 CHAPTER 5. CONCLUSION
5 Conclusion
5.1 Finished work
Primary aim of this thesis was to design and implement modular tool, which automatically
detects security incidents and anomalies in the IP network using Netflow analysis. This aim is
considered to be accomplished.
Introduction to Intrusion Detection Systems and the Netflow technology is given in the first
chapter. General approach of Netflow-based incident detection and requirements to such a sys-
tem are specified in chapter 2. Factual design of the implemented system with implementation
details is described in chapter 3.
System has been tested on several sites and according to results of these tests, certain
modifications were made (mainly performance related). System is in functional state and is
able to detect certain security incidents. This has been demonstrated in chapter 4, also with
testing details.
In Appendix A below covers installation and configuration steps, needed for launching the
system from the scratch and recommended approach of configuring the infrastructure, which the
system should cooperated with. Printed version of the reference manual, which is distributed
with the system sources in a form of Unix manual page, is attached in Appendix B.
5.2 Future work
The most significant problem of implemented system might be it’s performance. This perfor-
mance problem arises only when the system is deployed in very high speed networks (more than
20 000 flow records per second generated). In Czech republic, there are a few sites with such
a huge traffic, however with data transfer growth this might become more serious problem in
several years. Performance problem is caused mainly by performance issues of Perl program-
ming language and can be solved by rewriting the project in C. This approach was already
known when designing the project and is discussed in chapter 3.2 on page 15.
There are also certain modules and features, which might be handy to implement. Auto-
matic baseline establishment in TopN and Scan module would result in more granular acting
and their incident detection would become more accurate. In the future, aggregator module,
which joins results of many IP addresses into a predefined group might be implemented.
CHAPTER 6. BIBLIOGRAPHY 33
6 Bibliography
[1] Symantec Corporation: Symantec Internet Security Threat Report, vol. XI
http://www.symantec.com/enterprise/theme.jsp?themeid=threatreport
[2] Institute of Electrical and Electronics Engineers, Inc.: 802.1X – Port Based Network
Access Control
http://www.ieee802.org/1/pages/802.1x.html
[3] Innela, P.: An Introduction to Intrusion Detection Systems
Securityfocus, 2001.
[4] Cisco Systems, Inc.: Cisco IOS Software Documentation
http://www.cisco.com/univercd/home/home.htm
ˇ ı a
[5] Nejman, J., Smrha, P.: Inteligentn´ analyz´tor NetFlow exportu
Technical report, Cesnet, 2001.
[6] Gong, Y.: Detecting Worms and Abnormal Activities with NetFlow.
Securityfocus, 2004.
[7] Computer Emergency Response Team: “Code Red” Worm Exploiting Buffer Over-
flow In IIS Indexing Service DLL
CERT, 2001.
[8] Claise, B.: Cisco Systems NetFlow Services Export Version 9
RFC3954, The Internet Society, 2004.
[9] Cooperative Association for Internet Data Analysis: cflowd: Traffic Flow Analysis
Tool, http://www.caida.org/tools/measurement/cflowd/, CAIDA, 1998.
[10] Mark Fullmer: Flow-tools Documentation
http://www.splintered.net/sw/flow-tools/, 2005.
[11] Lonvick, C.: The BSD syslog Protocol
RFC3164, The Internet Society, 2001.
[12] Klensin, J.: Simple Mail Transfer Protocol
RFC2821, The Internet Society, 2001.
[12] Galstad, E.: Nagios R Version 2.x Documentation
http://nagios.sourceforge.net/docs/2 0/, 2006.
[13] Team Cymru: The Team Cymru Bogon List
http://www.cymru.com/Documents/bogon-list.html, Team Cymru, 2001.
[14] Internet Assigned Numbers Authority: Internet Protocol v4 Address Space
http://www.iana.org/assignments/ipv4-address-space, IANA, 1984.
[15] Slava Astashonok: flowprobe
http://fprobe.sourceforge.net/, 2005.
[16] Samhain labs: The SAMHAIN file integrity / intrusion detection system
http://www.la-samhna.de/samhain/, 2006.
[17] Lehti, R., Virolainen, P., Berg, R.: Advanced Intrusion Detection Environment
http://www.cs.tut.fi/ rammer/aide.html, 2003.
34 CHAPTER 6. BIBLIOGRAPHY
[18] Cisco Systems, Inc.: Cisco Security Agent
http://www.cisco.com/go/csa, 2000.
[19] Computer Associates International, Inc.: CA Host-Based Intrusion Prevention
http://www.ca.com/us/products/product.aspx?id=5785, 2000.
[20] Sourcefire, Inc.: Snort
http://www.snort.org/.
[21] Cisco Systems, Inc.: Cisco IDS 4200
http://www.cisco.com/go/ips.
[22] Cisco Systems, Inc.: Cisco Security Monitoring, Analysis and Response Systems
http://www.cisco.com/go/mars.
[23] Computer Associates International, Inc.: CA Security Command Center
http://www.ca.com/us/products/product.aspx?ID=4351, 2002.
APPENDIX A. INSTALLATION AND CONFIGURATION 35
A Installation and configuration
NNIDS installation and configuration is not a tricky part. In this chapter, recommended proce-
dure how to do it will be described. Since NNIDS might cooperate with Netflow infrastructure
and additional products like syslog server, or monitoring system, basic configuration steps for
these products will be also discussed.
A.1 Netflow infrastructure
In the fact, you don’t need any Netflow infrastructure at all to run NNIDS. NNIDS works only
with export files, so you can use it for example as a tool for post mortem analysis of captured
data. All you have to do is either write your own input module or convert your data to a format,
that NNIDS recognizes and it will work. As you can see – there are no Netflow devices involved
in this case at all. However in most cases you will probably want to analyze (nearly) real-time
data from your network, so you will need to configure Netflow.
A.1.1 Netflow probe
Netflow probe is a device, which takes the data and converts them into Netflow flows. A probe
is obviously any kind of router, but when your router does not support Netflow Data Export,
it is also possible to create flows from data, which are carried to a stand alone Netflow probe
using network tap device or switch port mirroring technology like SPAN.
You should check your product’s configuration manual to find out how to configure your
device as a Neflow probe. NNIDS currently support only the most common Netflow version 5
format. Below are specified instructions how to configure NDE on two most common platforms
– Cisco router and Unix box.
A.1.1.1 Configuring NDE on Cisco router
In a Cisco world, almost all the routers and most of the high-end Ethernet switches are able to
do Netflow Data Export. You should check your product’s configuration guide if it supports it.
NDE configuration is pretty straightforward on most platforms. First you have to set the
NDE destination – i.e. IP address of the Netflow collector and destination port number. Then
Netflow export version should be specified. Optionally you can set the interface, which IP
address will be set as source IP for export packets, BGP AS numbers exports or aging timers of
flow records. When NDE is configured, you have to enable Netflow for each desired interface.
Example configuration might look like this:
ip flow-export destination 10.0.0.1 9800 ! set destination IP and port
ip flow-export source Vlan 10 ! set source IP
ip flow-export version 5 origin-as ! set export version, also export AS
ip flow-cache entries 131072 ! flow table can have max 131072 ent.
ip flow-cache timeout active 20 ! set active flow timeout to 20 min.
ip flow-cache timeout inactive 130 ! set inactive flow timeout to 130s
!
interface GigabitEthernet 0/1
ip flow ingress ! enable Netflow in ingress direction
!
36 APPENDIX A. INSTALLATION AND CONFIGURATION
In IOS releases 12.2(14)S, 12.0(22)S, 12.2(15)T, configuration command used to enable
NetFlow on an interface has changed to ip flow ingress or ip flow egress, depending which
direction you want to gather statistics. In earlier IOS releases the ip route-cache flow command
was used to enable NetFlow on an interface. The resulting configuration in the older IOS
releases might look like this:
ip cef
! The following two lines are for 65xx, 76xx, or relative platforms only
mls netflow ! enable Netflow on the PFC
mls flow ip interface-full ! set flow-mask
!
ip flow-export destination 10.0.0.1 9800 ! set destination IP and port
ip flow-export source Vlan 10 ! set source IP
ip flow-export version 5 ! set export version
!
interface GigabitEthernet 0/1
ip route-cache flow ! enable Netflow on this iface
!
As you can see in the example, some platforms need to have CEF enabled before NDE
can be configured and some platforms (for example 6500/7600) also need to have explicitly
configured MLS flowmask (full-interface) to work correctly. Once again – consult your product’s
configuration guide for further information.
A.1.1.2 Configuring NDE on a Unix box
As far, as the author knows, no Unix based operating system’s kernel has native support of the
Netflow Data Export. There are certain kernel probe projects like the ng-netflow (NetGraph
based kernel module for BSD platforms), but these projects are rather rare and platform de-
pendent. In the user space, the situation is better. Feature rich and well maintained probe is
for example fprobe-ng by Slava Astashonok [15], which will be used in this example. fprobe
performs well on even high traffic networks (> 2.5Gbps) and thus should be suitable in most
cases.
First you have to install the fprobe into your operating system. fprobe package exists in
most operating systems and Linux distributions, so you probably won’t need to compile it on
your own. For example in Debian Linux you should be able to do:
apt-get install fprobe-ng
The installation script will ask you to specify interfaces, collector’s IP address/port, and
installation and configuration is over. In not so intuitive systems you might need to run fprobe
from the script or command line. Basic command line syntax is:
fprobe -i interface name collector IP:UDP port
For example to capture traffic on interface eth1 and send the flows to the collector at 10.0.0.1
and destination port 9800, following command should be issued:
fprobe -i eth1 10.0.0.1:9800
Since you probably want your fprobe to survive system reboot, you will have to modify
your start-up scripts appropriately. No further configuration should be required.
APPENDIX A. INSTALLATION AND CONFIGURATION 37
A.1.2 Netflow collector
Netflow collector is a device or piece of software, that receives, processes and in most cases
stores Netflow exports generated by the probes. There are many Netflow collectors by many
vendors – for example flow-tools flow-capture, cflowd, Cisco Netflow Collection Engine,...
Since NNIDS currently supports only flow-tools and cflowd format, you will need your
already deployed collector to either store the flow in one of these two formats, convert the
data it into one of these two formats, or forward the flows to cflowd or flow-tools running
collector. Another option is to write your own input module, that recognizes your collector’s
data format. In this document, we will assume you have no collector deployed yet and you
want to use flow-tools toolkit for this.
A.1.2.1 Configuring flow-capture
Again, flow-tools have to be installed on your system and again, most modern operating systems
and Linux distributions have flow-tools in their package repository. In Debian Linux, all you
need to do is:
apt-get install flow-tools
As the subject implies, flow-capture utility is used to collect and store flows. This utility
(like every other from the flow-tools package) takes configuration arguments from the command
line. All the command line options are specified in it’s manual page or in [10], only the most
important options will be discussed above. In Debian, the init.d script has been created, which
can pass command line arguments to flow-capture from configuration file /etc/flow-tools/flow-
capture.conf, so command line options can be specified even in this file, when init.d script is
being used for flow-capture start-up. Simplified configuration syntax of flow-capture might look
like this:
flow-capture [-N nesting level] [-V pdu version ] [-f filter fname] [-F filter definition]
[-E expire size ] [-n rotations ] -w workdir localip/remoteip/port
When flow-capture is started it, will listen on interface with localip on UDP port port and
only packets with source address remoteip will be processed. Zero (“0”) used instead of IP
means “any”, so when you specify 0/0/9800 as a last argument of flow-capture, it will listen
on all interfaces on port 9800/udp and Netflow export from any IP will be accepted. All flows
will be stored in directory workdir or in it’s sub-directory (see nesting level).
Parameter rotations specifies, how many flow files will flow-capture produce per day. When
compression is enabled, every file is written to the disk only when the new one is being created.
Thus if we want to run NNIDS every 10 minutes, we should also run flow-capture with -n 144
to produce one flow file every 10 minutes.
Running flow-capture in a high-traffic network would produce very large amount of data
after a while. If we don’t need to archivate this data, we can set expire size parameter, which
is telling the flow-capture it can use at most expire size bytes of a disk space. When new flow
file would exceed this space, the oldest one is removed. The letters b,K,M,G can be used as
multipliers, i.e. 16 Megabytes is “16M”.
Sometimes the probe is sending us even data flows, which we don’t want to store. To
filter these data, filter expression with name filter definition, defined in file filter fname can be
specified. Filter rules in the file are specified using flow-nfilter rules (see [10]).
38 APPENDIX A. INSTALLATION AND CONFIGURATION
Almost any Netflow version can be captured and stored by flow-capture, pdu version spec-
ifies which version we really want to work with. The last parameter specified is nesting level.
After months of flow-capture use, thousands flow files can be produced. For better orientation
in these files, directories can be created according to date, which the file has been produced.
The following numbers can be used:
-3 YYYY/YYYY-MM/YYYY-MM-DD/flow-file
-2 YYYY-MM/YYYY-MM-DD/flow-file
-1 YYYY-MM-DD/flow-file
0 flow-file
1 YYYY/flow-file
2 YYYY/YYYY-MM/flow-file
3 YYYY/YYYY-MM/YYYY-MM-DD/flow-file
Default nesting level is number 3, it’s configuration does not affect NNIDS’s function, since
NNIDS’s does not care about directory structure. It is intended only for human readability in
this case. The example of flow-capture use is:
flow-capture -V 5 -w /var/flow/flow2 -N -1 0/0/9801
It will listen for Netflow version 5 exports on port 9801/udp on all interfaces and will accept
any source IP. All the flow files will be stored in directory /var/flow/flow2 and nesting level
used will be -1. Flow file will be produced every 15 minutes, which is the default configuration.
A.2 NNIDS installation and configuration
A.2.1 Installation requirements
NNIDS requires operating system with Perl version 5 and its modules Storable, Getopt::Std,
Config::IniFiles. The first two modules are included in Perl version 5.8 and higher and it’s highly
recommended to use this version. This is very modest requirement and should be satisfied on
most modern Unix operating systems. Config::IniFiles module is not so common and your
system might not have it. This module is often distributed as a optional package with most
modern Unixes, so for example in Debian Linux, it can be easily installed:
apt-get install libconfig-inifiles-perl
When your operating system does not have package for this module, it can be easily down-
loaded from the Comprehensive Perl Archive Network (CPAN)1 . You can easily find out, if
your system is capable of running NNIDS by running nnids.pl -V. If you will see something like
this:
confein:~/dp> ./nnids.pl -V
nnids $Revision: 1.10 $, $Date: 2007/05/15 12:34:41 $
... then you are able to run NNIDS with no problems and no additional software have to
be installed. If you can see something like this:
1
˜
See http://search.cpan.org/wadg/Config-IniFiles-2.38/
APPENDIX A. INSTALLATION AND CONFIGURATION 39
cofein:~/nnids> ./nnids.pl -V
./nnids.pl: Command not found.
... then you either don’t have Perl installed at all, or you don’t have it in path /usr/bin/perl,
which is the default path NNIDS will search. Please determine first if you have proper Perl
version installed and where is your Perl binary:
cofein:~> perl -V | head -4
Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
Platform:
osname=netbsd, osvers=3.0.1, archname=i386-netbsd-thread-multi
uname=’netbsd 3.0.1 netbsd 3.0.1 (generic_laptop) #0: thu jul 13 23:46:23
utc 2006 builds@b3.netbsd.org:homebuildsabnetbsd-3-0-1-releasei38620060713182
6z-objhomebuildsabnetbsd-3-0-1-releasesrcsysarchi386compilegeneric_laptop i38
6 ’
cofein:~> which perl
/usr/pkg/bin/perl
If you have Perl installed, but its binary is not in /usr/bin/perl (like in the example above),
adjust the path on the first line of nnids.pl file appropriately. If your output looks like this:
cofein:~/nnids> ./nnids.pl -V
Can’t locate Config/IniFiles.pm in @INC (@INC contains: /etc/perl /usr/local/
lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /u
sr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at ./nnids.pl
line 9.
BEGIN failed--compilation aborted at ./nnids.pl line 9.
... then your Perl is installed and in the correct path, but you don’t have your Config::IniFiles
module installed properly. Please install appropriate package with your favorite packaging
system or download the source from the CPAN and follow the instructions in it.
If you are going to use cflowd input module, you will need to have flow-tools installed. Mail
output module will require properly configured mail system with sendmail-like interface (most
of the MTAs should have it). When you are going to use Nagios output module, you should
also have nrpe, nsca or Nagios itself installed – this depends on the type of configuration you
are going to use. These dependencies are optional and you might not need them. So let’s make
a summary. To run NNIDS you will need:
• Perl version 5.8 (/usr/bin/perl expected, otherwise you have to edit nnids.pl)
• Config::IniFiles
• flow-tools installed (optional – required by Input::cflowd)
• MTA with sendmail-like interface (optional – required by Output::Mail)
• Nagios subsystem (optional – required by Output::Nagios)
40 APPENDIX A. INSTALLATION AND CONFIGURATION
A.2.2 Installation
The NNIDS system consists of the module files, main nnids code, configuration file and a manual
page file. These parts can be used anywhere on the file system when paths are properly set, so
you don’t need to install it in the privileged locations like /usr/local path is. You can freely
use it for example from your home directory instead. It is however wise to install the NNIDS
in locations where utilities are usually expected.
To simplify the installation simple script called install.sh was created. The script takes
installation paths for individual parts of NNIDS from the command line, creates the desired
directories if needed and copies the source files in the destinations:
cofein:~/dp> ./install.sh --help
Illegal option --
Usage: install.sh [-p prefix] [-b binary-path] [-M modules-path]
[-m manual-path] [-c config-path]
Options:
-p prefix set the prefix path to the prefix [default: /usr/local]
-b binary-path set the path, where binaries will be
installed to [default: $prefix/bin]
-M modules-path set the path, where modules will be
installed to [default: $prefix/libexec/nnids]
-m manual-path set the path, where manual page will be
installed to [default: $prefix/man/man1]
-c config-path set the path, where configuration file will be
installed to [default: $prefix/etc]
By default it will install NNIDS in the /usr/local/bin/nnids.pl, modules will be copied to the
/usr/local/libexec/nnids/*, manual page will reside in the /usr/local/man/man1/nnids.1 and
configuration file in the /usr/local/etc/nnids.conf. To install NNIDS in the /usr/bin/nnids.pl,
modules in the /usr/libexec/nnids/*, manual page in the /usr/share/man/man1/nnids.1 and
configuration file in the /etc/nnids.conf following command will be issued:
cofein:~/dp# ./install.sh -p /usr -m /usr/share/man/man1 -c /etc
NNIDS installation
Creating directory /usr/libexec/nnids
Copying modules
Copying manual page
Copying nnids.pl
Copying nnids.conf
A.2.3 NNIDS configuration
I assume you have your NNIDS installed now, as well as the prerequisite components. If you
call nnids -V, it will show you its version with no error messages appended or prepended. Now
it’s time to configure it. Valid configuration file must always be present when running NNIDS.
If it’s not, “ERROR: Config file /etc/nnids.conf missing or empty!” error message will appear.
Config file must always contain at least one definition of input, output and method module,
APPENDIX A. INSTALLATION AND CONFIGURATION 41
otherwise it will complain with “ERROR: No Input modules found, exiting!” or similar error
message.
Configuration file is the classic .INI-style file, commonly found in the Microsoft Windows
world. INI files are divided into sections, each preceded with the section name in square
brackets. The first non-blank character of the line indicating a section must be a left bracket
and the last non-blank character of a line indicating a section must be a right bracket. The
characters making up the section name can be any symbols at all, however in NNIDS there are
only alphanumeric characters are used.
There are 4 types of sections used in the NNIDS configuration: Global, Input, Output
and Method. Global section is defined only once and it contains configuration related to
the NNIDS core or other configuration options, common for all modules. Input, Output and
Method sections are always followed by a module name. These sections say, that Input, Output,
or Method module of the specified name should be loaded and used.
Parameters for the particular parts are specified in each section as “Name = Value” pairs.
Parameter names must be unique within a section. Both the hash mark (#) and the semicolon
(;) are comment characters. Lines that begin with either of these characters will be ignored.
Any amount of white-space may precede the comment character. Multi-line or multi-valued
parameters may also be defined ala UNIX ”here document” syntax, continuation lines are
allowed:
[Global]
# This is a comment
warn badness = 5
; and this is also a comment
critical badness = 10
[Input Flowtools]
data path = <<EOF
/var/flow/flow1
/var/flow/flow2
/var/flow/flow3
EOF
[Method PatternIP]
hosts = 192.168.9.4 \
192.168.9.1 \
192.168.4.0/24
badness = 10
A.2.3.1 Global section
Global section is not a mandatory part of the configuration, but you will very likely want to
use it. Parameters specified here are affecting overall behavior of NNIDS and will probably
affect all the modules used. Main logic has been already described in 3.2.1 on page 17 and we
assume you are already familiar with it.
In [Global], modules path parameter can be specified to provide path to all modules (top
directory, where Input, Output and Method sub-directories with modules should exist). By de-
fault, there is no additional path set and only paths in standard Perl’s @INC variable are used.
42 APPENDIX A. INSTALLATION AND CONFIGURATION
Threshold badness for the warning and critical severity can be specified by warn badness and
critical badness parameters. IP addresses with badness below warn badness won’t be reported,
unless retention is set to 1 and IP is recovering from the previous non-OK state. retention
parameter enables (1) or disables (0) the retention mechanism, the retention parameter then
specifies the maximum number of message retentions. Last parameter, which might be config-
ured in the [Global] section is state file, which specifies path to the file, where state information
should be kept. Example of [Global] might look like this:
[Global]
warn badness = 5
critical badness = 10
state file = /home/nnids/nnids.state
time block = 15
block count = 8
retention = 1
max retention = 8
modules path = /home/nnids
In this configuration, only IP’s with badness ≥ 5 will be reported – badness below 10 will
result in warning severity, greater will result in critical severity. Data will be divided into time
blocks, 15 minutes each. NNIDS will work with only 8 most recent time blocks, older will
be always deleted. State information (precalculated time blocks from previous runs, positions
in input files, etc.) will be kept in /home/nnids/nnids.state. Modules will be searched in
/home/nnids and standard Perl module paths.
A.2.3.2 Flowtools input module
At least one input module must be defined in the configuration file. Currently, there are only
two input modules implemented. Flowtools is recommended input module, because it’s faster
than cflowd input module, flowtools’ compressed flow files require less disk space, and cflowd
software support has been dropped by CAIDA. By adding section “[Input Flowtools]”, you
are telling NNIDS to load and use input module called Flowtools. This module has only two
configuration parameters. data path specified directory, where the flow files should be searched.
You can also specify several directories, separated by a space character.
file prefix is a string, which should every valid flow file file-name start with. When flow-
capture is receiving data for new flow file, it creates an incomplete flow file named like “tmp-
v05.2007-04-28.210826+0200”. This file might contain only header with no flow data (when
compression is enabled and some flows has been already received), or it might be empty (when
no data has been received yet). When file rotation should be done, flow-capture flushes data
into file and renames it to something like “ft-v05.2007-04-28.210826+0200”. New file with
name starting with tmp-* is created and the whole process is repeated again. Reading empty
or incomplete flow file might confuse the input module, thus only complete files, which name
starts with file prefix are inspected. The example configuration might look like this:
[Input Flowtools]
data path = /var/flow/flow1 /var/flow/flow2
file prefix = ft
In this case, flow files will be searched in directories /var/flow/flow1, /var/flow/flow2, and
their sub-directories. Only files with file-name starting with “ft” will be used.
APPENDIX A. INSTALLATION AND CONFIGURATION 43
A.2.3.3 cflowd input module
As was explained in the previous section, use of cflowd input module is not recommended. If
you don’t want to use it, simply don’t specify its name in configuration file and it won’t be
loaded. However situations where this module is needed might arise. Then you have to create
section “[Input cflowd]” in your configuration file.
This module needs flow-tools package installed in your system, to read cflowd files. In the
fact, these files are being convert on the fly by utilities flow-export and flow-import. Data are
read as these utilities’ output. This module accepts parameters flow-export command and flow-
import command, which can set path to appropriate flow-tools utilities. As in the Flowtools
input module, data path parameter specify directories, which are being searched for flow files.
Simple configuration follows:
[Input cflowd]
data path = /var/flow/flow1 /var/flow/flow2
flow-export command = /usr/pkg/bin/flow-export
flow-import command = /usr/pkg/bin/flow-import
A.2.3.4 TopN method module
This module measures amount of data, which every detected source IP address generates. For
each source IP it measures number of bytes, packets and flows transferred. Its configuration
options and logic has been already discussed in chapter 3.2.5.1 on page 25. This module should
be suitable for almost every network and it’s use is highly recommended. Example configuration
might be following:
[Method TopN]
max octets = 8000000000
max sessions = 80000
max packets = 8000000
max badness = 10
min octets = 1000000000
min sessions = 50000
min packets = 5000000
min badness = 1
ignore hosts = 192.168.9.4/32 192.168.4.0/24 192.168.9.1
In this case, badness of 1 will be assigned, when source IP transfers in all time blocks more
than 1 000 000 000 bytes altogether, or creates more than 50 000 flows altogether, or generates
more than 5 000 000 packets. More bytes, packets, or flows it generates, higher badness will be
assigned. Three badness numbers are counted – byte, packet and flow badness. The highest
badness wins. Maximum badness assigned is 10, at 8 000 000 000 bytes transferred, or 80 000
flows generated, or 8 000 000 packets transferred. When the source IP transfers more data,
badness will not raise any more. It will still be 10. In the ignore hosts parameter are specified
network or hosts, for which TopN will not count any results. These are usually hosts, that
legally produce higher amount of traffic, than other hosts.
44 APPENDIX A. INSTALLATION AND CONFIGURATION
A.2.3.5 Scan method module
Scan module detects host scans and port scans, originated in every detected source IP. Module
tracks destination IP addresses, which is the source IP connecting to and also which port
numbers is it connecting to. Since counting of every destination port number would result in
extremely large hash tables, only ports, which port number is lower than max port parameter
are counted. By default max port is 1024, so only privileged port numbers will be counted.
When the source IP is connecting to more than max hosts, module will consider it as a host
scan. When the source IP is connecting to more than max ports, it will consider it as a port
scan. Certain hosts like name server or proxy servers might legally connect to large number
of hosts. These servers should be specified in the ignore hosts parameter and thus ignore in
Scan’s statistics. Example configuration follows:
[Method Scan]
max ports = 64
max hosts = 64
max port = 1024
ignore hosts = 192.168.9.4/32 192.168.4.0/24 192.168.9.1
A.2.3.6 PatternIP method module
This module scans every detected source IP address for occurrence in a list of forbidden IP
addresses, specified in parameter hosts. If a match is found, badness is assigned to this IP.
Assignment is not cumulative, i.e. badness is assigned only once, it doesn’t matter how many
times was the give source IP detected. Example configuration:
[Method PatternIP]
hosts = 192.168.9.4 \
192.168.9.1 \
192.168.4.0/24
badness = 10
A.2.3.7 PatternPort method module
PatternPort is similar to the PatternIP. It scans destination port number for a occurrence
in a list of UDP or TCP port numbers. If a match is found, badness is assigned to this
IP. Again badness assignment is not cumulative. List of forbidden UDP or TCP ports is
specified in tcp ports or udp ports parameter respectively. Port numbers can be also specified
as port ranges starting port-ending port – in this case any port number that satisfy condition
(starting port ≤ port number ≤ ending port), will match and badness will be assigned. Again
ignore hosts parameter can be specified to exclude certain IP addresses from the statistics.
Example configuration:
[Method PatternPort]
tcp ports = 6881-6889 1433 31337
udp ports = 69
badness = 10
ignore hosts = 192.168.9.4/32 192.168.4.0/24 192.168.9.1
APPENDIX A. INSTALLATION AND CONFIGURATION 45
In the example, TCP ports 1433, 31337, and port range 6881–6889 will be forbidden and any
occurrence of these numbers in the destination port field of flow record will result in assignment
of badness 10 for the particular source IP. In UDP, only port number 69 is forbidden. IP’s
from network 192.168.4.0/24, and IP’s 192.168.9.1 and 192.168.9.4, won’t be included in the
statistics and thus can freely contact any port number without any badness assignment by the
PatternPort module.
A.2.3.8 DoS method module
DoS method module is basically a TopN module, that measures amount of data per destination
IP address. It has the same configuration parameters, the same badness counting algorithm
and nearly the same code, only the direction is different. Simply said – while TopN checks if
the IP is doing anything bad to someone else, DoS checks if someone else is doing anything bad
to the IP. Example configuration:
[Method DoS]
max octets = 8000000000
max sessions = 80000
max packets = 8000000
max badness = 10
min octets = 1000000000
min sessions = 50000
min packets = 5000000
min badness = 1
ignore hosts = 192.168.9.4/32 192.168.4.0/24 192.168.9.1
In this case, badness of 1 will be assigned when traffic destinated to the IP in all time
blocks is greater than 1 000 000 000 bytes, 50 000 flows, or 5 000 000 packets altogether. More
bytes, packets, or flows is destinated, higher badness will be assigned. Maximum badness is 10
with 8 000 000 000 bytes or greater, 80 000 flows or greater, or 8 000 000 packets or greater
destinated to the IP. Once again – in the ignore hosts parameter are specified network or hosts,
for which DoS will not count any results.
A.2.3.9 Stderr output module
Output modules are responsible for alerting the administrator, when anything bad occurs, i.e.
when severity code of any IP changes. Stderr is the simplest module. It has no configuration
parameters and its only role is to print out the results to NNIDS’s standard error output. It
might become handy, when running NNIDS from the command line, or for debugging purposes.
To load the module, following line has to specified in the config file:
[Output Stderr]
A.2.3.10 Syslog output module
This module utilizes syslog to inform the administrator about security incidents. Its parameter
type specifies, which type of transport should be used for syslog messages. Allowed values are
46 APPENDIX A. INSTALLATION AND CONFIGURATION
socket for using local Unix socket (usually /dev/log), udp for logging to the remote host using
UDP transport, or experimental tcp for logging to the remote host by TCP transport. When
remote logging is to be used, logging host parameter with destination IP address has to be
specified. Example configuration:
[Output Syslog]
type = udp
logging host = 10.0.0.1
In this case, syslog server at 10.0.0.1 and UDP transport will be used. Messages are with
LOG USER syslog facility and the daemon name is “nnids”. These parameters can be used
for example for saving NNIDS generated messages into separate file. For example following
configuration lines can be used in the syslog-ng config file to accomplish this:
filter f_nnids { program("nnids"); };
destination d_nnids { file("/var/log/nnids.log"); };
log { source(s_all); filter(f_nnids); destination(d_nnids); };
In the example above, syslog-ng will store all messages generated by the NNIDS in file
/var/log/nnids.log. You should modify the log line appropriately, if your source is not named
s all.
A.2.3.11 Mail output module
To inform the administrator about detected incidents via e-mail, Mail output module can be
used. This module uses interface of sendmail binary for mail sending. Besides the original
Sendmail, most of the modern Mail Transfer Agents have this interfaces, since it is de facto
standard in the Unix world. Example configuration, that uses all the configuration parameters:
[Output Mail]
sendmail command = /usr/pkg/bin/msmtp
recipients = root@nowhere.net
subject = NNIDS events
from = root@nowhere.net
Module builds a mail body and headers with subject subject, sender’s address from and
recipients address recipients. This body will be put on standard input of sendmail command. In
this case /usr/pkg/bin/msmtp -f root@nowhere.net root@nowhere.net will be executed.
A.2.3.12 Nagios output module
When Mail module is not enough and you would like to contact groups of administrators,
depending on the daytime, do incident escalations, analyze incident trends, and do plenty of
other stuff, you will probably need to use monitoring system to accomplish this. In my humble
opinion one of the best open-source monitoring systems these days is Nagios.
Nagios system and its logic was described in chapter 3.2.4.4 on page 23, or in document [12].
There are two basic types of checks in Nagios – active and passive. You should tell the output
module which type of check do you want it to serve in parameter check type.
APPENDIX A. INSTALLATION AND CONFIGURATION 47
If the type is active, you should also specify the output file parameter, which tells the
module, where it should store its state to. The example configuration is the following:
[Output Nagios]
check type = active
output file = /var/tmp/nnids.nagios
Nagios output module will then update /var/tmp/nnids.nagios each run. In the nrpd
configuration, you should call the check nnids.pl script, which will check the state file and
return the state back to the Nagios system. The check nnids.pl script should be edited to read
the state from the proper location. Configuration line of the nrpd daemon (running on the
same machine as NNIDS does) can be the following:
command[check_nnids]=/usr/lib/nagios/plugins/check_nnids.pl
The Nagios service definition on the Nagios server is then simple active check. For example:
define service {
use generic-service
host_name nnids-host
contact_groups nnids-admins
service_description NNIDS service state
check_command check_nrpe!check_nnids
normal_check_interval 10
retry_check_interval 10
}
You will probably want to set check intervals to be the same as intervals in which NNIDS
is being run. Shorter intervals will be wasting of the CPU time, longer intervals will result in
some result’s loss.
If the check type is set to passive, hostname and svc description parameters should be set.
These values are used by Nagios to assign result of the passive check to the proper host and
service. These values must correspond with the host name and the service description field of
the service definition in Nagios configuration. Also submit command parameter must be set.
This parameter must contain a command, which will take the result on it’s standard input and
send it to the Nagios. This is usually done by the send nsca utility of nsca package. NNIDS’s
configuration can look for example like this:
[Output Nagios]
check type = passive
submit command = /usr/bin/send_nsca -H 192.168.250.30 -c /etc/nagios/nsca.cfg
hostname = gw
svc description = NNIDS
Nagios output module will then let send nsca connect Nagios server at 192.168.250.30 (which
must be running nsca daemon), at send the result. Example Nagios server configuration for
this case follows:
48 APPENDIX A. INSTALLATION AND CONFIGURATION
define service {
use generic-service
check_command check-host-alive
host_name gw
service_description NNIDS
is_volatile 1
max_check_attempts 1
normal_check_interval 1
retry_check_interval 1
notification_interval 31536000
passive_checks_enabled 1
check_period none
contact_groups router-admins
notification_period 24x7
}
Service definition above is standard passive check. Check intervals had been set to 1,
but they are going to never happen, since check period is set to none. host name and the
service description correspond with the definition used in NNIDS configuration.
APPENDIX B. MANUAL PAGE 49
B Manual page
NNIDS (1) User’s Reference Manual NNIDS (1)
NAME
nnids — Open source Netflow based network intrusion detection system
SYNOPSIS
nnids.pl [ -d] [ -s] [ -V] [ -c config-file]
DESCRIPTION
nnids is a modular intrusion detection system, which takes as an input data flows, recorded and col-
lected using Cisco’s Netflow technology.
When the program is started, it scans for configuration file. Since the configuration file contains
names of all used modules it must always be present and must always contain at least one input, one
output and one method module definition. If anything above is not present, program dies with an
error message.
When the configuration file is parsed, every module is loaded. If loading of any module fails, pro-
gram dies with an error message.
The main cycle buffers flow records received from the input modules and distributes them among all
method modules. When the end of the time block has been detected, all method modules are
informed. Every method module is then probed for it’s results and the resulting data structures are
passed to the enumerator. Enumerator groups the results from all method modules by their IP
addresses. For every IP address, sum of all penalties is counted and according to the configured
thresholds, severity levels are assigned. The result is a set of output records, which is passed to every
output module. This cycle is repeated, until all flow records from all input module are processed.
When the last record has been processed, state information from all input and method modules is
gathered and saved into the state file, which will be loaded on the next run, so calculations can con-
tinue. After that, program exits.
OPTIONS
-d show debug messages on standard error output
-s don’t save state information
-V show program version and exit
-c config-file
use config-file file instead of default /etc/nnids.conf
EXAMPLES
The following examples are shown as given to the shell:
nnids -d -c ./nnids.conf
Run nnids with debug messages enabled. Use configuration file ./nnids.conf.
nnids -d -s
Run nnids with debug messages enabled. Don’t save state information. Use default configura-
tion file.
CONFIGURATION FILE
Program is controlled by a .ini style configuration file, commonly used in a wide variety of software.
Configuration is divided into sections, each section starts with declaration [Name of section]. Cur-
rently there are defined 4 types of sections.
Section Global controls main NNIDS functions (enumerator, state information, retention policies,
etc.). Section Input (followed by the module name) contains configuration parameters for particular
input module and also instructs the NNIDS to load this module automatically. Sections Output and
NetBSD 4.0 May 16 2007 1
50 APPENDIX B. MANUAL PAGE
NNIDS (1) User’s Reference Manual NNIDS (1)
Method are similar and contains configuration for output or method modules respectively.
[Global]
Not a mandatory section. However parameters specified here are affecting overall behaviour of
NNIDS and will probably affect all the modules used.
warn badness
IP addresses with badnes greater or equal to warn badness will be reported with severity
WARNING.
Example warn badness = 50
Default warn badness = 5
critical badness
IP addresses with badnes greater or equal to critical badness will be reported with sever-
ity CRITICAL.
Example critical badness = 100
Default critical badness = 10
state file
Path to the file, where state information can be stored for the next run.
Example state file = /var/run/nnids/nnids.state
Default state file = /var/tmp/nnids.state
time block
Length of time block in minutes. Time blocks are small units of constant length (15 minutes
by default), where the flows are parceled into.
Example time block = 2
Default time block = 15
block count
Number of time blocks which will be used for counting. Oldest time block is deleted, when
newly created time block would exceed the block count
Example block count = 3
Default block count = 8
retention
Enables (1) or disables (0) the retention mechanism. When retention is enabled, only the first
occurrence of security incident of the same severity is reported per one IP address. When the
same severity should be reported for the same IP in the next block (or run), no output mes-
sage is generated.
Example retention = 0
Default retention = 1
max retention
To remind the administrator after a while, maximum number of message retentions can be
specified (8 by default). When more than specified number of messages has been suppressed,
message is generated again.
NetBSD 4.0 May 16 2007 2
APPENDIX B. MANUAL PAGE 51
NNIDS (1) User’s Reference Manual NNIDS (1)
Example max retention = 10
Default max retention = 8
modules path
Provides a path to the modules top directory. In this directory Input, Output and Method
subdirectories must exist, where modules are stored.
Example modules path = /home/nnids
Default No default value
[Input Flowtools]
This input module reads flow records in a format, used by all utilities from the flow-tools(1) pack-
age. NNIDS expects files created by the flow-capture, thus only Netflow version 5 format support is
implemented, streaming files are not supported. Version 3 index is required. Both compressed and
uncompressed files are supported.
data path
Directory, where the flow files should be searched. Several directories separated by a space
character can be specified.
Example data path = /var/flows/flow1 /var/flows/flow2
Default No default value
file prefix
String, which should every valid flow file filename start with.
Example file prefix = flow
Default file prefix = ft
[Input cflowd]
cflowd(1) is an old, but still popular flow analysis tool used for analyzing Cisco’s NetFlow enabled
switching method. NNIDS is able to read the cflowd file format using supporting utilities
flow-export(1) and flow-import(1) . cflowd input module is much slower in reading records, than
Flowtools module, which is the only recomended.
data path
Directory, where the flow files should be searched. Several directories separated by a space
character can be specified.
Example data path = /var/flows/flow1 /var/flows/flow2
Default No default value
flow-export command
Path to the flow-export(1) utility.
Example flow-export command = /usr/pkg/bin/flow-export
Default flow-export command = flow-export
flow-import command
Path to the flow-import(1) utility.
Example flow-import command = /usr/pkg/bin/flow-import
Default flow-import command = flow-import
NetBSD 4.0 May 16 2007 3
52 APPENDIX B. MANUAL PAGE
NNIDS (1) User’s Reference Manual NNIDS (1)
[Method TopN]
TopN module measures amount of data, which every detected source IP address generates. For each
source IP it measures number of bytes, packets and flows transferred.
min octets
Minimum number of bytes, that will result in min badness assignment
Example min octets = 1000000000
Default min octets = 1000000
min sessions
Minimum number of sessions, that will result in min badness assignment
Example min sessions = 50000
Default min sessions = 1000
min packets
Minimum number of packets, that will result in min badness assignment
Example min packets = 5000000
Default min packets = 10000
max octets
Maximum number of bytes, that will result in max badness assignment
Example max octets = 8000000000
Default max octets = 10000000
max sessions
Maximum number of sessions, that will result in max badness assignment
Example max sessions = 80000
Default max sessions = 10000
max packets
Maximum number of packets, that will result in max badness assignment
Example max packets = 8000000
Default max packets = 100000
min badness
Badness assigned at minimum tresholds.
Example min badness = 10
Default min badness = 1
max badness
Badness assigned at maximum tresholds.
Example max badness = 20
Default max badness = 10
ignore hosts
Hosts or networks, that won’t be included in the statistics.
NetBSD 4.0 May 16 2007 4
APPENDIX B. MANUAL PAGE 53
NNIDS (1) User’s Reference Manual NNIDS (1)
Example ignore hosts = 192.168.1.0/24 192.168.2.2
Default No default value
[Method Scan]
Scan module detects host scans and port scans, originated in every detected source IP. Module
tracks destination IP addresses, which is the source IP connecting to and also which port numbers is
it connecting to. Only ports, which port number is lower than max port parameter are counted.
max ports
Maximum number of legally contacted ports. Higher number is considered a port scan.
Example max ports = 512
Default max ports = 64
max hosts
Maximum number of legally contacted hosts. Higher number is considered a host scan.
Example max hosts = 128
Default max hosts = 64
max port
Count only lowest max port port numbers in the port scan detection.
Example max port = 65535
Default max hosts = 1024
ignore hosts
Hosts or networks, that won’t be included in the statistics.
Example ignore hosts = 192.168.1.0/24 192.168.2.2
Default No default value
[Method PatternIP]
Module scans every detected source IP address for occurence in a list of forbidden IP addresses,
specified in parameter hosts . When match is found, badness is assigned.
hosts
List of forbidden IP addresses.
Example hosts = 192.168.9.4 192.168.4.0/24
Default No default value
badness
Badness assigned to the IP, when it matches the list.
Example badness = 50
Default badness = 10
[Method PatternPort]
Module scans destination port number for a occurence in a list of UDP or TCP port numbers. If a
match is found, badness is assigned to this IP.
tcp ports
List of forbidden port ranges. Individual port numbers or port range is allowed. Port numbers
and port ranges are separated by a space character.
NetBSD 4.0 May 16 2007 5
54 APPENDIX B. MANUAL PAGE
NNIDS (1) User’s Reference Manual NNIDS (1)
Example tcp ports = 6881-6889 1433 31337
Default No default value
udp ports
List of forbidden port ranges. Individual port numbers or port range is allowed. Port numbers
and port ranges are separated by a space character.
Example udp ports = 69
Default No default value
badness
Badness assigned to the IP, when is found.
Example badness = 50
Default badness = 10
ignore hosts
Hosts or networks, that won’t be included in the statistics.
Example ignore hosts = 192.168.1.0/24 192.168.2.2
Default No default value
[Method DoS]
DoS method module is basically a TopN module, that measures amount of data per destination IP
address. It has the same configuration paramaters, the same badness counting algorithm and nearly
the same code, only the direction is different. Simply said - while TopN checks if the IP is doing any-
thing bad to someone else, DoS checks if someone else is doing anything bad to the IP.
min octets
Minimum number of bytes, that will result in min badness assignment
Example min octets = 1000000000
Default min octets = 1000000
min sessions
Minimum number of sessions, that will result in min badness assignment
Example min sessions = 50000
Default min sessions = 1000
min packets
Minimum number of packets, that will result in min badness assignment
Example min packets = 5000000
Default min packets = 10000
max octets
Maximum number of bytes, that will result in max badness assignment
Example max octets = 8000000000
Default max octets = 10000000
max sessions
Maximum number of sessions, that will result in max badness assignment
NetBSD 4.0 May 16 2007 6
APPENDIX B. MANUAL PAGE 55
NNIDS (1) User’s Reference Manual NNIDS (1)
Example max sessions = 80000
Default max sessions = 10000
max packets
Maximum number of packets, that will result in max badness assignment
Example max packets = 8000000
Default max packets = 100000
min badness
Badness assigned at minimum tresholds.
Example min badness = 10
Default min badness = 1
max badness
Badness assigned at maximum tresholds.
Example max badness = 20
Default max badness = 10
ignore hosts
Hosts or networks, that won’t be included in the statistics.
Example ignore hosts = 192.168.1.0/24 192.168.2.2
Default No default value
[Output Stderr]
Output module, that prints out result to the standard error output. It has no configuration parame-
ters.
[Output Syslog]
This module uses the standard BSD syslog protocol to send messages to local or remote host. Mes-
sages can be logged using 3 methods: local socket, udp network transport and tcp network trans-
port.
type
Type of transport used for syslog messages. Allowed values are socket (local Unix socket),
udp (logging to the remote host using UDP transport), or tcp (logging to the remote host
using experimental TCP transport).
Example type = udp
Default type = socket
logging host
IP address of remote host for udp or tcp transport.
Example logging host = 10.0.0.1
Default No default value
[Output Mail]
This module uses interface of the sendmail(1) binary to inform the administrator about detected
incidents via e-mail.
NetBSD 4.0 May 16 2007 7
56 APPENDIX B. MANUAL PAGE
NNIDS (1) User’s Reference Manual NNIDS (1)
sendmail command
Path to the sendmail(1) binary or compatible MTA interface.
Example sendmail command = /usr/pkg/bin/msmtp
Default sendmail command = /usr/lib/sendmail
recipients
Recipients of the resulting mail.
Example recipients = root@nowhere.net sg
Default No default value
from
Sender address of the resulting mail.
Example from = root@nowhere.net
Default No default value
subject
Subject of the resulting mail.
Example subject = New detected incidents
Default subject = NNIDS output
[Output Nagios]
NNIDS interface to the nagios(1) monitoring system. There are two basic modes of Nagios checks -
active and passive. If the type is active output file must be specified, where the state will be
stored and picked up by the check nnids plugin. If the type is passive hostname and svc
description must be specified to match service identification in Nagios, and submit command
parameter must be set to call send nsca properly.
check type
Type of Nagios check. Allowed values are active or passive .
Example check type = passive
Default check type = active
output file
Output file, where state will be stored. Used for check nnids plugin. Active checks only.
Example output file = /var/run/check_nnids.state
Default No default value
submit command
send nsca command for passive checks.
Example submit command = /usr/bin/send_nsca -H 192.168.250.30 -c
/etc/nagios/nsca.cfg
Default submit command = /usr/bin/send_nsca
hostname
hostname attribute matching with the definition in Nagios configuration. Used in passive
checks only.
NetBSD 4.0 May 16 2007 8
APPENDIX B. MANUAL PAGE 57
NNIDS (1) User’s Reference Manual NNIDS (1)
Example hostname = nnids-host
Default No default value
svc description
service description attribute matching with the definition in Nagios configuration. Used in
passive checks only.
Example svc description = NNIDS
Default No default value
CONFIGURATION EXAMPLE
An example of configuration file:
[Global]
state file = /home/andrew/dp/nnids.state
critical badness = 10
warn badness = 5
[Input Flowtools]
data path = /home/andrew/dp/flow2
[Output Stderr]
[Method TopN]
max octets = 10000001
[Method Scan]
max ports = 64
FILES
/etc/nnids.conf
Main configuration file
/var/tmp/nnids.state
File with state information for the next run
SEE ALSO
flow-tools(1) flow-import(1) flow-export(1) flow-capture(1) nagios(1) perl(1) sendmail(1)
HISTORY
The nnids (1) manual page appeared in nnids revision 1.10.
NetBSD 4.0 May 16 2007 9
58 APPENDIX C. CONTENTS OF ENCLOSED CD-ROM
C Contents of enclosed CD-ROM
/: Stderr.pm,v
Syslog.pm,v
CVSROOT/
dipl.pdf /CVSROOT/nnids/contrib:
nnids/
check nnids.pl,v
text/
/nnids:
/CVSROOT:
Input/
CVSROOT/
Method/
nnids/ Output/
contrib/
/CVSROOT/CVSROOT:
install.sh*
/CVSROOT/nnids:
nnids.1
Input/ nnids.conf
Method/ nnids.pl*
Output/
/nnids/Input:
contrib/
install.sh,v* Flowtools.pm
nnids.1,v cflowd.pm
nnids.conf,v
/nnids/Method:
nnids.pl,v*
DoS.pm
/CVSROOT/nnids/Input:
PatternIP.pm
Flowtools.pm,v PatternPort.pm
cflowd.pm,v Scan.pm
TopN.pm
/CVSROOT/nnids/Method:
/nnids/Output:
DoS.pm,v
Mail.pm
PatternIP.pm,v
Stderr.pm
PatternPort.pm,v
Syslog.pm
Scan.pm,v
TopN.pm,v /nnids/contrib:
/CVSROOT/nnids/Output: check nnids.pl
Mail.pm,v /text:
APPENDIX C. CONTENTS OF ENCLOSED CD-ROM 59
chap1.tex /text/img:
chap2.tex
LogoCVUT.pdf
chap3.tex
LogoK336.pdf
chap4.tex
man.pdf
chap5.tex
nagios-screenshot.pdf
chapA1.tex
nagios-screenshot.png
chapA2.tex
netflow-example.pdf
chapA3.tex
netflow-example.svg
dipl.tex
netflow9.jpg
graphs/
nnids-arch.pdf
img/
nnids-arch.svg
macro.tex
nnids-deployment.pdf
mr.cls
nnids-deployment.svg
/text/graphs: tcp.pdf
tcp.svg
topn.plot topn.pdf
