Docstoc

PowerPoint XP - FSU Computer Science

Document Sample
PowerPoint XP - FSU Computer Science Powered By Docstoc
					Automated Worm
Fingerprinting

     Sumeet Singh, Cristian Estan,
     George Varghese, and Stefan
     Savage
Introduction

   Problem: how to react quickly to
    worms?
   CodeRed 2001
       Infected ~360,000 hosts within 11 hours
   Sapphire/Slammer (376 bytes) 2002
       Infected ~75,000 hosts within 10 minutes
Existing Approaches

   Detection
       Ad hoc intrusion detection
   Characterization
       Manual signature extraction
           Isolates and decompiles a new worm
           Look for and test unique signatures
           Can take hours or days
Existing Approaches

   Containment
       Updates to anti-virus and network filtering
        products
Earlybird

   Automatically detect and contain new
    worms
   Two observations
       Some portion of the content in existing
        worms is invariant
       Rare to see the same string recurring
        from many sources to many destinations
Earlybird

   Automatically extract the signature of
    all known worms
       Also Blaster, MyDoom, and Kibuv.B
        hours or days before any public
        signatures were distributed
   Few false positives
Background and Related Work

   Almost all IPs were scanned by
    Slammer < 10 minutes
       Limited only by bandwidth constraints
The SQL Slammer Worm:
30 Minutes After ―Release‖




 - Infections doubled every 8.5 seconds
 - Spread 100X faster than Code Red
 - At peak, scanned 55 million hosts per second.
Network Effects Of The SQL
Slammer Worm
   At the height of infections
       Several ISPs noted significant bandwidth
        consumption at peering points
       Average packet loss approached 20%
       South Korea lost almost all Internet
        service for period of time
       Financial ATMs were affected
       Some airline ticketing systems
        overwhelmed
Signature-Based Methods

   Pretty effective if signatures can be
    generated quickly
       For CodeRed, 60 minutes
       For Slammer, 1 – 5 minutes
Worm Detection

   Three classes of methods
       Scan detection
       Honeypots
       Behavioral techniques
Scan Detection

   Look for unusual frequency and
    distribution of address scanning
   Limitations
       Not suited to worms that spread in a non-
        random fashion (i.e. emails)
       Detects infected sites
       Does not produce a signature
Honeypots

   Monitored idle hosts with untreated
    vulnerabilities
       Used to isolate worms
   Limitations
       Manual extraction of
        signatures
       Depend on quick infections
Behavioral Detection

   Looks for unusual system call patterns
       Sending a packet from the same buffer
        containing a received packet
       Can detect slow moving worms
   Limitations
       Needs application-specific knowledge
       Cannot infer a large-scale outbreak
Characterization

   Process of analyzing and identifying a
    new worm
   Current approaches
       Use a priori vulnerability signatures
       Automated signature extraction
Vulnerability Signatures

   Example
       Slammer Worm
           UDP traffic on port 1434 that is longer than
            100 bytes (buffer overflow)
   Can be deployed before the outbreak
       Can only be applied to well-known
        vulnerabilities
Some Automated Signature
Extraction Techniques

   Allows viruses to infect decoy progs
       Extracts the modified regions of the
        decoy
       Uses heuristics to identify invariant code
        strings across infected instances
   Limitation
       Assumes the presence of a virus in a
        controlled environment
Some Automated Signature
Extraction Techniques

   Honeycomb
       Finds longest common subsequences
        among sets of strings found in messages
   Autograph
       Uses network-level data to infer worm
        signatures
   Limitations
       Scale and full distributed deployments
Containment

   Mechanism used to deter the spread
    of an active worm
       Host quarantine
           Via IP ACLs on routers or firewalls
       String-matching
       Connection throttling
           On all outgoing connections
Defining Worm Behavior

   Content invariance
       Portions of a worm are invariant (e.g. the
        decryption routine)
   Content prevalence
       Appears frequently on the network
   Address dispersion
       Distribution of destination addresses
        more uniform to spread fast
Finding Worm Signatures

   Traffic pattern is sufficient for detecting
    worms
       Relatively straightforward
       Extract all possible substrings
       Raise an alarm when
           FrequencyCounter[substring] > threshold1
           SourceCounter[substring] > threshold2
           DestCounter[substring] > threshold3
Detecting Common Strings

   Cannot afford to detect all substrings
   Maybe can afford to detect all strings
    with a small fixed length
Detecting Common Strings

    Cannot afford to detect all substrings
    Maybe can afford to detect all strings
     with a small fixed length

A horse is a horse, of course, of course

    F1 = (c1p4 + c2p3 + c3p2 + c4p1 + c5) mod M
Detecting Common Strings

    Cannot afford to detect all substrings
    Maybe can afford to detect all strings
     with a small fixed length
      F2 = (c2p4 + c3p3 + c4p2 + c5p1 + c6) mod M

A horse is a horse, of course, of course

    F1 = (c1p4 + c2p3 + c3p2 + c4p1 + c5) mod M
Detecting Common Strings

   Cannot afford to detect all substrings
   Maybe can afford to detect all strings
    with a small fixed length
    F2 = (c2p4 + c3p3 + c4p2 + c5p1 + c6) mod M
       = (c1p5 + c2p4 + c3p3 + c4p2 + c5p1 + c6 - c1p5) mod M

       = (pF1 + c6 - c1p5) mod M
Detecting Common Strings

   Cannot afford to detect all substrings
   Maybe can afford to detect all strings
    with a small fixed length
       Still too expensive…
Too CPU-Intensive

   A packet with 1,000 bytes of payload
       Needs 960 fingerprints for string length of
        40
   Prone to Denial-of-Service attacks
CPU Scaling

   Random sampling may miss many
    substrings
   Solution: value sampling
       Track only certain substrings
           e.g. last 6 bits of fingerprint are 0
       P(not tracking a worm)
        = P(not tracking any of its substrings)
CPU Scaling

   Example
       Track only substrings with last 6 bits = 0s
       String length = 40
       1,000 char string
           960 substrings  960 fingerprints
           ‗11100…101010‘…‗10110…000000‘…
       Use only ‗xxxxx….000000‘ as signatures
           Probably 960 / 26 = 15 signatures
CPU Scaling

   P(finding a 100-byte signature) = 55%
   P(finding a 200-byte signature) = 92%
   P(finding a 400-byte signature) =
    99.64%
Estimating Content
Prevalence

   Finding the packet payloads that
    appear at least x times among the N
    packets sent
       During a given interval
Estimating Content
Prevalence

   Table[payload]
       1 GB table filled in 10 seconds
   Table[hash[payload]]
       1 GB table filled in 4 minutes
       Tracking millions of ants to track a few
        elephants
       Collisions...false positives
    Multistage Filters

Array of                                 stream memory
counters
             Hash(Pink)




                   [Singh et al. 2002]
    Multistage Filters

Array of                   packet memory
counters
             Hash(Green)
    Multistage Filters

Array of                   packet memory
counters
             Hash(Green)
Multistage Filters

                     packet memory
     Multistage Filters

Collisions                packet memory
 are OK
    Multistage Filters
Reached                    packet memory
threshold
                            packet1 1


                  Insert
Multistage Filters

                     packet memory

                      packet1 1
Multistage Filters

                     packet memory

                      packet1 1

                      packet2 1
   Multistage Filters

Stage 1                 packet memory

                           packet1 1




                        No false negatives!
                        (guaranteed detection)

Stage 2
     Conservative Updates

Gray = all prior packets
Conservative Updates
                       Redundant




              Redundant
Conservative Updates
Estimating Address Dispersion

   Not sufficient to count the number of
    source and destination pairs
       e.g. send a mail to a mailing list
           Two sources—mail server and the sender
           Many destinations
   Need to count the unique source and
    destination traffic flows
       For each substring
Bitmap counting – direct
bitmap



                                    Set bits in the
                                    bitmap using hash
                                    of the flow ID of
                                    incoming packets

    HASH(green)=10001001

              [Estan et al. 2003]
Bitmap counting – direct
bitmap



                     Different flows
                     have different hash
                     values


    HASH(blue)=00100100
Bitmap counting – direct
bitmap



                     Packets from the
                     same flow always
                     hash to the same
                     bit

    HASH(green)=10001001
Bitmap counting – direct
bitmap



                     Collisions OK,
                     estimates
                     compensate for
                     them

    HASH(violet)=10010101
Bitmap counting – direct
bitmap




    HASH(orange)=11110011
Bitmap counting – direct
bitmap




    HASH(pink)=11100000
Bitmap counting – direct
bitmap



                     As the bitmap fills
                     up, estimates get
                     inaccurate


    HASH(yellow)=01100011
Bitmap counting – direct
bitmap


                     Solution: use more
                     bits




    HASH(green)=10001001
Bitmap counting – direct
bitmap


                     Solution: use more
                     bits
                     Problem: memory
                     scales with the
                     number of flows
    HASH(blue)=00100100
  Bitmap counting – virtual
  bitmap


Solution: a) store only a portion of the bitmap
         b) multiply estimate by scaling factor
Bitmap counting – virtual
bitmap




    HASH(pink)=11100000
Bitmap counting – virtual
bitmap



                    Problem: estimate
                    inaccurate when
                    few flows active


    HASH(yellow)=01100011
 Bitmap counting – multiple
 bmps




Solution: use many bitmaps, each accurate
          for a different range
Bitmap counting – multiple
bmps




    HASH(pink)=11100000
Bitmap counting – multiple
bmps




    HASH(yellow)=01100011
Bitmap counting – multiple
bmps




Use this bitmap to estimate number of flows
Bitmap counting – multiple
bmps




Use this bitmap to estimate number of flows
 Bitmap counting – multires.
 bmp

                                     OR


                        OR



Problem: must update up to three bitmaps
          per packet
Solution: combine bitmaps into one
Bitmap counting – multires.
bmp




    HASH(pink)=11100000
Bitmap counting – multires.
bmp




    HASH(yellow)=01100011
Multiresolution Bitmaps

   Still too expensive to scale
   Scaled bitmap
       Recycles the hash space with too many
        bits set
       Adjusts the scaling factor according
  Putting It Together
                                           Address Dispersion Table
                                           (scalable counters)

                                               key   src cnt dest cnt

header      payload
     substring fingerprints AD entry exist?
      substring fingerprints update counters


else         key   cnt
                                                counters > dispersion threshold?
update
                                                report key as suspicious worm
counter

                                cnt > prevalence threshold?
                                create AD entry
     Content Prevalence Table
     (multistage filters)
Putting It Together

   Sample frequency: 1/64
   String length: 40
   Use 4 hash functions to update
    prevalence table
       Multistage filter reset every 60 seconds
System Design

   Two major components
       Sensors
           Sift through traffic for a given address space
           Report signatures
       An aggregator
           Coordinates real-time updates
           Distributes signatures
Implementation and
Environment

   Written in C and MySQL (5,000 lines)
   Prototype executes on a 1.6Ghz AMD
    Opteron 242 1U Server
       Linux 2.6 kernel
EarlyBird Performance

   Processes 1TB of traffic per day
   200Mbps of continuous traffic
   Can be pipelined and parallelized for
    achieve 40Gbps
Parameter Tuning

   Prevalence threshold: 3
       Very few signatures repeat
   Address dispersion threshold
       30 sources and 30 destinations
       Reset every few hours
       Reduces the number of reported
        signatures down to ~25,000
Parameter Tuning

   Tradeoff between and speed and
    accuracy
       Can detect Slammer in 1 second as
        opposed to 5 seconds
           With 100x more reported signatures
Memory Consumption

   Prevalence table
       4 stages
           Each with ~500,000 bins (8 bits/bin)
       2MB total
   Address dispersion table
       25K entries (28 bytes each)
       < 1 MB
   Total: < 4MB
Trace-Based Verification

   Two main sources of false positives
       2,000 common protocol headers
           e.g. HTTP, SMTP
           Whitelisted
       SPAM e-mails
       BitTorrent
           Many-to-many download
False Negatives

   So far none
   Detected every worm outbreak
Inter-Packet Signatures

   An attacker might evade detection by
    splitting an invariant string across
    packets
   With 7MB extra, EarlyBird can keep
    per flow states and fingerprint across
    packets
Live Experience with EarlyBird

   Detected precise signatures
       CodeRed variants
       MyDoom mail worm
       Sasser
       Kibvu.B
Variant Content

   Polymorphic viruses
       Semantically equivalent but textually
        distinct code
       Invariant decoding routine
Extensions

   Self configuration
   Slow worms
Containment

   How to handle false positives?
       If too aggressive, EarlyBird becomes a
        target for DoS attacks
       An attacker can fool the system to block a
        target message
Coordination

   Trust of deployed servers
   Validation
   Policy
Conclusions

   EarlyBird is a promising approach
       To detect unknown worms real-time
       To extract signatures automatically
       To detect SPAMs with minor changes
   Wire-speed signature learning is viable

				
DOCUMENT INFO