Docstoc

Comparison of Blackbox and White

Document Sample
Comparison of Blackbox and White Powered By Docstoc
					Comparison of Blackbox and Whitebox
  Fuzzers in Finding Software Bugs
              Marjan Aslani, Nga Chung, Jason Doherty,
               Nichole Stockman, and William Quach

        Summer Undergraduate Program in Engineering Research at Berkeley

                                (SUPERB) 2008

               Team for Research in Ubiquitous Secure Technology




          TRUST Autumn 2008 Conference: November 11-12, 2008
Overview

    Introduction to Fuzz testing
    Our research
    Result




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   2
What Is Fuzzing?

    A method of finding software holes by feeding
     purposely invalid data as input to a program.
      – B. Miller et al.; inspired by line noise


      –   Apps: image processors, media players, OS
      –   Fuzz testing is generally automated
      –   Finds many problems related to reliability; many of which are
          potential security holes.




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   3
Types of Fuzz Testing


       BlackBox : Randomly generated data is fed to a program as
        input to see if it crashes.

       –     Does not require knowledge of the program source code/ deep
             code inspection.
       –     A quick way of finding defects without knowing details of the
             application.

       WhiteBox: Creates test cases considering the target
        program's logical constraints and data structure.

       –     Requires knowledge of the system and how it uses the data.
       –     Deeper penetration into the program.




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   4
Zzuf - Blackbox Fuzzer


     Finds bugs in applications by corrupting
      random bits in user-contributed data.
     To make new test cases, Zzuf uses a range
      of seeds and fuzzing ratios (corruption ratio).




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   5
Catchconv - WhiteBox Fuzzer

    To create test cases, CC starts with a valid input,
     observes the program execution on this input, collects
     the path condition followed by the program on that
     sample, and attempts to infer related path conditions
     that lead to an error, then uses this as the starting
     point for bug-finding.

    CC has has some downtime when it only traces a
     program and is not generating new fuzzed files.




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   6
Valgrind

    A tool for detecting memory management
     errors.
    Reports the line number in the code where the
     program error occurred.
    Helped us find and report more errors than we
     would if we focused solely on segmentation
     faults.




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   7
Types of errors reported by Valgrind

By tracking a program’s execution of a file, Valgrind determines the
types of errors that occur which may include:

    Invalid writes
    Double free - Result 256
    Invalid reads
    Double free
    Uninitialized values
    Syscal Pram
    Memory leak




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   8
Program run under Valgrind




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   9
Methodology




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   10
Metafuzz

          All of the test files that triggered bugs were
           uploaded on Metafuzz.com.
            –   The webpage contained:
                     Link to the test file
                     Bug type
                     Program that the bug was found in
                     Stack hash number where the bug was located




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   11
Metafuzz webpage




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   12
Target applications

    Mplayer, Antiword, ImageMagick Convert and Adobe
     Flash Player
    MPlayer the promary target:
      –   OS software
      –   Preinstalled on many Linux distributions
      –   Updates available via subversion
      –   Convenient to file a bug report
      –   Developer would get back to us!
    Adobe bug reporting protocol requires a certain bug to
     receive a number of votes form users before it will be
     looked at by Flash developers.
    VLC requires building subversions from nightly shots.
"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   13
Research Highlights

        In 6 weeks, generated more than 1.2 million
         test cases.
        We used UC Berkeley PSI-cluster of
         computers, which consists of 81 machines
         (270 processors).
          –    Zzuf, MPlayer, and CC were installed on them.
        Created a de-duplication script to find the
         unique bugs.
        Reported 89 unique bugs; developers have
         already eliminated 15 of them.
"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   14
Result

          To provide assessments for the two fuzzers,
           we gathered several metrics:

            –   Number of test cases generated
            –   Number of unique test cases generated
            –   Total bugs and total unique bugs found by each
                fuzzer.




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   15
Result con’t

    Generated 1.2 million test cases
      –   962,402 by Zzuf.
      –   279,953 by Catchconv.
    From the test cases:
      –   Zzuf found 1,066,000 errors.
      –   Catchconv reported 304,936.
    Unique (nonduplicate) errors found:
      –   456 by Zzuf
      –   157 by Cachconv


"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   16
Result con’t

    Zzuf reports a disproportionately larger
     amount of errors than CC. Is Zzuf better than
     CC?
    No! The two fuzzers generated different
     numbers of test cases.
    How could we have a fair comparison of the
     fuzzers’ efficiency?
      –   Need to gauge the amount of duplicate work
          performed by each fuzzer.
      –   Find how many of these test cases were unique.

"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   17
Average Unique Errors per 100 Unique Test
Cases


    First, we compared
     performance of the
     fuzzers by the average
     number of unique bugs
     found per 100 test
     cases.
      –   Zzuf: 2.69
      –   CC : 2.63


    Zzuf’s apparent
     superiority diminishes.


"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   18
Unique Errors as % of Total Errors

    Next, we analyzed fuzzers’
     performance based on the
     percentage of unique
     errors found out of the
     total errors.
      –   Zzuf: .05%
      –   CC: .22%


    Less than a quarter
     percent difference
     between the fuzzers.


"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   19
Types of Errors (as % of Total Errors)

    Also considered
     analyzing the fuzzer
     based on bug types
     found by the fuzzers.

    Zzuf performed better in
     finding “invalid write”,
     which is a more
     important security bug
     type.

    Not an accurate
     comparison, since we
     couldn’t tell which bug
     specifically caused a
     crash.
"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   20
Conclusion

    We were not able to make a solid conclusion
     about the superiority of either fuzzer based on
     the metric we gathered.

    Knowing which fuzzer is able to find serious
     errors more quickly would allow us to make a
     more informed conclusion about their
     comparative efficiencies.



"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   21
Conclusion con’t

    Need to record the amount of CPU clock
     cycles required to execute test cases and find
     errors.
    Unfortunately we did not record this data
     during our research, we are unable to make
     such a comparison between the fuzzers.




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   22
Guides for Future Research

To perform a precise comparison of Zzuf and CC:
1.  The difference between the number of test cases
    generated by Zzuf and CC for a given seed file and
    specific time frame should be recorded.
2.  Measure CPU time to compare the number of
    unique test cases generated by each fuzzer for a
    given time.
3.  Need a new method to identify unique errors avoid
    reporting duplicate bugs:
           Need to use automatically generate a unique hash for each
            reported error that can then be used to identify duplicate
            errors.

"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   23
Guides for Future Research con’t


4. Use a more robust data collection
       infrastructure that could accommodate the
       massive amount of data colected.
      –     Our ISP shut Metafuzz down due to excess
            server load.
      –     Berkeley storage full.
5. Include an internal issue tracker that keeps
       track of whether or not a bug has been
       reported, to avoid reporting duplicate bugs.


"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   24
WhiteBox or BlackBox??

    With lower budget/ less time: use Blackbox
    Once low-hanging bugs are gone, fuzzing
     must become smarter: use whitebox
    In practice, use both.




"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   25
Acknowledgment

    National Science Foundation (NSF) for funding this
     project through the SUPERB-TRUST (Summer
     Undergraduate Program in Engineering Research at
     Berkeley - Team for Research in Ubiquitous Secure
     Technology) program
    Kristen Gates (Executive Director for Education for
     the TRUST Program)
    Faculty advisor David Wagner
    Graduate mentors Li-Wen Hsu, David Molner,
     Edwardo Segura, Alex Fabrikant, and Alvaro
     Cardenas.

"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   26
                                       Questions?


                                       Thank you


"Comparison of Blackbox and
Whitebox Fuzzers in Finding Software
Bugs", Marjan Aslani                   TRUST Autumn 2008 Conference: November 11-12, 2008   27

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:15
posted:5/16/2010
language:English
pages:27