Testing by xuyuzhu

VIEWS: 3 PAGES: 15

									Signature-Builder


    Testing
   Document
1 Testing functional requirements ............................................................................... 3
1.1 SignatureBuilder ..................................................................................................... 3
1.2 Evaluation Envoironment and CFL ..................... Error! Bookmark not defined.6
2 Testing non-functional requirements...................................................................... 10
2.1 Speed, Capacity & Throughput ............................................................................ 10
2.2 Reliability: ............................................................................................................. 10
2.3 Safety & Security: ................................................................................................. 11
2.4 Usability ................................................................................................................. 10
3 Test-Driven development ......................................................................................... 11
4 Random & automatically-generated tests ............................................................... 13
5 Testing the user interface ........................................................................................ 14
6 Testing build, integration & deployment................................................................. 15
1 Testing functional requirements
1.1 Signature Builder – server
Requirement #1 in the ARD: Extract Functions
Testing IDAPRO extractor
Test No.   Test Description                          Expected Result

1          Success scenario – trying to extract      A list of functions extracted
           functions from file – while file exists



2          failure scenario – trying to extract      Error message
           functions but file doesn’t exist




Testing dataMiningExtractor
Test No.   Test Description                          Expected Result

1          Success scenario – trying to extract      A list of functions extracted
           functions from file – while file exists



2          failure scenario – trying to extract      Error message
           functions but file doesn’t exist
Requirement #2 in the ARD: Filter Functions
Test No.   Test Description                          Expected Result

1          Success scenario – filtering a list of    A list of non-common functions
           functions and the list exists



2          Failure scenario – trying filter a list   Empty List
           while there is no list of functions




Requirement #3 in the ARD: Generate Signatures
Test No.   Test Description                          Expected Result

1          Success scenario – expanding the          The function is expanded
           function while the attributes are not
           null



2          failure scenario – trying to expand       Error message
           the function while the attributes are
           null




Requirement #4 in the ARD: Select Candidate
Testing RandomSelctor
Test No.   Test Description                          Expected Result

1          Success scenario – Select a               A random signature selected
           signature from the candidates and
           the list exists



2          failure scenario – trying to Select a     Error message
           signature while the list doesn’t exist
Testing ClusterSelctor
Test No.   Test Description                          Expected Result

1          Success scenario – Select a               A signature with maximum score of
           signature from the candidates and         clusters selected
           the list exists



2          failure scenario – trying to Select a     Error message
           signature while the list doesn’t exist




Testing MinEntropySelctor
Test No.   Test Description                               Expected Result

1          Success scenario – Select a signature          A signature with minimum
           from the candidates and the list exists        entropy selected



2          failure scenario – trying to Select a          Error message
           signature while the list doesn’t exist




Testing probabilitySelector
Test No.   Test Description                          Expected Result

1          Success scenario – Select a               A signature with minimum
           signature from the candidates and         probability of existence in a benign
           the list exists                           file.



2          failure scenario – trying to Select a     Error message
           signature while the list doesn’t exist
Requirement #5 in the ARD: create Signature
Test No.   Test Description                          Expected Result

1          Success scenario – trying to create       Returns a signature.
           signature after initialize all the
           fields.



2          Failure scenario: trying to create        Error message
           signature while selector is not
           initialize.

3          Failure scenario: trying to create        Error message
           signature while extractor is not
           initialize.

4          Failure scenario: trying to create        Error message
           signature while CFL is empty




1.2 Evaluation Envoironment and CFL
    Requirement #1 in the ARD: CFL
Testing Create CFL
Test No.   Test Description                          Expected Result

1          Success scenario – creating CFL and       The cfl is initialized
           there is an input file



2          Failure scenario – trying to create cfl   Error Message
           while there is no input file
Testing Reset CFL

Test No.   Test Description                         Expected Result

1          Success scenario – success clearing      The CFL is Empty
           the CFL while the CFL exists



2          Failure scenario – trying to reset the   Error Message
           cfl while there is no CFL




Testing Save CFL
Test No.   Test Description                         Expected Result

1          Success scenario – success saving        The CFL is saved as a file
           the CFL while the CFL exists



2          Failure scenario – trying to save the    Error Message
           cfl while there is no CFL




Testing Update CFL
Test No.   Test Description                         Expected Result

1          Success scenario – the Functions         The CFL includes the new functions
           were added to the cfl
                                                    AND
           while the CFL exists
                                                    CFL.size>0

2          Failure scenario – update the cfl        Error Message
           while there is no CFL
Requirement #2 in the ARD: Signature Malwares
Test No.   Test Description                      Expected Result

1          Success scenario – trying to produce Returns a set of signatures.
           signatures from each malware after
           initialize all the fields.



2          Failure scenario: trying to create    Error message
           signatures while selector is not
           initialize.

3          Failure scenario: trying to create    Error message
           signatures while extractor is not
           initialize.

4          Failure scenario: trying to create    Error message
           signatures while CFL is empty


Requirement #3 in the ARD: Signature Benign files


Test No.   Test Description                      Expected Result

1          Success scenario – trying to produce Returns a set of signatures.
           signatures from each benign file
           after initialize all the fields.



2          Failure scenario: trying to create    Error message
           signatures while selector is not
           initialize.

3          Failure scenario: trying to create    Error message
           signatures while extractor is not
           initialize.

4          Failure scenario: trying to create    Error message
           signatures while CFL is empty
Requirement #4 in the ARD: Show Statistics
Test No.   Test Description                       Expected Result

1          Success scenario – Compare each        Return the probability of the
           malware's signature with the set of    uniqueness for the malware's.
           benign file's signatures.

2          Failure scenario: there is no set of   Error message
           benign file's signatures

3          Failure scenario: there is no          Error message
           malwares signature
2 Testing non-functional requirements
2.1 Speed, Capacity & Throughput
Test No.   Test Description                       Expected Result

1          Loading CFL                            created from 1GB of files should
                                                  take no more than 10mins

2          Generating CFL                         Generating CFL from 1GB of files
                                                  should take no more than 2hrs

3          Generating signature for a malware     1MB should not take more than 5sec
           file

4          Searching a 48byte signature in GB     should take less than 8sec
           of files

5          Generating 3grams and 2-grams              created from 1 GB of file, should
           repository from the CFL files                      take no more than 2hrs.




2.2 Reliability:
Test No.   Test Description                       Expected Result

1          generate signatures up to 5Gb files    The system will have an output
                                                  without memory problems.



    We will perform many tests on the system with different sizes of cfl, benign files,
    malicious files and measure the system's reliability by calculating the false alarms
                                                                           percentage.
2.3 Safety & Security:
The system domain expert will be authorized to configure the system and all the
components in it.



Test No.   Test Description                        Expected Result

1          Change configuration of the             Check if the output is corresponding
           signature builder                       to the input and the configuration of
                                                   the system




2.4 Usability:
Testing usability will be by letting someone to try the system, doing the follows:
Installing the evaluation environment and the signature builder.
Change configuration of the system
Create cfl, add files and delete files.
Create signature to malicious files
Use the evaluation environment and get statistics for a set of files.
Check the response of the system.


At last get his opinion of his understanding and using the system.
3 Test-Driven developments
We didn't use the TDD approach.
We think we don’t have enough experience in software developing to use this
approach for such a big project.

In addition, as part of the course we had to present a full design of the
system, this software developing approach contradicts the TDD approach,
which the design should be created through the tests.
4 Random & automatically-generated
tests
Random and automatically tests for the signature builder are done by the
evaluation environment. The signature builder is a server and the evaluation
environment is actually a client that tests the signature builder according to different
configurations. The main idea is to help deciding on a good signature: unique and
optimal.

Each configuration consists the following input:

-   CFL size in MB
-   maximum signature length in byte
-   Function similarity threshold
-   Offset size in byte
-   Function Extractor
-   function selection.

The output will consist the following:

-Processed - The number of malware files that the system managed to generate a
signature for them.

-Processed (%) - Processed / Total Malware Files.

-Signature Hits - The number of malware files that gives at least one False Alarm,
which means the number of unique malware files that produced False Alarm.

-Signature Hits (%) - Signature Hits / Processed.

-Hits In Files - The number of the signature's hits in the Control Group for any pair
(signature, file)

-Unique Signature - The number of unique signatures produced FA.

-Different Files - The number of distinct files in the CG that has at least one hit.

-Different Files (%) – Different Files / Total Control Group Files.
5 Testing the user interface
We have two components with GUI to test – one is the signature builder itself and
the other one is evaluation environment.
Testing the configuration in the signature builder – the test is simple, for every
change in the configuration screen, all we have to check if there was a change in the
configuration.txt file.
Cfl manager – for any change of the cfl we will test the new cfl, when we delete it –
we check if the size of it is 0, if we add/delete function – check new size of the cfl.


For the other screens in both the signature builder and the evaluation environment
all the tests of their functionality were mention on the other parts of this document.
6 Testing build, integration &
deployment
The signature builder and the evaluation environment package will be a standard
windows installation executable, thus our project doesn’t need any special needs for
the building and deployment stages. The use of this standard is different from a
GNU sources packages, therefore, the installation and the deployment for the project
is easy to test – such an installer has already been tested in the past.

								
To top