Unit testing (OHJ-306x)

Document Sample
Unit testing (OHJ-306x) Powered By Docstoc
					               Unit testing (OHJ-306x)
                               Tools, techniques, tips and tricks

                                           Mika Maunumaa
                                    Tampere University of Technology

                                               Unit Testing (OHJ-306x)

Department of Information Systems

                  What is unit testing?
                           • Quick and short intro
                           • Terminology review
                  The assignment (Firefox 3.6 alpha 1)
                           • Test target
                           • Usage scenario
                  Setting up the test environment
                  Tools for unit testing
                  Reading and comprehending the code
                           • Things about C-language

Department of Information Systems                      Unit Testing (OHJ-306x)
                  Unit testing in big picture

                  Testing levels
                           • Acceptance testing
                                • Carefully selected tests verifying that the system works as customer
                                • In real environment
                           • System testing
                                • Tests verifying that the system works are specified
                                • In laboratory or real environment
                           • Integration testing
                                • Depending on abstraction level
                                          – Tests interaction between modules
                                          – Tests interaction between subsystems
                                • In simulated or laboratory environment
                           • Unit testing
                                • Testing external behavior of a unit or small cluster of units

Department of Information Systems                             Unit Testing (OHJ-306x)
                  Unit testing

Testing for individual building blocks of a                               Test Suite
system                                                                            • Group of tests
                                                                                  • Group of test suites
A unit can be a module, a class or a function                                     • Tests for same part of a unit
         • Logically bound set                                                    • Tests using same technique
         • Unit Under Test (UUT
                  • MUT, CUT, FUT etc.
                                                                          Test Case
                                                                                  • Test sequence with purpose
In object-oriented testing the definition of                                      • A set of test data, test sequence and
unit can be a bit vague                                                             expected result
         • When object cluster is big enough to be a unit                 Test Case has three stages:
         • When the cluster becomes too large
                                                                                  • Setup and initialization
            • Integration testing
                                                                                  • Test sequence
                                                                                  • Cleanup and reset
A unit depends on other units
         • Those that uses and drives            drivers
                  • Imitates the real usage of the UUT
         • Those that are used by UUT              stubs
                  • Imitates the functionality of the real code

Department of Information Systems                                 Unit Testing (OHJ-306x)
                  Unit Test Case execution phases

                  1a) Test setup
                           • Constructs the test setting
                           • Prepares stubs for particular test case
                               • Correct or incorrect answer
                               • ‘Hard to find’ error situation
                                          – Disk Full, Memory Full, etc.
                  1b) Test initialization
                           • Prepares UUT
                               • Correct initial state
                               • Connects UUT to stubs, if necessary
                           • Reserves resources
                               • DB connection, TCP/IP connection, dynamic memory etc.
                  2 ) Test sequence exercises the UUT
                           • Driver executes usage scenario, verifies the result and prepares the verdict
                           • If top-down approach is used, the indirect driver can be used
                                 • Actual driver exercises indirect driver (some system class) that exercises the actual
                                    unit under test
                  3a) Test cleanup free reserved resources
                  3b) Test reset unwinds the test setting

                  Depending on test environment and framework you are using, each phase may or may
                    not overlap with next phase

Department of Information Systems                                    Unit Testing (OHJ-306x)
                  Making unit test cases

                  Problem: where to acquire knowledge about UUT
                           • Specification for external behavior
                              • Black-box testing
                              • Mozilla provides no documentation, make assumption about the
                           • Reading the code
                              • White-box testing
                              • Requires code reading and comprehending skills
                              • No documenting comments in code      code is a document
                           • Social engineering
                              • Common coding rules and how coders normally avoids to using
                  Testing techniques
                           • Boundary values, NULL-pointers, domain violations
                           • Test data tools to extract (good) test data

Department of Information Systems                       Unit Testing (OHJ-306x)
                  Making unit test cases

                  In general, when you don’t have interface specification or the
                    specification is rubbish, you as a tester have to:
                           • Use your domain knowledge and practical experience to make
                             educated guess about the interface specification
                           • Look for the code for hints that might give you the clue how it should
                           • Count on coverage metric tools and create tests that cover 100% of
                             the code
                           • Write tests that verifies if the code works as it is coded (???)
                  Each method should have at least one TestSuite of their own
                           • Basic functionality tests for all methods can be in the same TestSuite
                  Each test case should be as small as possible, but not smaller
                  Each test sequence should test one thing and one thing only
                           • If there are two or more sequences places in one test case, the
                             failure in one will break the sequence  perfect and/or importance
                             test sequence are ignored

Department of Information Systems                         Unit Testing (OHJ-306x)
                  Making unit test cases

                  The only successful test is the one that finds bugs
                     • No bugs found no bugs in the system            poor testing
                     • “A bug is a social creature”, if there is one, there can be more
                     • Testing more you can only rise your confidence about
                       systems quality
                     • However, test which verifies if the unit is behaving as it
                       should are important
                  Enough test cases?
                           • Test until you drop
                           • Deadline
                           • McCabe’s cyclomatic complexity metric       estimate for lower
                             boundary of the number of test cases
                           • Multiconditional coverage -> boolean statements every truth value
                             combination will be tested
                               • If (a>1 && b<5)     { (a=1,b=5), (a=2, b=5), (a=1, b=4),
                               • Short circuit mechanism

Department of Information Systems                        Unit Testing (OHJ-306x)
                  Making unit test cases

                  Multiconditional coverage requirement
                           • 100 % required, but less can be reached
                           • It is only a mean to measure ‘ENOUGH’, not the end
                  If you are pursuing only for maximum MC coverage you will forget to test
                           • Test contains ALWAYS four parts
                               1. Prepare (initialize UUT and other data)
                               2. Execute (prepare method parameters and execute method/function)
                               3. Examine return values and compare them to expected values
                               4. Clean up

                           • How do you call sequence that does only 1, 2, and 4?
                               • A function call
                               • Is not a TEST

Department of Information Systems                           Unit Testing (OHJ-306x)
                                          Control flow graph of ParseURL
                                                                                                                                                                                                            IF 3
                                                                                                                                                   rivillä 126
                                                                                                                                                                        [stop != 0]        [colon != 0]
                                                                                                                                                [colon > stop]
                                                                                                                                                                                [stop == 0]             [colon == 0]

                      [specLen < 0]                                                                                                                        [colon <= stop]
         [true]                                                                                                                                                                                             IF 4

            specLen < 0
                                                                                                                                                                            [return NS_OK]
            specLen = strlen(spec)                                                                                                                                                                     [specLen != 0]
                                                                                                                                                       r 130-134

                                                                                                                                                                                                          FOR 2

                  FOR 1

                                                                          [true]              [true]                                                                                                                                  [p != spec]
       [false]        [len != 0]                     [*p > '\0']                                                                                                                                                      [*p <= ' ']
                                                                         [*p <= ' ']
                                                                                                                                                                                                   [*p > ' ']                         [p == spec]
                     [*p != '\0'] [true                     [false]              [false]
       [false]                                             [*p == ':']          [*p == ':']     [true]          [false]                                                                                             [colon == 0]
                     *p != '\0']                                                               [colon != 0]
                     [true]                                                                                                                                                                                     [colon != 0]
                     [colon == 0] [true                                                                         [true]
       [false]                                                                [false]                                                                  then haara                     [colon < slash]                                          else haara
                     colon == 0]                                             [*p == '/']                                                               r 143 - 165                                                                             r 167 - 186
                      [true]                                                                                                                                                                                       [colon >= slash]
       [false]       [slash == 0]]
                                                                                                [true]                                                                                  [slash != 0]                  [slash == 0]                                                 [authorityLen != 0]
                       [true]                                                                                                                                                                                                                                [authorityLen == 0]
                                                                              [false]                                                    [!net_IsValidScheme]
                                                                             [*p == '?']        [true]                                                                                                                                                                     [pathLen != 0]
IF 3                                                                          [false]                         [slash != 0]
                                                                             [*p == '#']                                                          [*(colon+1) == ':']            [*(colon+1) != ':']                                                           [pathLen == 0]
                                                                                                [true]                     [false]

                                                                              [false]                                    [true]      NS_ERROR_MALFORMED_URI
                                                                             [*p == ';']        [true]                                                                     [authorityLen != 0]

                                                                                                                                                                                                   [authorityLen == 0]
                                                                             [*p == '@']        [true]                                                                                [pathLen != 0]
                                                                                                                [stop != 0]
                                                                                                                                                                                                        [pathLen == 0]
                                                                             [*p == '[']        [true]                     [false]

                 Department of Information Systems                                                                                      Unit Testing (OHJ-306x)
                  Unit test assignment

                  Test testing target is Firefox browser’s URL-parser
                           • Firefox version 3.6 alpha 1
                           • Parser is line and character based
                           • Parses line from left to right
                  UUT is class called nsStdURLParser
                           • In file nsURLParsers.cpp (.h)
                  Methods that must be tested (McCabe)
                           •    ParseURL                                (32)
                           •    ParseAfterScheme                        ( 7)
                           •    ParseFilePath                           (12)
                           •    ParseFileName                           (6)

Department of Information Systems                             Unit Testing (OHJ-306x)
                  Unit Test Configuration

                     package UUT [     Test Environment ]

                         nsURLParsers                                         nsIURLParser


                                            nsBaseURLParser                       +net_IsValidScheme( s : char"*", l : int )
                                                                                -str : char"*"
                                                                                -len : int
                              nsNoAuthURLParser       nsAuthURLParser           <<constructor>>+nsCAutoString( s : char"*", l : int )
                                                                                +ToInteger( err : PRInt32"*" ) : PRInt32

                           UUT                       +ParseAfterScheme()                         Stub interfaces

Department of Information Systems                                          Unit Testing (OHJ-306x)
                  UUT usage scenario

  spec = ”http://www.cs.tut.fi:8080/~testaus”
  specLen = strlen(spec)
  NS_IMETHOD ParseURL(const char *spec, PRInt32 specLen,
                     PRUint32 *schemePos, PRInt32 *schemeLen, PRUint32 *authorityPos, PRInt32 *authorityLen,
                     PRUint32 *pathPos, PRInt32 *pathLen);
  spec2 = (spec + authorityPos + 1)                  //www.cs.tut.fi:8080/~testaus
  specLen = authLen + pathLen - 1
  void     ParseAfterScheme(const                    char *spec2, PRInt32 specLen,
                                              PRUint32 *authPos, PRInt32 *authLen,
                                              PRUint32 *pathPos, PRInt32 *pathLen)
  serverinfo = (spec + authorityPos)                      //www.cs.tut.fi:8080
  serverinfoLen = authorityLen
  NS_IMETHOD          ParseServerInfo(const             char *serverinfo, PRInt32 serverinfoLen,
                                                  PRUint32 *hostnamePos, PRInt32 *hostnameLen, PRInt32 *port)
  ||//www.cs.tut.fi|:8080                      port == 8080
  path = (spec + pathPos)                           /~testaus
  pathLen2 = pathLen
  NS_IMETHOD          ParsePath(const            char *path, PRInt32 pathLen2,
                                    PRUint32 *filepathPos, PRInt32 *filepathLen, PRUint32 *paramPos, PRInt32 *paramLen,
                                    PRUint32 *queryPos, PRInt32 *queryLen, PRUint32 *refPos, PRInt32 *refLen)
  Rest output paramaters are 0 and -1 (*Pos and *Len)

Department of Information Systems                                    Unit Testing (OHJ-306x)

                  Setting up the unit test environment for assigment
                           • Copy assignment files to your working directory
                               • mkdir ~/testaus
                               • cd ~/testaus
                               • /share/tmp/testaus/scripts/setup_unittest
                               • cd unittest
                           • Coverage measurement tool from Testwell
                           • Instruments the source code with measuring code
                                • Executed before compilation
                                • gcc file.cpp ctc –i m gcc file.cpp (instrument with multiconditional cov.)
                           • Collects data to mon.dat file
                                • Cumulatively, so delete this file before your final test and delivery
                           • Requires some environment variables
                                • For every login you must execute
                                          . /share/testwell/bin/testwell_environment
                           • Reading results
                                • ctcpost MON.dat | less
                                • ctc2html for html output
                                • Consult their man pages

Department of Information Systems                              Unit Testing (OHJ-306x)
                  CppUnit - unit test framework

          Unit test declaration                                 Unit test implementation

          #include <cppunit/extensions/HelperMacros.h>          #include "testCAutoString.h"
                                                                #include "nsIURLParser.h"
          class TestCAutoString : public CppUnit::TestFixture
          {                                                     CPPUNIT_TEST_SUITE_REGISTRATION( TestCAutoString );
            CPPUNIT_TEST_SUITE( TestCAutoString );
              CPPUNIT_TEST( testNormalString );                 void TestCAutoString::setUp()
            CPPUNIT_TEST_SUITE_END();                           {
                                                                   // before each test function
          public:                                               }
           void setUp();
           void tearDown();                                     void TestCAutoString::tearDown()
            void testNormalString();                               // after each test function
            void testNullString();
          };                                                    void TestCAutoString::testNormalString()
                                                                   // test steps here

Department of Information Systems                               Unit Testing (OHJ-306x)
                  Make and Makefile (MTP 15.5)

                  Build management
                           •    Compilation rules
                           •    Compilation targets

                  Assignment contains a Makefile and there is only one line that must be modified
                  # Do not modify anything else but the list of object
                  # files required by the "unittest" target:

                  # !!! ADD HERE CPPUNIT TESTCASES
                  unittest: unit.o stubs.o testRunner.o testSomething.o
                     $(CTCCC) -o $@ $^ $(LINKFLAGS)

                  stubtest: stubs.o testRunner.o testCAutoString.o testUtilityFunctions.o unit.o
                  You should not need to modify anything else

Department of Information Systems                               Unit Testing (OHJ-306x)
                  Things that they did not taught in school

                  A pointer to memory location
                  Always ends with null character ‘\0’
                  Vector of 8 bit integer values                                                  Hello\0
                  Common translation to ASCII characters                                      p
                  Pointer shows the head
                           • p == 0x100
                                                                                                  0x100     72    H
                           • The length of the string p is 5
                                • strlen(p) = 5                                                   0x101     101   e
                           • length of char vector itself is 6
                  How to access data                                                              0x102     108   l
                           • *p == 72 == H
                           • *(p+5) == 0                                                          0x103     108   l
                           • p[3] == 108 == ‘l’
                  C-String cannot be copied as an object                                          0x104     111   o
                           • char * s=p copies only value 0x100, not the string
                             itself                                                               0x105     0     \0
                           • char *s = strdup(p) // free s after use
                             std::string ss(p)
                  Common security risks
                           • Do not use C-String functions that does rely on the
                             presence of trailing null character
                           • Use strnXXX-functions that takes the maximum
                             number of characters as a parameter

Department of Information Systems                                   Unit Testing (OHJ-306x)
                  Odd things while implementing stubs

                  Do not return stack variable
                  int * func()
                     int bar = 5;
                     return &bar;
                  Do not store stack variable to class member
                  Do not store vector pointer parameter to class member
                           • Content may have been deleted
                  void someClass::func(const char * p, int length)
                     strClassMember = p;          | strClassMember = strndup( p, length );
                  Do not rely on atoi-function
                      • atoi(“Hello World”) == 0
                  Sometimes it is enough to implement simple stub
                      • Test data can order UUT to always call the stub with certain data
                                    • Implement stub that statically returns what is wanted and error otherwise
                  c-string functions often introduces security risks
                        • strdup vs. strndup
                        • char * strndup( pSrc, copyLength )

Department of Information Systems                                      Unit Testing (OHJ-306x)
                  Thank you

                                    Questions !

Department of Information Systems       Unit Testing (OHJ-306x)