Docstoc

Testing Telecoms Software with Quviq QuickCheck

Document Sample
Testing Telecoms Software with Quviq QuickCheck Powered By Docstoc
					Testing a Media Proxy with
  …        QuickCheck
Thomas Arts    Joakim Johansson
John Hughes        Ulf Wiger
Chalmers/ITU
                   Ericsson
  QuickCheck: Properties not Test
             Cases
• Write general properties of code instead of
  test cases
prop_reverse() ->
  ?FORALL(Xs,list(int()),
  ?FORALL(Ys,list(int()),
    lists:reverse(Xs++Ys)==
    lists:reverse(Xs)++lists:reverse(Ys))).

• Test in many, many randomly generated
  cases
         QuickCheck Testing
3> eqc:quickcheck(test:prop_reverse()).
...........Failed! After 12 tests.
[-3,2]                       A random counter-
[-3,1]                      example: Xs and Ys
Shrinking..........(10 times)
[0]                          NEW: Automatic
[1]                       simplification of failing
false                              cases


• Simplification is extremely important—
  separates the signal from the noise!
           Why Did it Fail?
prop_reverse() ->
  ?FORALL(Xs,list(int()),
  ?FORALL(Ys,list(int()),
    lists:reverse(Xs++Ys)==
    lists:reverse(Xs)++lists:reverse(Ys))).


• QuickCheck says: Xs=[0], Ys=[1]
• reverse([0,1])==reverse([0])++reverse([1])?
• Xs, Ys the wrong way round
    QuickCheck in a Nutshell
• Features
  – Properties, not test cases
  – Controlled random generation
  – Automated simplification


• Result
  – Compact, reusable test code
  – Specification checked against the code
  – Shortcut from property to bug
  Property First Development?
• Code code code… quickcheck       A Bug!

• Fix it… quickcheck      A Bug!


• Fix it… quickcheck… quickcheck
  quickcheck quickcheck

• Code some more…
       Ericsson’s Media Proxy
                    Media
                   Gateway
                   Controller


IP network,
  servers,
   etc etc



                                Media
                                stream
        How hard can it be?
• All we need to do is open and close
  ‖media pinholes‖ when the controller
  says…

• Megaco H.248 protocol
  – ITU standard… 212 pages!
  – Ericsson Interwork Description… 183 pages!
  – Control software… 150KLOC
     • 20-30K Megaco
Our Strategy
   QuickCheck
   Sends random command
   sequences
   Results make sense?



              Completely random
             commands are all just
            rejected—non-sensical!
                 Poor test data
Generating Sensible Messages
           Message Generators
• Example: A media descriptor contains a
  list of streams
  – ASN.1 from the standard (simplified):
   MediaDescriptor     ::= SEQUENCE
   {
     streams           CHOICE
     {
         oneStream     StreamParms,
         multiStream   SEQUENCE OF StreamDescriptor
     }
   }
            QuickCheck Generator
mediadescriptor(Streams) when Streams=/=[]->
    {mediaDescriptor,
     #'MediaDescriptor'{            Records generated by
        streams =
          case Streams of
                                       ASN.1 compiler
               [{Id,Mode}] ->
                   oneof([{oneStream,streamParms(Mode)},
                          {multiStream,[stream(Id,Mode)]}]);
               _ ->
                   {multiStream,
                       [stream(I,M) || {I,M}<-Streams]}
          end}}.
                                          Message construction
                                           Logic from the IWD
stream(I,Mode) ->
                                               QuickCheck
    #'StreamDescriptor'{
        streamID = I, streamParms = streamParms(Mode)}.
                 Conditions in the IWD
Add Request
Desc.   Desc.    Desc.     Properties,   Package   M   Comment

Media                                              M
        Stream                                     M   Multiple Stream descriptors can be
                                                            included.
                 Local                             O   LocalControl will be included in all
                 control                                   cases except when no media
                                                           (m-line) is defined in the remote
                                                           SDP.
                           mode                    O   The default value of the mode property is
                                                           ―Inactive‖, the property is not
                                                           mandatory if the wanted value is
                                                           ―Inactive‖.


  StreamParms         ::= SEQUENCE
  {
     localControlDescriptor LocalControlDescriptor OPTIONAL,
     localDescriptor      LocalRemoteDescriptor OPTIONAL,                               Remot
     remoteDescriptor LocalRemoteDescriptor OPTIONAL,                                   e SDP
     …,                                                                                 in here
     statisticsDescriptor StatisticsDescriptor OPTIONAL
  }
       Two Cases: With and Without
             Remote Media
streamParms(Mode) ->                          Included in this
    ?LET(RemoteMediaDefined, bool(),                case
         if RemoteMediaDefined ->
                  #'StreamParms'{
                         localControlDescriptor =
                          localControl(Mode),
                        localDescriptor =
                          localDescriptor(RemoteMediaDefined),
                        remoteDescriptor =
                          remoteDescriptor(RemoteMediaDefined)};
            true ->
                  ……
         end).
                                     Passed on to ensure an
                                       m-line is generated
Generating Sensible Commands
Megaco Commands
         Context
                   Context
Megaco Commands
                Context
                          Context

   Context

       Termination

                          Add
       Megaco Commands
                           Context
                                     Context
   Add
              Context

                  Termination
Termination
       Megaco Commands
                           Context
                                       Context

              Context

                  Termination
Termination
                                      Modify
                         Stream
               Stream
                                         Uses
                                     termination ID
       Megaco Commands
                           Context
                                     Context
  Subtract
              Context

                  Termination
Termination
                         Stream
               Stream
Megaco Commands
                Context
                            Context

   Context

       Termination

                          Subtract
              Stream
    Stream
Megaco Commands
         Context
                   Context
  Sensible Command Sequences?
• Track the state of each test case
  – What calls in progress, which terminations in each
    call…
  – Simple to model in Erlang
• State machine model
  – Preconditions say which commands are appropriate
  – Postconditions check command results
  – Next state function specifies how commands behave
• QuickCheck library
  – generates valid sequences, tests, and shrinks them
  – (developed ‖just in time‖)
                    Faults
• Proxy was already well tested

• 6 days of work, mostly writing generators

• 5 faults found:
  – One in Megaco message encoding/decoding
  – Four command sequences crashed
    Error-provoking Sequences

•   Add   Mod         Streams must have two ends


•   Add   Sub         Here today, gone tomorrow…

                            …with differing numbers
•   Add   Add   Mod         of streams
               The Best Error!

                              Shrinking reduced 160
                              commands to seven!




Add   Add     Sub     Add     Sub     Add     Sub


                                    No one in their right
                                    minds would test this!
Due to data corruption here
      ‖Most Likely Bug‖ Effect
• Once a bug is found, almost every
  QuickCheck run finds the same one!
  – (or at least, shrinks to it)


• Add bug preconditions to the model to
  avoid it and find the next one
  – Formulate a ‖bug hypothesis‖
  – Test the hypothesis!
  – Document it as a precondition
          Real bugs or not?
• Most of these sequences are not sent by
  Ericsson’s Media Gateway Controller
  – Controller and Proxy were tested together!


• But the Interwork Description doesn’t say
  so!

• One of the IWD/the code was buggy!
   Using QuickCheck Earlier?
• Testing an older version
  – Using the same generators and model


• 9 errors found in 6 hours!
  – (2 TRs for this version)


• Most of the work: writing bug preconditions
    QuickCheck is Moreish…
• Follow-up projects under way




• Testing non-Erlang systems (the big one!)
              Conclusions
• QuickCheck + Erlang
  – Simple declarative models of the SUT
  – Concise, maintainable test code

• Testing: a great application for FP
  – Performance irrelevant
  – No need to commit to Erlang in the product

• A winning combination?

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:9
posted:3/15/2010
language:Swedish
pages:31