Software Engineering Software Testing and Verification

Document Sample
Software Engineering Software Testing and Verification Powered By Docstoc
					                                                            Course Staff

                                                            • Instructor
                                                               – Alex Aiken
           Software Engineering:
      Software Testing and Verification                     • Teaching Assistant
                                                               – Brittney Fraser
                       Lecture 1
                        CS295



                   Prof. Aiken CS 295 Lecture 1         1                          Prof. Aiken CS 295 Lecture 1       2




Course Communication                                        Course Structure

• All class materials will be on the web                    • Lectures
                                                               – Course taught from notes
  – Lecture notes, handouts, papers to read, etc.
                                                               – Focus on programmer’s view of software engineering
                                                                  • Technology over business issues
• Read the web site for announcements
                                                            • Homeworks
                                                               – For those taking the class for 3 credits
• Ask questions in the newsgroup                               – Use a verification/testing tool, report findings
  – Preferred for most questions over email
                                                            • In-class midterm, final TBD


                   Prof. Aiken CS 295 Lecture 1     3                              Prof. Aiken CS 295 Lecture 1       4




Why Take This Class?                                        This Lecture

• Be a better programmer                                    The course is about improving software quality.

• See the future
  – This is where the field is headed
                                                             Claim: This is central to software engineering.
• Prepare for research in the area

• For the war stories

                   Prof. Aiken CS 295 Lecture 1     5                              Prof. Aiken CS 295 Lecture 1       6
The Ariane Rocket Disaster                           Post Mortem

                                                     • Failure due to unhandled floating point
                                                       exception

                                                     • Cost
                                                       – $100M’s for loss of mission
                                                       – Multi-year setback to the Ariane program




                 Prof. Aiken CS 295 Lecture 1    7                     Prof. Aiken CS 295 Lecture 1   8




East Coast USA                                       East Coast USA: 2003 Blackout




                 Prof. Aiken CS 295 Lecture 1    9                     Prof. Aiken CS 295 Lecture 1   10




Post Mortem                                          Mars Polar Lander

• Local failure rapidly cascaded through grid

• Major contributing cause was unnoticed crash
  of automated alarm systems

• 10M’s of people affected




                 Prof. Aiken CS 295 Lecture 1   11                     Prof. Aiken CS 295 Lecture 1   12
Post Mortem                                                                       The Big Question

• A units problem
   – Caller expected values in inches/feet                                               How do we know the code works?
   – Callee assumed values in meters
   – Essentially, a type error

                                                                                    Look first at how software is developed . . .
• Total loss of $100M mission




                      Prof. Aiken CS 295 Lecture 1                      13                            Prof. Aiken CS 295 Lecture 1                      14




Software Development Today                                                        Typical Scenario (1)

                                                                                                                                      “OK, calm down.
Why do we have                                                                                                                         We’ll slip the
this structure?                                                                                                                        schedule. Try
                                                                                                                                          again.”
                            Decision                                                                        Decision
                            Maker                                                                           Maker


                                                                                                                                                   “It doesn’t
                                                                                                                                                     #$%&
                                                                                                            “I’m                                    compile!”
                                                                                                           done.”
         Programmer                                  Tester                              Programmer                                  Tester




                      Prof. Aiken CS 295 Lecture 1                      15                            Prof. Aiken CS 295 Lecture 1                      16




Typical Scenario (2)                                                              Typical Scenario (3)

                                                       “Now remember,                                                                     “Let’s have a
                                                        we’re all in this                                                                  meeting to
                                                        together. Try                                                                  straighten out the
                                                            again.”                                                                          spec.”
                            Decision                                                                        Decision
                            Maker                                                                           Maker

                                                                                                                                                   “It does the
                                                                    “It doesn’t
                                                                                                                                                   wrong thing
                                                                      install!”
                            “I’m                                                                            “I’m                                    in half the
                           done.”                                                                          done.”                                     tests.”
         Programmer                                  Tester                              Programmer                                  Tester
                                                                                                           “No, half of
                                                                                                            your tests
                                                                                                           are wrong!”
                      Prof. Aiken CS 295 Lecture 1                      17                            Prof. Aiken CS 295 Lecture 1                      18
Typical Scenario (4)                                                                  Typical Scenario (5)


                                                                                                                                              “Oops, the world has
                                                        “Try again, but                                                                       changed. Here’s the
                                                       please hurry up!”                                                                          new spec.”
                            Decision                                                                                Decision
                            Maker                                                                                   Maker

                                                                    “It still fails
                                                                     some tests
                                                                     we agreed                                                                             “Yes, it’s
                            “I’m                                                                                    “I’m
                                                                         on.”                                                                               done!”
                           done.”                                                                                  done.”
         Programmer                                  Tester                                      Programmer                                  Tester




                      Prof. Aiken CS 295 Lecture 1                         19                                 Prof. Aiken CS 295 Lecture 1                     20




Software Development Today                                                            Key Assumptions

                                                                                      •   Independent development and testing
Why do we have
this structure?                                                                       •   Specifications must be explicit
                            Decision                                                  •   Specifications evolve
                            Maker
                                                                                      •   Resources are finite
                                                                                      •   Human organizations need decision makers


         Programmer                                  Tester                           • Examine each of these separately


                      Prof. Aiken CS 295 Lecture 1                         21                                 Prof. Aiken CS 295 Lecture 1                     22




                                                                                      Independent Testing and Development of
Independent Testing and Development
                                                                                      Software

• Testing is basic to every engineering discipline                                    • In what way is software different?
   – Design a drug
                                                                                      • Folklore:
   – Manufacture an airplane
                                                                                          – “Programmers are optimists”
   – Etc.                                                                                 – The implication is that programmers make poor testers

                                                                                          – Economics: “Programming costs more than testing”
• Why?                                                                                    – The implication is that programming is a higher-skill
   – Because our ability to predict how our creations will                                  profession
     behave is imperfect
   – We need to check our work, because we will make                                  • How valid is the folklore, and how much is due to the
     mistakes                                                                           current state of the art in testing?

                      Prof. Aiken CS 295 Lecture 1                         23                                 Prof. Aiken CS 295 Lecture 1                     24
Explicit Specifications                                    Specifications Change

• Software involves multiple people                        • Why?
  – At least a programmer and a user
  – But usually multiple programmers, testers, etc.        • Many software systems are truly “new”
                                                              – Differ from all that went before in some way
• Any team effort requires mutual                             – Initial specification will change as problems are
                                                                discovered and solved
  understanding of the goal
  – A specification
                                                           • The world is changing
  – Otherwise, team members inevitably have different
                                                              – What people want
    goals in mind
                                                              – The components you build on (e.g., the OS version)

                    Prof. Aiken CS 295 Lecture 1      25                          Prof. Aiken CS 295 Lecture 1          26




Software Specifications                                    Finite Resources

• Software specifications are usually                      • Organizations make trade-offs
                                                              – Not all goals can be achieved
  – in prose                                                  – Because resources are finite
  – imprecise
  – out of date                                            • $’s express relative costs among goals
                                                              – Goals that are hard to quantify pose a problem
                                                              – E.g., correctness, completeness
• Current state of specification is not conducive
  to automation                                                “We have 2 months, 5 programmers, and 2 testers. Here is a
                                                                priority list of features. A feature is finished when it passes
  – Not consumable by tools                                     all of the tests for that feature; a programmer does not move
  – Without a spec, there is nothing to check                       on to a new feature until all higher priority features are
                                                                   finished or assigned to other programmers. We start now
                                                                      and ship whatever features are finished in 60 days.”
                    Prof. Aiken CS 295 Lecture 1      27                          Prof. Aiken CS 295 Lecture 1          28




Reality                                                    The Purpose of Testing

• Many proposals for improving software quality            • Two purposes:

• But the world tests                                      • Find bugs
  – > 50% of the cost of software development                 – Find important bugs


• Conclusion                                               • Elucidate the specification
  – Testing is important
  – Take a closer look at testing practice . . .


                    Prof. Aiken CS 295 Lecture 1      29                          Prof. Aiken CS 295 Lecture 1          30
Example                                                       Specifications

• Test case                                                   • Good testers clarify the specification
      Add a child to Mary Brown’s record                        – This is creative, hard work


• Version 1                                                   • There is no hope tools will automate this
   – Check that Ms. Brown’s # of children is one more           – This part will stay hard work

• Version 2
   – Also check Mr. Brown’s # of children
                                                              • An extreme example
                                                                – Warning: cheap laughs
• Version 3
                                                                – “AOL spilled my coffee”
   – Check that no one else’s child counts changed
                          Prof. Aiken CS 295 Lecture 1   31                         Prof. Aiken CS 295 Lecture 1   32




Manual Testing                                                Manual Testing

• Test cases are lists of instructions                        • Manual testing is very widespread
   – “test scripts”
                                                                – Maybe not dominant, but very, very common
• Someone manually executes the script
   – Do each action, step-by-step                             • Why? Because
        •   Click on “login”
        •   Enter username and password                         – Some tests can’t be automated
        •   Click “OK”                                             • Usability testing
        •   …
                                                                – Some tests shouldn’t be automated
   – And manually records results
                                                                   • Not worth the cost

• Low-tech, simple to implement

                          Prof. Aiken CS 295 Lecture 1   33                         Prof. Aiken CS 295 Lecture 1   34




Manual Testing                                                Automated Testing

• Those are the best reasons                                  • Idea:
                                                                – Record manual test
• There are also not-so-good reasons                            – Play back on demand
   –   Not-so-good because innovation could remove them
   –   Testers aren’t skilled enough to handle automation     • This doesn’t work as well as expected
   –   Automation tools are too hard to use                     – E.g., Some tests can’t/shouldn’t be automated
   –   The cost of automating a test is 10X doing a manual
       test



                          Prof. Aiken CS 295 Lecture 1   35                         Prof. Aiken CS 295 Lecture 1   36
Fragility                                                              Breaking Tests

• Test recording is usually very fragile                               • When code evolves, tests break
   – Breaks if environment changes anything
                                                                         – E.g., change the name of a dialog box
   – E.g., location, background color of textbox
                                                                         – Any test that depends on the name of that box
                                                                           breaks
• More generally, automation tools cannot generalize
   – They literally record exactly what happened
   – If anything changes, the test breaks                              • Maintaining tests is a lot of work
                                                                         – Broken tests must be fixed; this is expensive
• A hidden strength of manual testing                                    – Cost is proportional to the number of tests
   – Because people are doing the tests, ability to adapt tests to
                                                                         – Implies that more tests is not necessarily better
     slightly modified situations is built-in

                       Prof. Aiken CS 295 Lecture 1             37                        Prof. Aiken CS 295 Lecture 1   38




Improved Automated Testing                                             Discussion

• Recorded tests are too low level                                     • Automated testing is the state of the art
   – E.g., every test contains the name of the dialog box

• Need to abstract tests                                               • Testers have two jobs
   – Replace dialog box string by variable name X                        – Clarify the specification
   – Variable name X is maintained in one place                          – Find (important) bugs
      • So that when the dialog box name changes, only X needs to be
        updated and all the tests work again
                                                                       • Only the latter is subject to automation
• This is just structured programming
   – Just as hard as any other system design
   – Really, a way of making the specification more concise
                                                                       • Helps explain why there is so much manual
                                                                         testing
                       Prof. Aiken CS 295 Lecture 1             39                        Prof. Aiken CS 295 Lecture 1   40




Summary

• How do we know the code works?
   – Compare the code with a specification
   – We need a specification!!


• Testing is important
   – But very expensive
   – And clearly not sufficient


• We can do better!
   – Many useful techniques
   – The topic of this course

                       Prof. Aiken CS 295 Lecture 1             41