Docstoc

Testing Software _ Microsoft

Document Sample
Testing Software _ Microsoft Powered By Docstoc
					Testing Software @ Microsoft

     Matthew Hayward SDET
               Introduction
          J Hayward
• Matthew J. Hayward, SDET
  – 2007 Oregon State University EECS
    Graduate
    • Major: Computer Science
    • Minor: Business Administration
  – Two years contracting in Test
    • 1st Year: Software Test Engineer
                                g
    • 2nd Year: Software Development Engineer in Test
  – Hired by Microsoft in June 2010 as SDET
               Introduction
        Osborne,
• James Osborne SDET
  – 2002, 2004 Oregon State University Graduate
    • Physics MS – 2004
    • Physics BS – 2002
  – SDET since 2005 at Microsoft
             Introduction
• Amy Hickerson




• Jay Hickerson
   First,
   First Are You Really Testing?
THREE THINGS YOU DON’T       THREE THINGS YOU DON’T
WANT TO HEAR AT THE EXPO     WANT TO SAY AT THE EXPO
• “Is it supposed to         • “I’ve never seen it do that
                   that?
  crash/hang like that?”       before ”
                               before…
• “This looks really         • “I swear this worked on
     ggy
  buggy…”                         y            g
                               my box last night…”
• “Every time I try to use   • “This is just a minor
  your product I get           issue, let me skip over it
               results.”
  unexpected results ”                debugger…”
                               in the debugger ”
                  Overview
• What is Software Testing really?
• Software Engineering @ Microsoft from
  Test s
  Test’s Perspective
  – Planning
  – Coding
  – Maintaining
• Day in the Life
• Finding bugs before customers do
• Developing testing skills
What is Software Testing really?
• Misconceptions
   –   Software testing is unnecessary.
   –   Software testers are just automatons.
   –         p
       Developers can test their own code.
   –   Developers should just write bug-free code.
• The Reality
   – Software testing finds bugs before they reach the
     consumer.
   – Software testers develop tools, automation, and
     infrastructure to drive the quality of the product at all times.
   – Developers don’t test their own code very well.
   – Writing bug-free code is difficult.
What is Software Testing really?
           p
• Developers work on discrete p problems in p        ,
                                             products,
  testers work on everything else.
• Testing isn’t about finding bugs but rather about
      tti th    fi d
  getting them fixed.
• It’s just more fun
   – I get paid to watch movies and listen to music!
   – Internet Explorer testers get paid to be on Facebook!
   – Game testers get paid to test games!
• It's about understanding your customers needs
  and making sure the product meets them.
     Why is Testing Important?
             y       g            p
• “I’m already testing as I develop!”
  – Are you tracking bugs?
  – Have you written down test cases?
    Do     h       test l ?
  – D you have a t t plan?
• If you answered “No” to all of those or any of
             you re
  those then you’re not really testing.
• Developers often test just the basic inputs to
  get a basic output.
• Most importantly, the earlier a bug is found
  the easier and cheaper it is to fix.
Software Engineering @ Microsoft
     from Test’s Perspective
     f    T ’ P          i
         g
• Planning
  – Leadership comes up with a feature that it wants.
  – A team is formed in the product to design, build, and test the
    feature
            d i
      • PM designs.
      • Dev builds.
      • Test challenges and verifies.
                     feature,           resources,
  – PM designs the feature identifies resources and communicates
    what will be delivered.
  – Developer plans out how the feature will be implemented, what
    technologies will be needed, and how it will fit into the product.
  – Test writes plan that details how the feature will be verified, what
    tools will be needed, and the resources needed to test the
    feature.
 Software Engineering @ Microsoft
      from Test’s Perspective
      f    T ’ P          i
• Codingg
  – Developer starts breaking the product writing
    product code and simple regression tests.
    Tester dd test           to the database, b ild up
  – T t adds t t cases t th d t b             builds
    infrastructure, and tests the code as it becomes
    available.
  – Once the developer finishes working, the tester
    verifies that the functionality matches the
    specification and that no major bugs were
    introduced.
                                                g
  – Tester writes automation and starts running tests.
 Software Engineering @ Microsoft
      from Test’s Perspective
      f    T ’ P          i
            g
• Maintaining
  – Tester investigates test failures in automation and
    manual runs to determine if there is a product bug.
  – Testers also look at defect reports from both internal
    and external customers by reading forums, watson
    dump reports, and listening to beta testers.
    PM, Developer, and T t t i
  – PM D      l        d Tester triage b     found i t t
                                        bugs f    d in test
    process.
          g
  – As bugs are fixed the Tester verifies the fix and that
    no new bugs have been introduced.
  – More automation is added if necessary.
                             necessary
      • Automation is always necessary.
                                Life
                     Day in the Life*
•    Workout @ 8:30am
•    Step into work @ 10:00am
•    Check e-mail until test reports come out
•                 failures.
     Investigate failures
•    Status update with rest of team
•    File/triage bugs until lunch
•    Lunch
•    Help dev reproduce bugs
•    Write automation for feature
•    Close resolved bugs
•    Go home.

    *Your mileage may vary.
                  Day in the Life - 2
•   Workout @ 7:00am, bike or run to work.
•   S    in        d 8:30
    Step i around 8 30
•   Check blogs, forum postings, comments, various searches, xkcd,
    thedailywtf, etc.
•           One Note Where                    yesterday?”
    Check One-Note – “Where did I leave off yesterday?
•   Any of the following, including combinations:
    – Investigate run failures
        • Infrastructure issues, common code bugs, or tests that broke in new and unusual
          ways
    – Fix broken tests
    – Create repros for devs
        • Extract stand-alone from automation
          Reproduce customer b
        • R     d        t      bugs
    – Help other testers fix their broken, legacy tests
    – Write new tests
        • From test plans
          From bugs th t don’t h
        • F    b                      t   t d tests
                     that d ’t have automated t t
        • From customer issues
    – Planning – Design reviews, new Test Plans, etc.
          The Basics of Bugs
• Costs
  – Planning: Low Cost, easy to fix.
  – Coding: Medium cost, still easy to fix.
  – Beta: Medium to high cost, harder to fix.
               g         ,
  – Release: Highest cost, hardest to fix.
• Priorities
      g
  – High: Must fix soon.
  – Medium: Should fix soon
  – Low: Fix if able.
    Developing Test Strategies
                             ,
• Determine the “Golden Path”, the most basic
  scenario for the feature.
  – This scenario is what MUST work before the
                               finished.
    feature can be declared as finished
• Branch off from the Golden Path. Figure out
  other ways to give the feature inputs to
  produce the desired outputs.
• Figure out the errors that the developer
                                        errors
  accounted for and test around those errors.
• Figure out what inputs change the outputs in
         g       y
  meaningful ways.
    Developing Test Strategies
• Think like the user.
  – Define different user types and think like those
    types.
  – Install as many things as possible to create
    problems
  – Ultimately, you’re the user’s advocate.
• Think maliciously
  – Give bad inputs and see how things go.
• Most importantly, record everything.
      Where Do Bugs Hide?
 Think about boundary conditions When
                       conditions.
  does the output change?
   <insert Example>
 Test between boundaries.
  E i l       Cl          i    t t here.
   Equivalence Classes are important h
 Look for places where the code path
  changes.
   Branch statements especially.
        Developing testing skills

                                       p
• Think like a user. Be malicious, stupid, afraid, or
  as technical as you need to be.
• Look at code with known bugs next to the fix.
• Use code analysis tools to detect defects that you
  may not have noticed.
• Learn to use a debugger and profiling tools to find
           diti             leaks,      th
  race conditions, memory l k or other odd    dd
  behavior.
                    yp             you
• Beta test as many products as y can!
• Best of all: those developer skills are useful as
  well!
           Evolution of Test
• Questions
  – How does an SDET go Beyond Functional,
    Scenario,           Ad Hoc
    Scenario Stress and Ad-Hoc
  – How does one measure the impact of an
    SDET?
    • Bug counts?
    • Number of tests?
            Evolution of Test
• Impact
  – Drive larger initiatives to improve the test
    organization
  – Affect the design of the product during the
    design phase
  – Be the customer advocate
       Co bu e o design
     • Contribute to des g
     • Define standards
     • Define user scenarios
            Evolution of Test
• Further role expansion:
  – Drive improvement in test systems
  – Drive process and culture change
     • Such as: improvement in accountability, more
                              p
       interaction with developers and PMs
  – Expand customer outreach
          gg g
     • Blogging
     • Forums
     • Research
            Testing is Fun!
• We get to find the solutions to the
  problems no one expected we’d have.
                   like,
• We get to think like and interact with our
  customers.
  We t t d        l         ll   test!
• W get to develop as well as t t!
            <EndOfLine>
               How         Software”
• Reference: “How to Break Software by
  James Whittaker
              Testing          Software
• Reference: “Testing Computer Software”
  by Cem Kaner


  Questions?
• Q   ti   ?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:8/24/2011
language:English
pages:24