Docstoc

System Implementation Details

Document Sample
System Implementation Details Powered By Docstoc
					System Implementation Details:
Understand, Test, Ignore

           John Lambert / Microsoft
            JLamb@microsoft.com
  (Based on material first presented at STAR East 2005)
Agenda
   Motivation
   Current state of world
   System implementation details
   What to do
   Lessons learned
   Questions


                                    2
Motivation
   Product complexity
   Time to market
   Newer technologies, sooner




                                 3
Current state of world
   Test focuses on customer-facing testing
       Scenario, functional, non-functional…
   Development makes many quiet
    decisions
       Programming language, libraries,
        patterns…



                                                4
Current state of world




                         5
Example: StockQuote.exe MSFT
   Scenarios and functionality are covered
       But customers are still finding issues!
   Errors if two instances are running
       Because cache file opened for exclusive use
   Long wait if HTTP firewall is enabled
       Because we fall back to cache after 5 minutes
   Stress failures when using API
       Security limit imposed on the number of
        concurrent TCP connect attempts

                                                        6
We get surprised…
   Hidden things at all levels of the system

   Development
       “What is this library doing?”
       “Why isn’t the compiler working right?”
   Test
       “Is this even our code?”
       “Why is it failing in the lab?”
                                                  7
… and customers get angry
Customers always blame your product because
  it’s right in front of them

   Even if it’s not “your” code
       ThirdParty.dll doesn’t like IPv6
   Even if the customer didn’t “do” anything
       The administrator turned on the firewall
   Even if you say “exactly” what’s wrong
       Error code 0x8007000E

                                                   8
How can Test help?
   Understand the things used by the system
       And how they work
   Test these things
       Or make an informed choice not to
   Ignore our knowledge about these things
       Play the innocent customer

Product: higher quality, happier customers
Test: fewer surprises, more transferable skills
                                                  9
Define “things”?
   System implementation details are things
    the customer doesn’t usually interact with,
    but that our developers should think about
       Things we could swap without customers noticing
       Things that other teams own


   Like putting on clothes
       You think a little bit about them early in the day
       You mostly ignore them… until there are issues

                                                             10
1. Understand
   What we use and how those things work
   Catalog the system implementation details
       For each place in the system, ask questions
   Learn about how each thing works
       Transferable knowledge between projects

   Danger: infinite zoom
       Don’t lose track of the bigger picture


                                                      11
What are we using?
   Consult the specification, developers,
    tools, bugs, other testers, previous
    versions, your experience, …
   For each item
       What are we using?
       Where are we using it?
       Why are we using it?

                                             12
Places to look at in your system
(see paper)
   Coding
       Programming language, patterns, standards, …
   Compile
       Compiler settings, code generation, build process , …
   Link
       External binary libraries, output executables, …
   Runtime
       Platform, operating system, hosting, …
   Resources
       Network, files, caches, databases, …

   Others depending on your system: configuration,
    setup, patching, upgrading, uninstall, services…
                                                                13
        Example Catalog: StockQuote.exe
What?                   Where?              Why?
C#                      everywhere          familiar with it
Kim’s generic helpers   cache management    less code to write
wsdl.exe                build system        generates proxy code
Microsoft NMake         build system        compiles everything
System.Web.Services     proxy code          handles communication
System.Data.dll         proxy code          service returns DataSet
.NET “Whidbey”          client              needed for generics
Microsoft Windows       client              customer platform
HTTP connection         proxy code          handles communication
Stocks.cache file       program directory   cache stock prices        14
Understand X
   For each X
       How does X work?
       What’s inside of X?
       What do we need to do to test our interaction?
       What issues might we face?
   Your system drives the priorities
       Can learn about things “just-in-time”
       Don’t dive too deeply!


                                                         15
Things to learn about
(see paper)
   Languages
       Macro usage, inlining, header files, type conversions, …
   Error and exception handling
       Checking return values, control flow, recovery, logging, …
   Data types
       Ranges, conversions, pointers, references, strings, …
   Communication
       Naming, firewalls, transient-ness, speed, latency, …
   Displays
       Screens, resolution, DPI, fonts, resizing, …


                                                                     16
           Example: C# Language 1.2
Category            Bits Type     Range/Precision

Signed integral       8 sbyte     –128...127
                     16 short     –32,768...32,767
                     32 int       –2,147,483,648...2,147,483,647
                     64 long      –9,223,372,036,854,775,808...
                                     9,223,372,036,854,775,807
Unsigned integral     8 byte      0...255
                     16 ushort    0...65,535
                     32 uint      0...4,294,967,295
                     64 ulong     0...18,446,744,073,709,551,615
Floating point       32 float     1.5 × 10−45 to 3.4 × 1038, 7-digit precision
                     64 double    5.0 × 10−324 to 1.7 × 10308, 15-digit precision
Decimal             128 Decimal   1.0 × 10−28 to 7.9 × 1028, 28-digit precision     17
2. Test the interaction
   Now that we understand what’s there…
   We need to test the system’s
    interactions
       Decide on investment and approach
       Are we using the right X?
       Are we using X right?
   Use your testing skills (like usual)
       On a different aspect of the system

                                              18
Investment in testing X?
   Take a stand on the “bar”
       What’s the [relative] severity/priority?
            Versus customer scenarios, other X’s


   Take a stand on coverage/approach
       How active will you/others be in testing X?


Make an active, informed, early decision
    (not a random reaction to a late surprise)

                                                      19
Approach for testing X?
   Prevent introduction (great!)
       Tools, education, remove X
   Indirectly (more realistic)
       Do what the customer would do
   Directly (might not get fixed)
       Use fault injection, reflection, test hooks
   Help from others (trust and verify)
       Developers, third-parties, tools, users
   Nothing (scary)
       Is it an informed decision?
                                                      20
The right X?
   Is X the right thing to use?
       Do we have good reasons for using it?
       Does someone understand how it works?
       Are we inside X’s expected usage?
            Our 80% case may be their 20% case
   Can we remove it?
       Cut the features that need it?
       Replace with “better” Y? [testability!]
       Do it ourselves instead? (!)
                                                  21
      Example Catalog: StockQuote.exe
What?                   Where?              Why?
C#                      everywhere          familiar with it
Kim’s generic helpers   cache management    less code to write
wsdl.exe                build system        generates proxy code
Microsoft NMake         build system        compiles everything
System.Web.Services     proxy code          handles communication
System.Data.dll         proxy code          service returns DataSet
.NET “Whidbey”          client              needed for generics
Microsoft Windows       client              customer platform
HTTP connection         proxy code          handles communication
Stocks.cache file       program directory   cache stock prices

                                                                      22
Using X right?
   Does our system honor X’s
       Pre-conditions?
       Post-conditions?
       Error conditions?
       System implementation details? (recursively)
   Avoid re-testing X
       Find out test coverage they already have
       Hit the common mistakes
       Use your testing skills (like usual)
                                                       23
Example: Adding two things
result = a + b;

   What are the data types?
   What are “+” and “=” really doing?
   What happens when things are “too big”?
       What are the boundaries?
 Depends on language and compiler
         (Python vs. C# vs. VB.NET vs. C++ vs. …)


                                                    24
3. Ignore what you know
   We understand and we’re testing…
   Play the “innocent” customer
       Make mistakes (on purpose)
   More tests, more bugs, more ideas



Ignore whether the fix is in “our” code
    (the customer doesn’t care who fixes it!)

                                                25
Ignore local behavior
   What do we globally expect?
       Also called “implicit spec”
   Shortcut keys and menu items
       Consistent with UI guidelines?
   Functionality
       Strikethrough in PowerPoint?
   Naming
       Consistent with other files, API’s, sites?
                                                     26
Ignore requirements
   How does the system respond?
   Violate what X needs
       Unplug the network cable
       Change the resolution/DPI
       Skip the login web page
   Ignore how X affects things afterwards
       Clear your browser cookies
       Open a bunch of instances
       Don’t call the “Close()” API’s

                                             27
Ignore understanding of X
   Can you perform tasks without using
    your knowledge of X?
       Will things make sense to a customer?
       Do error messages reference X?




                                                28
Example
   Add Internet Explorer favorite with “:”




                                              29
Lessons learned (1/2)
   Finding issues is good
       Prevention is better: cut, cut, cut!
   Catalog up-front
       Watch “the list” closely
       Learn as you go
   Problems can be product-wide and dev-wide
       Sometimes easily fixed
       Watch out for easy reintroduction
   Not using things is a whole different problem
       Re-inventing and re-testing the wheel

                                                30
Lessons learned (2/2)
   Innocent user
       Or malicious hacker?
   Check early and often
       Code reviews
       Static analysis
   Understanding reduces runtime surprises
       Debugging, isolation, prevention, automation
   Teach others use your system correctly
       FxCop rules, docs, acceptance tests

                                                       31
Summary
   Understand your system
       Transfer these skills to other projects
   Test coverage is up to you
       Make an informed decision
   Ignore what you’ve learned
       Be an innocent customer



                                                  32
Questions?




JLamb@microsoft.com (PS: we’re hiring!)


                                      33

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:4
posted:7/4/2012
language:
pages:33