Docstoc

fault detection techniques - testing and

Document Sample
fault detection techniques - testing and Powered By Docstoc
					Fault Detection Techniques –
 Testing & other approaches

          Chapter 8
                Why testing?
   Let’s look at some well-known software defects
       台灣高鐵售票系統:系統上線之後當機連連,系統無
        法應付突然湧現的購票人潮 (no stress testing)
       台灣彩卷系統:類似的問題
       2000-2005- 巴拿馬國家癌症中心─5個病人接受過量的
        迦瑪射線照射死亡。15人引發了嚴重的併發症
       2003 ─軟體造成美國東北部及加拿大停電。5000萬人
        受影響,3人喪生
       2000 美國海軍飛機墜落,4人喪生 (控制軟體問題)
       1997 韓國空難,225人喪生 (雷達控制軟體問題)
       1995 美國航空在哥倫比亞撞山159人喪生(導航軟體
        問題)

                          2
           Statistics
 2002- 美國國家標準局報告─軟體品質每年
  造成595億美元 (0.6% GDP)
 國內又如何?
     台北縣政府校務行政系統案例
     台師大林口校區學務系統案例




               3
          Quality Control and
          Quality Assurance
 Software is a product.
 Good quality product requires 品質控管
  quality control (QC) and quality assurance
  (QA) 品質確保




                      4
Manufacturing in other discipline
   在製造業的世界裡面
       產品品質(quality)是很重要的一個因素
       當你投入非常多成本與原物料時你永遠希望你生產出
        來100個產品有100個都是可以賣的
   QC (Quality Control) 品質控管
       製造過程從設計到生產可能有數十道到幾百道程序
       每一道程序都可能影響到後面的品質
       如何透過製造流程的改善(process improvement)來提
        高良率,一直都是製造工業的重要課題


                       5
        How to improve quality?
   在製造過程中如果能夠能夠使用機器,最
    好的改善品質的方式就是自動化。機器不
    容易犯錯而且可以不斷地重複單調無聊的
    工作。機器會出錯的時候通常是由於製造
    機器的磨損,必須重新校正。
       Ex. 日本的步進馬達精確度超高的秘密。
   Question
       當某部分的工作不能由機器來做的時候,製造
        過程如何做到品質控管?

                   6
Assembly line in manufacturing




              7
        生產線
 每個生產線員工只做一件夠簡單到不容易
  出錯的工作
 生產線員工不需要夠聰明
 生產線將複雜的產品製造切割成許多小步
  驟可以在短時間以及最少的技術內可以完
  成
 發現產品的問題並不難.



          8
  Again, let’s review the engineering
 process of other engineering fields.
1.   Idea (概念)
2.   Marketing analysis (Requirement
     Analysis) (市場分析)
3.   Analysis and Design (分析與設計)
4.   Manufacturing (QC) (製造)
5.   Testing (QA)(測試)
6.   Release product (產品出廠)


                     9
         Software engineering
1.   Idea (概念)
2.   Requirement analysis and specification (100%
     design) (需求分析與規格)
3.   Design and analysis (分析與設計)(100%
     design, QC here?)
4.   Implementation (程式實做)
     (95% design?, QC here?)
5.   Manufacturing (製造)(compilation – no cost
     manufacturing)
6.   Testing(測試)

                         10
                     Question?
   Is coding a design process or a manufacturing process
   Is Code a design or a product?
   Again, the analog breaks

More questions
 If coding is not a design process, we should be able to
  hire many 生產線女工 to control our quality
 If coding is a design process, bad news, errors are
  inevitable because programmers are always doing
  complicated design jobs, not simple and easy job.


                             11
                   Facts
   Now, you should know why software has
    so many bugs or notorious for being
    flawed




                      12
    The QC story
產
品
缺
陷
率



              software




                         其他工程製造

                   QC 改進時程



         13
             A Joke
有天晚上我參加一個老同學的喜宴,
                        頓時我成了眾矢之的,
同桌的人有律師、會計師,
                        這些電腦使用者的抱怨全部集中到我身上
只有我一個是電腦業界的人。
                        來。
為了引起話題,我說我羨慕其他行業
  的人,像是律師、會計師等,在        其中一位仁兄還舉了個有趣的例子,
  學校學的那一套終生都受用,每
  年就算有新法條也不會改太多。        他說:
不像我們電腦界的人,每天都在K新        「如果傢俱業和電腦業一樣,世界會變成
  的技術手冊,新產品新技術隨時        什麼樣子?」以下是他講的故事:
  大翻新,改朝換代的速度讓人措
  手不及,所以學電腦的人看起來
  都比較蒼老,因為太辛苦了。         如果傢俱業跟電腦業一樣,比如說我到傢
                        俱店買了一張桌子,搬回家往地板上一放,
我話剛說完,一位會計師馬上回我一        啪啦一聲桌子就塌了。這時候我不會生氣、
  句:「你錯了,其實我們最羨慕        不會罵人,我會先自己檢查一下出了什麼
  你們電腦界的人,因為沒有任何        錯。
  行業的消費者像你們電腦界的消
  費者一樣好欺負。」

                   14
我會先檢討自己,
是不是我做錯了什麼,
                    於是我恍然大悟原來是自己的錯,
或是我對桌子的使用不夠熟悉。      我就再去買一張「正式版」的桌子。

於是我會去買書來看           回家一擺還是啪啦又塌了!
(書名可能是《快快樂樂學修桌子》、
《21天學會修桌子》、《修桌子技巧   修了半天還是有問題,
  與實例》、
                    再請傢俱行的技術人員來做仔細檢查,
或是《修桌子的聖經》)。
                    最後終於發現問題的所在──
要是書看得不太懂,           「我家的地板和桌子不相容」,
我會再花錢去報名上修桌子的課程。    又是我自己的錯,
                    於是我得趕快幫家裡的地板升級......。
學完之後還是修不好,
我會請其他比較懂得修桌子的朋友來    等一切都忙完了,
  幫忙。
                    桌子可以使用了,
最後沒有辦法,             我趴在桌上寫字,
終於我打電話給原先的傢俱行,      心裡充滿了成就感,
(可能還要購買《技術支援方案》),   我很得意地跟網友分享我修理桌子的經驗,
結果他們跟我說:            並暗自慶幸自己在科技的潮流上沒有落
「唉呀!你買到的是搶鮮版啦!      伍......。
本來就應該有問題的」。
                    15
          Software Quality
 如同前面所說的,由於軟體產業的生產工
  具只有人,,人注定會犯錯,很多人還會
  犯很多的錯
 不要相信 programmers
 This is why testing is so important in
  matured software industry.



                    16
一座橋蓋好了,需要大量的測試嗎?




       17
How do you test a DVD player




             18
How do you test a software




            19
How do you test an Anti-Virus engine
                ?




                 20
How do you test an online-game which could
   have 10000 online at the same time?




                    21
    The complexity of software testing
 一般而言軟體有複雜的介面 (including user
  interface, network interface, file interfaces,
  etc. )
 一般而言軟體測試必須考慮更多的情境
     Correct execution paths(正確的路徑)
     Incorrect execution paths (不正確的路徑)
   有許多應用軟體的測試其實是一門困難的
    學問理論,技術,實做,程式能力的要求
    都不低
                       22
    SDLC (Software Development Life
                Cycle)
   需求分析 Requirement analysis (marketing research)
   軟體規格寫作 Software function/performance specification
   分析與設計 Analysis and Design (QA 在這時候就要參與專案)
        分析,選定PL,選定 platform, database, network protocols.
        定義軟體功能模組,模組之間的關連,合作,介面
   程式實做與單元測試 Coding and unit testing (by programmers)
   整合測試 Integration testing (by programmers/QAs)
   Alpha testing (by QAs)
        Most software functions and features are basically completed
        All functions are tested, no functions will be added beyond this point
        Serious flaws (high severity) are solved and addressed (show stopper)
   Beta testing (by Beta users)
        sub-serious bugs are all fixed
        Test plan has been completely executed
        Bug discovering rate is lower than bug fixing rate

   軟體釋出 Release
        Bug discovering rate is lower than bug fixing rate for a long period of time.
        The version after fixing bugs has been regressively tested (regression test)
        Quality is formally proved by QA team
        All documents are ready




                                                          23
    Common software defects
 70% occurs in design and hard to correct
 Software specifications are not precise
 Software is complicated
 Coding errors (20%)
 Function changes which affects other
  components
 3rd party software is flawed


                     24
 Early detection of errors
                                                                          Error found
                                                                            Mar 13
Work (person-days)
                                                Error found
                     Error occurs                 Feb 13
                        Jan 13
                                                        10                           20



                                                                                        Time
                     Work proceeds         10 person-days of work     If error found this late,
                     at (say) 10 person-   has been done assuming     20 person-days must
                     days per month        the error is not there.    be redone
                                           Now this must be redone.


                                                 25
What to test for a QA (要測什麼?)
   完整性 (completeness)
       軟體是否具備軟體規格書,設計文件中所描述的功能
        與性能
   正確性 (correctness)
   可靠性 (reliability)
   相容性 (compatibility)
   效率(efficiency)
   可使用性 (usability)
   可攜性 (portability)
   可變比例性 (scalability)
   易測性 (testability)
                      26
               迷思
   軟體有問題是測試人員的錯
       軟體測試只是一種有效的提高軟體品質手段,
        但無法百分之百解決軟體品質的問題
   軟體測試技術要求不高,比程式設計容易
     兩種人無法類比,程式設計能力好的人,可能
      可以更勝任軟體測試。
     自動化測試常需要程式設計高手




                 27
               就業市場概觀
   美國的軟體產業現況
     Big software company has their testing team
     Small software companies which do not have
      resources to test can outsource to 3rd party
      testing company
   台灣的軟體產業現況
     Big software company like 趨勢科技Trend
      Micro has QA team
     Many software companies use programmers
      as testers – a sad fact!
                         28
   Company Organization

Marketings                         Developers




             QAs (testing teams)
                      29
             Management Hierarchy

                                                           Director of
            Director of QA                                development




                  Test automation   Test facilities    Developer    Developer
Test manager                                           manager      manager
                  Team manager        manager




                 Test automation    Test database                   programmers
  testers                                             programmers
                  programmers       programmers
                                        30
Important reasons to make QA team and
  developer team equal and separated
   只有極少數處女座的 programmers 才會有品質(完美主
    義)的概念
   叫 programmer 寫完程式,自己找bugs 就如同叫法官判
    完案之後,承認自己的判案是錯誤的
       如果他真的找出很多bug ,他決不敢聲張與邀功,因為這代表自
        己程式寫太爛
       找的bug越多,表示待完成的工作越多,沒有人會自己找碴,讓自
        己無法喘息
   上述「人」的因素使得
       QA 必須在軟體品質上與 programmer 站在對立面
       QA 的升遷管道必須是抓到越多 bug,功勞越高,QA才會努力找
        碴
       QA 不能歸 developer team 管轄,這就像是司法不能在行政部門
        底下是一樣的

                         31
                    Marketing




                        恐怖平衡

QA (testers)                    Developer




               32
           軟體測試V模型                  回歸測試

軟體功能需求                            驗收測試
               以此作為驗收測試依據

  軟體規格說明書                      系統測試
              以此作為系統測試依據

                   以此作為整
    分析與設計文件                  整合測試
                   合測試依據


         軟體模組設計書           單元測試



                   程式實做

                     33
            Software Defects
 Software defects represent the
  undesirable aspects of a software's quality
 What is undesirable?


 A software defect is software behaviors
 that do not conform to specification

   Software specs define the correctness of
    the program
    Software defect classification
   Specs error
        A specs that is wrong in the beginning
        e.g.,
         in specs, a teacher can only teach one course
         in practice, two teachers can teach one course and
         share the credits/fee
   design error
        An error that is caused by wrong design
        e.g.,
          in design, UDP is used to transmit the holding (owning) of a treasure
          in practice, UDP can lose, program behaviors are incorrect
   coding error
        An error caused by programmer – the error type you are mostly familiar with

   performance inadequacy – failed to meet the nonfunctional requirement
   Scalability inadequacy – failed to be able to serve large amount of users
    and transactions
   memory management error – failed to meet the memory resource
    requirements
    Severity of Software Defects
 show stopper (severity 1)
 unusable software (severity 1)
 Microsoft ranks the bug severity from 1-30
 killing people, waste a lot of money.
        How to find software defect
   Dynamic
       testing
            testing, teamwork in a ACM survey are the three disciplines
             that should be trained in CS
            rely on dynamic execution of the program
            a program must be built first before testing
            The major way to test performance or scalability
   Static
       code review (survey shows 70% sever bugs can be
        discovered in this process)
       static analyzer – verify software defects without
        running a program but by analyzing program or
        models
            software defects detection
             techniques classification
   optimistic (樂觀的)
       testing
       using optimistic approach can only show how good
        your code may be. It can not show the absence of
        faults
   pessimistic (悲觀的)
       static analyzer, code review
       mostly want to show the absence of faults
       may report spurious results
            compiler complains about usage of noninitialized variable
      Optimistic v.s. Pessimistic
   Optimistic
                  Comments
   In principle, testing cannot show the absence of
    faults, so it is impossible to prove a program is
    correct by testing
   E.g., you can not prove a sorting algorithm is
    correct totally by testing. To prove a sorting
    program is correct exhaustively, you need to test
    N! cases. If N = 100, it intractable.
   Using exhaustive testing to obtain correctness of
    a program is often infeasible for common
    problem.
   Question: How Sorting algorithms in Algorithm
    Course are proved correct?
               Comments
 Proof is difficult. You need to show that
  your solution works for all cases. Even the
  cases are N! and N can be arbitrary
 Testing only works for a comparatively
  small set of input domain under
  reasonable cost and resources.
    Testing in Software industry
 Testing is still the most effective and
  mostly used fault detection techniques
  although testing in theory can not show
  the absence of faults.
 Testing only show the presence of faults
  not the absence of faults
 Most errors can still be found in extensive
  testing if you are willing to invest
  resources in it.
    Testing in software development
   Testing plan in the beginning of the project
   unit testing (programmer)
    when programmers complete their functions, procedures, modules, or
    components, they should perform unit testing. Often proceeded as white-
    box testing
   Integration testing (programmer)
    The process for development team to integrate their modules, components,
    classes, library into a complete executable system (a complete build is
    produced for later stages).
   Alpha Testing (QA- Testers)
    The process for internal testers to test a complete build. It is often
    proceeded as blackbox testing. During this stages, assertion is often turned
    on.
   Beta Testing (Power Users)
    A build is given to power users to test. During this stages, assertion is often
    turned off.
   Released
            Unit Testing
    (component testing in text book)
   Testing applied on software units. The
    typical types of system units are:
     classes
     components
     modules
     library
     functions
     subsystems
   Unit testing is programmer’s responsibility
         How to determine a unit’s
              correctness?
   Since software units are not complete systems,
    so they do not have specification to describe
    their correctness, how can we determine the unit
    behaviors under tested is correct?

     ANS: The correctness of units are specified
    in design not in specs. For example,
    correctness of units are specified in module
    interfaces, class interfaces, class
    descriptions, and so on.
              Unit testing
 Just like the module testing described
  before.
 The purpose is to produce test cases to
  feed the interface of software units and
  exercise the software units
 In principle
    The testing criteria for unit testing
 In principle, exhaustive testing of unit interface is
  also impossible.
  e.g.,
  void p(short int x, short int y) {
     ……………….
  }
 You need to test 65536* 65536 cases to make
  sure your unit behavior correctly and each test
  run needed to be judged by human (with eyes)
 So, how much we need to test to (at least)
  guarantee some degree of confidence.?
         Criteria of confidence
   Test Coverage (from weak to strong)
     statement coverage – each statement is at
      least executed once.
     branch coverage – each branch should be
      exercised at least once
     path coverage – each path should be at least
      exercised at least once
               1


                        while bottom < = top
bottom > top
               2



               3   if (elemArray [mid] == key



        8                  4
                                    (if (elemArray [mid]< key

                   5                6
        9

                           7               Binary search flow
                                                 graph
        The weakest criteria
 Statement coverage
  1 test case to run 1 2 3 8 9
  1 test case to run 1 2 4 6 7 2
  1 test case to run 1 2 4 5 7 2 8
 You only need 3 test cases to have each
  statement at least executed once.
                 Branch coverage
 each branch can produce two choices, every choice combinations
  should all be exercised
 there are 3 branches, so at most 2 * 2 * 2 branching choices should
  be exercised if loop is not considered.
 So at least you need to find test cases to meet this criteria
 1289
 12389
 123457289
 1234572389
 123467289
 1234672389

   In the example, there are less than 8 branching choices because it
    is not a complete tree
                          Path Coverage
    The highest coverage
    Try to find test cases which can cover all the paths
    equal to exhaustive testing if you want to cover them all, which is impossible
    You need to find infinite test cases which have finite (if the program must stop) or
     infinite length (if the program may run forever)
     1 2 8 9 (finite length)
    1 2 3 8 9 (finite length)
    123457289
    12345723457289
    123457234672389
    1 2 (3 4 5 7 2) * 8 9 ( finite length but very long sequence)
    1 2 (3 4 5 7 2 | 3 4 6 7 2) * 8 9 ( finite length with permutations)
    1 2 (3 4 5 7 2) ω 8 9
    ………
    ………
    where * is finite iteration in automata theory
    where ω is infinite iteration in automata theory
            Path coverage
 in a program which can run for ever, full
  path testing is equal to exhaustive testing
 it is infeasible in practice
 So, we must discover more practical path
  coverage criteria.
              Independent paths
               (a weaker criteria)
   1, 2, 3, 8, 9
   1, 2, 3, 4, 6, 7, 2
   1, 2, 3, 4, 5, 7, 2
   1, 2, 3, 4, 6, 7, 2, 8, 9
   each test case should explore statements that
    are not explored by the previous test cases to
    reduce redundant test cases
   A dynamic program analyser may be used to
    check that paths have been executed
   In practice, this is often difficult too
      Test coverage in real world
   Since different coverage criteria involve different
    degree of cost and time
   Often statement coverage is the lower bound.
   Branching coverage or path coverage can be
    applied to important components
   Test coverage testing is widely applied in
    software industry. Although coverage rate mean
    nothing to program correctness. It is only an
    index to show if testing has been done
    adequately. Most people accept that index is
    closely related to software quality since test
    cases are executed.
               Test coverage
   IMPORTANT
    If you have 100% statement coverage, it does
       not mean your program is correct.
   Sorting algorithm is an example
     you can easy derive only few test cases to
      have 100% coverage of your program
     But to prove your program is correct is totally
      a different scale of the program
    How to proceed test coverage
              testing?
 You need to have source code to know the
  coverage.
 You need to have tool to display the
  coverage rate
 Each test case, you need to have
  someone (or in very few cases, tools) to
  judge the correctness of program
  behaviors.
                 White box testing

Test cases
                                                                   output




 You test an equipment by observing the internal behaviors of the equipment
 also known as structural     testing (text book)
               Test coverage
   Test coverage must be done with white-box
    testing. The tester must have source code for
    testing
   Effective test cases must be derived from source
    code, which require programming skills.
   So, test coverage can be done by programmers
    or 3rd party testing company.
   In USA, Europe, there are many 3rd party
    testing companies.
   In Taiwan, never heard of it. Maybe you should
    start a new one.
          Test coverage tools
   Aonix Validator/Req   •Java Test Coverage
   BullseyeCoverage      •JavaCov
   Clover                •Koalog Code Coverage
   CodeTEST              •LDRA Testbed
                          •Logiscope
   CTC++
                          •Microsoft Visual Test
   Dynamic Code          •Rational PureCoverage
    Coverage              •Rational Test RealTime Coverage
   GCT                   •TCMON
   Glass JAR Toolkit     •TCA
   Hindsight/TCA         •TCAT C/C++
                          •TCAT for Java
   Hindsight/TPA
                          •TCAT-PATH
   Insure++              •TestCenter
             Black Box Testing


Test Cases
                                 Outputs
            Black Box Testing
 A testing process proceed without knowing
  the internal behaviors of the system.
 Often done by QA (Testers)
 QA only interact with system by
     feeding test cases by user interfaces
     feeding test cases by files
     feeding test cases by networking traffic
     feeding test cases by consoles
BlackBox testing and Coverage
 In principle, Black Box testing can still
  combine with test coverage tools, if source
  code is available.
 Test coverage tools can be used to show
  the index of “how good the test cases are
  derived by the QAs.”
           BlackBox Testing
 In practice, QAs use blackbox testing to
  assure Software quality after unit testing
  and integration testing. They take over
  from developers
 They focus on checking

     whether the program is conformed to
    specs.
     A simple 管理哲學 of testing
   It is a common knowledge that testing after
    alpha testing should not be done by developers.
       developers tends to feed good test cases only
       developers are not happy to find more bugs (find
        more bugs, more works)
       find more bugs may not get promoted. On the
        contrary, more bugs prove his coding quality is low.
   So, to obtain high quality software, a simple
    management rule
       have testers to test the program, not programmers
       find more bugs mean promotion
       just like 三權分立
               Marketing




                   恐怖平衡

QA (testers)               Developer
BlackBox (alpha testing) Testing
 QAs test program by specs.
 In common cases, specs may describe
     correct behaviors
     incorrect behaviors that should be avoid.
   QA’s job goal
     to show the presence of software errors as
      much as possible
     test the software from a user’s perspective
     test all positive, negative, and boundary cases
      Black-box testing
                              Inputs causing
                              anomalous
  Input test data        I    behaviour
                         e




                System




                              Outputs which reveal
                              the presence of
Output test results      Oe   defects
       Equivalence partitioning
          (partition testing)
 Input data and output results often fall into
  different classes where all members of a
  class are related
 Each of these classes is an equivalence
  partition where the program behaves in an
  equivalent way for each class member
 Test cases should be chosen from each
  partition
Equivalence partitioning

      Invalid inputs       Valid inputs




                 System




                 Outputs
        Equivalence partitioning with
              boundary cases
   Partition system inputs and outputs into
    ‘equivalence sets’
       If input is a 5-digit integer between 10,000
        and 99,999,
        equivalence partitions are <10,000, 10,000-99,
        999 and > 10, 000
   Choose test cases at the boundary of
    these
    sets
       00000, 09999, 10000, 99999, 10001
         Equivalence partitions
                        3                            11
                            4          7           10



          Less than 4           Between 4 and 10      More than 10

     Number of input values

                 9999                                 100000
                   10000             50000         99999



  Less than 10000           Between 10000 and 99999       More than 99999


Input values
          Automation of testing
    (including blackbox + whitebox)
 If software quality is important to you, of
  course, you want a higher criteria to be
  met.
 Unfortunately, higher criteria means higher
  test cases, more time, more cost.
 If your test cases outcome must to be
  judged by humans, higher criteria means
  much higher cost and time
              Testing automation
   In many applications or functionalities, testing
    can be automated at different degrees. In
    contrast, many functionalities and applications
    are not amenable to automatic testing, where
    humans are needed
   examples that is difficult to automate testing
       testing WORD.
   examples that is easy to automate
       testing a sorting algorithm – you can easily write a
        program to check if the output is listed in ascending or
        descending order
                 Test oracles
   If test cases can be derived automatically and
    then these test cases can be executed
    automatically and then report the error – What a
    perfect world ! We can fire all the testers.
   In practice, this is only a dream
   In many applications, Effective test cases can
    only be derived by human
   In many applications, Executions requires
    human to monitor and validate if program
    behaviors are conformed to specs.
                 Test Oracles
   Oracle – In ancient
    Greece, an oracle was a
    priest or priestess who
    made statements about
    future events or about the
    truths

   A test oracle is a program
    that can determine if the
    program behaviors or
    program input/output are
    conformed to specs.
         Testing automation
 For testing automation, test oracle must be
  presented to determine the conformity
  between specs and programs.
 In some applications, test oracles are easy
  to derive
 In most applications, test oracles are
  difficult to implement or in theory,
  impossible to derive.
               Testing automation
   In a big map, network games, a test oracle can be
       character’s movements are controlled by a test program. The
        test program keep feeding the mouse events and keyboard
        events randomly
       A simple test oracle is to see if the program crash. This test
        oracle is easy and close to none.
       A more complicated test oracle is to check if the virtual money
        given to the characters are controlled under a fixed amount, C
        How can you design this test oracles?
       complicated test oracle often require program instrumentation –
        program code that inserted in the program only for monitoring
        purpose.

   The difficulty of test oracles is depending on what kind of
    system properties you want to check.
         Integration testing
 Tests complete systems or subsystems
  composed of integrated components
 Integration testing should be black-box
  testing with tests derived from the
  specification
 Main difficulty is localising errors
 Incremental integration testing reduces
  this problem
    Incremental integration testing
                                                  A                   T1
                                             T1
                         A
                    T1                                                T2
A                                                 B
                                             T2
                    T2   B                                            T3
                                             T3
B                                                 C
                    T3                                                T4
                         C
                                             T4
                                                  D                   T5


    Test sequence            Test sequence            Test sequence
          1                         2                        3
        Approaches to integration
                testing
   Top-down testing
       Start with high-level system and integrate
        from the top-down replacing individual
        components by stubs where appropriate
   Bottom-up testing
       Integrate individual components in levels until
        the complete system is created
   In practice, most integration involves a
    combination of these strategies
    The key to incremental testing
 emulate the functions you will call
 emulate the functions that call you.
                Top-down testing
                      Testing
      Level 1                         Level 1                    . ..
                     sequence



                      Level 2   Level 2     Le vel 2   Level 2

Le vel 2
 stubs


                Le vel 3
                 stubs
          Top-down testing
 top down testing is used mostly by
  personal programming.
 Test cases are full test cases derived one
  by one from specs.
 it is a hot programming paradiagm called
  test driven programming.
main() {
  loadfile(myarray) ; // load an array from file
  sorting(myarray) ; // sort an array
  input a ;           // get input from user
  ret = binseary(myarray,a) ;
  if (ret) dosomethingA();
  else dosomethingB();
}
   An loadfile stubs

    loadfile(int array[]) {
       array[1] = 10 ;
       array[2] = 1 ;
       array[3] = 55 ;
       ……
     }
                Bottom-up testing

 Test
drivers

                                                               Testing
      Level N     Level N   Le vel N    Level N     Level N
                                                              sequence




 Test
drivers
           Level N–1        Level N–1        Level N–1
         Bottom up Testing
 Bottom up testing are suitable for
  teamwork project which proceeds
  concurrently and parallel.
 Test drivers are derived based on
  module’s interfaces, parameters, etc. They
  are typically not full test cases derived
  from specs.
     Comparisons of top-down and
         bottom up testing
   top-down test cases are derived from specs. So,
    many errors can be discovered earlier.
   top down testing is often less efficient than
    bottom up regarding teamwork and parallel
    cooperation.
   bottom up testing drawback – test cases are
    typically not derived from specs. If design is
    wrong, it can not be discovered earlier.
              Testing approaches
   Architectural validation
       Top-down integration testing is better at discovering
        errors in the system architecture
   System demonstration
       Top-down integration testing allows a limited
        demonstration at an early stage in the development
   Test implementation
       Often easier with bottom-up integration testing
   Test observation
       Problems with both approaches. Extra code may be
        required to observe tests
           Interface testing
 Takes place when modules or sub-
  systems are integrated to create larger
  systems
 Objectives are to detect faults due to
  interface errors or invalid assumptions
  about interfaces
 Particularly important for object-oriented
  development as objects are defined by
  their interfaces
Interface testing
      Test
     cases




 A             B


         C
                 Interfaces types
   Parameter interfaces
       Data passed from one procedure to another
   Shared memory interfaces
       Block of memory is shared between procedures
   Procedural interfaces
       Sub-system encapsulates a set of procedures to be
        called by other sub-systems
   Message passing interfaces
       Sub-systems request services from other sub-
        systems
                 Interface errors
   Interface misuse
       A calling component calls another component and
        makes an error in its use of its interface e.g.
        parameters in the wrong order
   Interface misunderstanding
       A calling component embeds assumptions about the
        behaviour of the called component which are
        incorrect
   Timing errors
       The called and the calling component operate at
        different speeds and out-of-date information is
        accessed
      Interface testing guidelines
   Design tests so that parameters to a called
    procedure are at the extreme ends of their
    ranges
   Always test pointer parameters with null pointers
   Design tests which cause the component to fail
   Use stress testing in message passing systems
   In shared memory systems, vary the order in
    which components are activated
                Stress testing
   Exercises the system beyond its maximum
    design load. Stressing the system often causes
    defects to come to light
   Stressing the system test failure behaviour..
    Systems should not fail catastrophically. Stress
    testing checks for unacceptable loss of service
    or data
   Particularly relevant to distributed systems
    which can exhibit severe degradation as a
    network becomes overloaded
            Stress testing
 require considerable cost and efforts.
 often require you to implement a system to
  test the system.
      Object-oriented testing
 The components to be tested are object
  classes that are instantiated as objects
 Larger grain than individual functions so
  approaches to white-box testing have to
  be extended
 No obvious ‘top’ to the system for top-
  down integration and testing
            Testing levels
 Testing operations associated with objects
 Testing object classes
 Testing clusters of cooperating objects
 Testing the complete OO system
           Object class testing
   Complete test coverage of a class involves
     Testing all operations associated with an
      object
     Setting and interrogating all object attributes
     Exercising the object in all possible states

   Inheritance makes it more difficult to
    design object class tests as the
    information to be tested is not localised
Weather station object interface

   WeatherStation
                           Test cases are needed
identifier                  for all operations
reportWeather ()
calibrate (instruments)
                           Use a state model to
test ()
startup (instruments)
                            identify state transitions
shutdown (instruments)      for testing
                           Examples of testing
                            sequences
                                 Shutdown  Waiting  Shutdown
                                 Waiting  Calibrating  Testing 
                                  Transmitting  Waiting
                                 Waiting  Collecting  Waiting 
          Object integration
 Levels of integration are less distinct in
  object-oriented systems
 Cluster testing is concerned with
  integrating and testing clusters of
  cooperating objects
 Identify clusters using knowledge of the
  operation of objects and the system
  features that are implemented by these
  clusters
    Approaches to cluster testing
   Use-case or scenario testing
     Testing is based on a user interactions with
      the system
     Has the advantage that it tests system
      features as experienced by users
   Thread testing
       Tests the systems response to events as
        processing threads through the system
   Object interaction testing
       Tests sequences of object interactions that
      Scenario-based testing
 Identify scenarios from use-cases and
  supplement these with interaction
  diagrams that show the objects involved in
  the scenario
 Consider the scenario in the weather
  station system where a report is generated
Collect weather data
          Weather station testing
   Thread of methods executed
       CommsController:request                      
        WeatherStation:report                        
        WeatherData:summarise
   Inputs and outputs
       Input of report request with associated
        acknowledge and a final output of a report
       Can be tested by creating raw data and
        ensuring that it is summarised properly
       Use   the   same   raw    data   to   test   the
        Testing workbenches
 Testing is an expensive process phase.
  Testing workbenches provide a range of
  tools to reduce the time required and total
  testing costs
 Most testing workbenches are open
  systems because testing needs are
  organisation-specific
 Difficult to integrate with closed design and
  analysis workbenches
           A testing workbench
                             Test data
                                                     Specification
                             generator



  Source        Test                                    Oracle
   code        manager       Test data



Dynamic         Program        Test                     Test
analyser      being tested    results                predictions




Execution                                   File
               Simulator
 report                                  comparator



                                          Report                 Test results
                                         generator                 report
    Tetsing workbench adaptation
 Scripts may be developed for user
  interface simulators and patterns for test
  data generators
 Test outputs may have to be prepared
  manually for comparison
 Special-purpose file comparators may be
  developed
Three Well Known Approaches
   Testing – execution on product or program
    (implementation) to detect faults (optimistic)
   Simulation – execution on artifacts (models)
    which is simpler representation of the original
    complexity (optimistic)
   Verification – explore all the state space of
    models which is a simpler representation of
    original complexity (pessmistic)
                   Models
   So, we want models can be
     Simulated!
     Verified !

 They can provide more clues of how our
  design is wrong
 By the way, what is design?
The foundation behind model
 simulation and verification
                      X            Y
                   (X > -1)   =>   (Y>-1)



                     Preorder
                     (implements)

                         
  deadlock exist                   deadlock exists
Fault Detection Tools in Industry

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:16
posted:9/1/2012
language:Unknown
pages:114