White Paper
Implementing a Risk-Based approach to Static Testing
Last Updated: 17th August, 2007
Introduction
The subject of Static Testing (Product Reviews in this context) has been widely documented, and most testing professionals are able to list the standard techniques that are involved, and the standard benefits that can be achieved. But in an environment of looming deadlines and tight budgets, it can be difficult to find the resources to perform what can initially seem like arduous and bureaucratic formal reviews, and the benefits of those reviews can be lost. This paper suggests how taking a risk-based approach to static testing can help focus effort and finite resources to get the most value from reviews, and describes how the process can be implemented in a project or organization. For the purpose of this paper, the terms Static Testing and Product Review will be used primarily in reference to document reviews – however many of the suggestions can still be applied to code reviews.
Requires lots of specialist training
While it is true that becoming a fully-fledged Inspection Moderator will involve formal training, this is not so for all review techniques. In fact, anyone can learn the basics of a general review process in a very short time, and through a little coaching and hands-on experience can go on to become proficient in planning and executing efficient reviews.
Benefits
Uses standard techniques that are recognized and
proven across the industry
Early identification of faults close to point of error – find
the big problems sooner
Lowest cost time to fix faults Early warning of potential future problems Encourages
A Few Misconceptions
We shall begin by dispelling a few of the more common misconceptions about reviews and static testing: Perception Reality Almost everyone can (and should!) be involved in static testing where appropriate. Planning the most effective review means getting the right mix of technical specialists, subject matter experts, quality assurance personnel, and sometimes just people who have a keen eye for detail. If a review is planned, managed and executed properly, it can be an engaging and motivating experience for both author and reviewer. The author emerges from the review with a product of higher quality, and the reviewer feels achievement that they have directly contributed to the production of a quality product. Just not true! Static Testing can be performed on any product or deliverable that is being produced either as an end product in its own right, or as a source for a subsequent product.
group understanding
working,
communication
and
Metrics can influence future process improvement
Just something that testers do
Implementing Static Testing in a Large Project or Organization
This first section provides practical advice on implementating static testing within organizations based on real-life scenarios. Experience has shown that the best way to ensure a successful implementation is to create a small dedicated implementation team with specific roles:
Process Champion
The implementation of Static Testing represents an investment in time and resource effort and as such will require senior management backing if it is to be successful – AppLabs experience is that without this level of support, projects will almost always quote lack of time and effort as the reason for not implementing this.
Time-consuming, dull and boring
Only applies to lengthy technical documents, or code
Trainer(s)
In order to maximize the return on this investment of time and resource, it is important that the reviews are conducted
AppLabs.com
AppUK_WhitePaper_Static_Testing_Paper_1v00 Page 2 © 2007 AppLabs
effectively and efficiently and past experience has proven the value in providing formal training for those undertaking reviews. AppLabs has found that training can be effectively split into training for the Leaders/Facilitators (of which only a relatively small number may be required) and training for the technical reviewers which will probably include most of the testing community. As a rough guide, AppLabs has found that the ratio of Lead/Facilitator training is of the order of 10-20% of the number trained for technical reviewing.
experienced at first hand how this approach can provide a successful basis for further implementation(s) AppLabs has recognized some essential factors that will contribute to success:
Target small area/project that is ideally in (or not much
later than) start-up phase; that is keen to be involved; will provide good exposure for the process; and that has a relatively short (weeks not months) time-boxed activity that can be used for the pilot;
Establish implementation plan (Are all products in
Implementors
There will be a need for processes to be developed and documented and for collateral to be produced to support these processes. Whilst a lot of this is generic in terms of the process, it will need to be tailored and/or developed to match the specific requirements of the organization in question Examples of the process materials and collateral required are:
Static Testing Policy, Strategy Review Process – Plan, Prepare, Execute, Record,
scope, or those selected by a Risk-based approach? What metrics to capture? Who are the project staff nominated for training? What are the timescales?);
Train
nominated project staff (Review Leaders, Moderators, Reviewers), both through standard courses and direct mentoring/shadowing;
Execute Review techniques on selected documents
(including review meetings and using all specified roles) and capture metrics;
Evaluate success of process, training and process
Complete
Review Techniques Guidelines Templates – Review Logs, Review Metrics, Document
materials (lessons learned), and identify areas for improvement (process gaps, better training, more detail in process materials, etc);
Rework process and training materials as appropriate
Review Matrix
Training courses
and create baseline set for publication.
Other factors to consider for the implementation are:
Test the process on the process! Internal verification
Full Implementation:
The full implementation of Static Testing (including the Risk Based elements if appropriate) should build upon the success of the pilot and it is particularly important to document and publicize the benefits demonstrated by the pilot. In addition, a further set of factors need to be taken into account:
Publish final process and all process materials Arrange training to cover all project/area staff (basic
of the materials using the process and templates themselves
Involve people from different projects/teams in the
development of the materials – less ‘selling’ to do when it comes to implementation When actually implementing Static Reviews, experience has shown that a phased approach is often best both in terms of minimizing impact and also being able to demonstrate early benefits as a result. For this reason AppLabs recommend an approach making use of a pilot and then a full implementation:
awareness training for all, detailed/leader training for 10-20% of staff)
Work directly with projects/teams, help to build Static
Testing into project plans and day to day work
Provide coaching/mentoring as required throughout the
Pilot implementation:
The strategy behind the pilot approach is to provide a least risk environment which will allow early demonstration of some or all of the benefits of Static Testing. Having
implementation
AppLabs.com
AppUK_WhitePaper_Static_Testing_Paper_1v00 Page © 2007 AppLabs
Risk-Based Static Testing
This section looks at the additional benefits that a riskbased approach to static testing can generate. Any kind of risk-based approach has the same fundamental premise: to use the finite resources we have available – time, money and people – to test in the most effective way possible, by focusing effort and using more rigorous techniques in the higher risk areas of the item under test. Risk-based Static Testing is no different:
Review Techniques can be broadly divided into formal and informal with each high level classification being further subdivided into individual techniques: Formal Techniques include Inspections, Structural Reviews and Walkthroughs, with Inspections being the most thorough (and time and resource consuming), Structured Reviews being the most common of the formal techniques and Walkthroughs being utilized to both review and evaluate an artifact but also at times to help educate the audience. For Formal Reviews and Structured Reviews in particular, the review will center around and culminate with a meeting/workshop to analyze and collate all the comments and to enable a decision to be made on the suitability of the artifact for publication/further processing. Informal Techniques employed might be “Buddy Checks” and Desk Reviews with Buddy Checks being a simple peer review of the artifact in question and Desk Checks being an even simpler “over the shoulder” walkthrough by the author – often used during production of the artifact to check that it is being produced more or less as would be expected. Informal Reviews may also be used to check that an artifact is ready for one of the formal review techniques. A pre-requisite to any review should be that the artifact in question has been quality checked by the author.
Identify your products
This may sound obvious, but the first step is to understand what exactly you are covering with this process. Compile a list of all the deliverables that are going to be created or changed by the work that you do: policy and strategy documents, project plans, requirements, specifications, designs, test artifacts, templates, reports, training material…anything and everything.
Make decisions about product risk
For each of the identified products consider the associated level of risk. What is the likelihood of there being a problem with the product, and what would be the impact of that problem should it manifest? Is it a brand new product, or an updated version of a product that is already well understood? Are there areas within the product that have caused problems in the past? Consider these factors and assign a rating to each product according to how ‘risky’ it is.
Identifying the right people to involve in reviews – rotate responsibility to encourage cross-skilling, and to keep the process ‘fresh’
Traditional product reviews where the artifact in question is circulated to a number of reviewers without any structure or guidelines, often results in the author receiving several sets of very similar comments (mostly spelling and grammar errors). By allocating specific review roles to individuals, the review coverage is increased and the level of duplication reduced. Suggested roles for reviewers might be:
Author – the person who developed and documented
Establish a Product Review Matrix
This is a way of recording and agreeing which review techniques should be considered for each of your products, driven by the level of risk you have identified for those products. Higher risk areas mean you should generally apply more rigorous review techniques, although this is dependent on many other factors as well and should be regarded therefore as “Best Practice” rather than an absolute requirement.
the document/process under review
Review Leader – the person who will organize and drive
Plan the right breadth and depth of reviews
The level of risk established when creating the Product Matrix may be used to determine the review technique(s) to be used and the resources to be allocated both in terms of the size of the circulation and also the skill level required to carry out the review.
the review
Moderator/Facilitator – a person with the skills to
facilitate the review and who generally would not take an active part in the review itself.
Scribe – will capture and document all the comments/
decisions and questions raised during the review
AppLabs.com
AppUK_WhitePaper_Static_Testing_Paper_1v00 Page © 2007 AppLabs
Reviewer(s) – those people who have actively reviewed
Sponsorship/endorsement from senior management
the document; there will be more than one and they will have been specifically targeted (by the Review Leader) to review the artifact from a single focus:
(including publication of a company-wide Policy or Mandate if possible)
Active support and provision of resources from more
Quality Usability Business Legislation (if applicable) Technical (may be several different technical views)
‘visible’ levels of management
Communication!
Get people involved during development of the process, and keep everyone informed as to what is happening and what the (positive) impact on them will be
Decision Maker – the person who will make the ultimate
Training courses should enable people to pick up and
decision on the suitability/acceptability of the artifact under review. Note that not all of these roles need be unique and some of them (but not all) can be combined and allocated to the same individual. Roles which should NOT be combined are the Moderator, the Decision Maker and the individual Review focus points.
run the process immediately (direct mentoring/coaching should gradually become less necessary) AppLabs advocates early engagement of testing in the systems lifecycle. By engaging in the “Idea” and “Requirements Definition” stages of the systems lifecycle AppLabs is able to perform “static testing”, which involves the use of reviews, inspections and walkthroughs to ensure clarity and completeness of requirements and the eradication of assumptions and ambiguity at the analysis stage. It is widely accepted that defects found late in the lifecycle cost a factor of 10 to resolve compared with each preceding phase, therefore the commercial benefits of “static testing” are evident. The British Computer Society estimates that for a % investment of the overall project budget in Static Testing, 70% of defects can be found by eradicating assumptions and ambiguity from documents before a line of code is written or a system is configured.
Early Mitigation of the Risks
The major purpose in Risk Based Static Testing is in mitigating the highest areas of risk as early as possible in the lifecycle with the most appropriate use of time and resources. This translates into identifying defects in the artifact under review as early as possible in order to minimize the chances that these defects are not resolved before any further processing using this artifact takes place. Industry statistics have demonstrated that identifying and correcting defects at the source document stage is several orders of magnitude less costly than correcting them in the development or dynamic testing (or heaven forbid Live) stages.
Key Success Factors:
In summary, Static Testing provides early identification of defects and in combination with a risk based approach can significantly improve the quality of source documentation and other artifacts. It does, however, require an investment of effort and should not be perceived as an “overnight” activity; it will require planning, training, managing and monitoring, but can be implemented in a phased manner. Experience has shown that there are a number of key factors that contribute to a successful implementation:
Have a dedicated, enthusiastic and knowledgeable
team in place to develop and launch the process
AppLabs.com
AppUK_WhitePaper_Static_Testing_Paper_1v00 Page © 2007 AppLabs