Docstoc

Measurement of Software Product Quality

Document Sample
Measurement of  Software Product Quality Powered By Docstoc
					Chemuturi Consultants – Do it well or not at all Measurement of Software Product Quality
Murali Chemuturi For a COTS product, the market judges the product quality and the result would be acceptance or rejection by the market. It is subject to market forces. For a “made-to-order” software product, the quality is assured by the software development process and the acceptance testing. Acceptance testing can never be exhaustive. Few companies measure the performance of software process even though it is mandated by standards such as ISO and models such as CMMI® (Capability Maturity Model Integration) of SEI (software Engineering Institute) of Carnegie Mellon University. Therefore, there is a need for measuring the software product quality, especially in a “made-to-order” software development scenario. Measurement of product quality does not easily lend itself to measurement leave alone accurate measurement. Second aspect is when do we measure the quality of a software product. In a physical product like a car, it would be subjected to extensive actual or simulated field trials before it is released to public. Software products that are COTS in nature are also subjected to field trials. But other software products that are built for a single customer, there would be no field trials except for provision of support during warranty period. So, if quality is to be measured at all for a software product, it has to be at the release stage. I propose a composite metric to measure the quality of a software product. This is based on the following parameters – 1. 2. 3. 4. 5. Organizational environment that fosters product quality Effectiveness of the organizational QA activities Review coverage of software artifacts Unit testing coverage of code Exhaustiveness of software testing

We rate each of these parameters and work out a Composite Product Quality Rating (CPQR) by assigning a weight to each of these parameters. Let us now see how to rate each of these parameters.

Organizational Environment that fosters product quality
Table 3.3 gives the attributes for rating this parameter. Attribute Rating

32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
Existence of a quality department – weight – 0.30 (Wo1) Fully staffed, independent testing team, quality head equal in rank and pay with delivery head, champions process and product quality Fully staffed, independent testing team, quality head lower in rank and pay compared to delivery head, champions process and product quality Fully staffed, independent testing team, quality head lower in rank and pay compared to delivery head, champions process Staffed to quality audits and coordinate process activities, stop gap quality head, coordinates process activities, depends on project teams for independent testing Staffed to coordinate process activities, no qualified quality head, coordinates process, independent testing is the responsibility of project teams Existence of software development processes – weight 0.40 (Wo2) Process covers all aspects of software development and quality assurance, standards and guidelines for all software development activities, a formal metrics and analysis mechanism exists, a formal process improvement mechanism is in place Process covers all aspects of software development and quality assurance, standards and guidelines exists for all software development activities, formal process improvement mechanism is in place Process covers most aspects of software development but not fully on quality assurance, standards and guidelines do not fully cover all aspects of software development, process improvement is event driven Process covers some aspects but not all aspects of software development, process improvement is not really an important aspect Organization is not driven by process at all 5 4 3 2 1

5

4

3 2 1

32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
Process Conformance –weight – 0.20 (Wo3) Process conformance audits are process driven, all audits are carried out conforming to an approved plan, all NCRs receive serious attention by management, management review meetings are held conforming to approved plan without fail Process conformance audits are process driven, most audits are carried out conforming to an approved plan, important NCRs receive serious management attention, management review meetings are held regularly Process conformance audits are process driven, audits are conducted regularly but not according to an approved plan, NCRs are resolved by head of quality or by the software engineering group or delivery head, management review meetings are conducted by head of quality or by the software engineering group or delivery head Process audits are process driven, audits are conducted based on organizational convenience, management review meetings are conducted based on the convenience of the person chairing the meeting, NCRs are resolved mostly by the auditor Process audits are conducted as and when convenient. Not process driven, NCRs remain open for a long time Rewards and recognition – weight – 0.10 (Wo4) A formal mechanism exists to recognize and reward efforts to improve quality of products and the awards are regularly given A formal mechanism does not exist but rewards are given regularly for efforts at improving product quality Recognition and rewards for efforts on product quality improvement are given occasionally only when outstanding efforts are seen Now compute the value for Organizational Environment as – Rating for Quality Department X W o1 + Rating for Software Process X W o2 + Rating for Process Conformance X W o3 + Rating for rewards & recognition X W o4 Let us assume these values Rating for Quality Department = 4 Rating for Software Process = 5 Rating for Process Conformance = 3 Rating for rewards & recognition = 2 Now the value for Organizational Environment Rating (OER) is – (4 X 0.3) + (5 X 0.4) + (3 X 0.2) + (2 X 0.1) That is – 4. Now keep this aside until we compute other values also. 5

4

3

2

1

5 3 1

32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
Effectiveness of the organizational QA activities
The effectiveness of the organizational QA activities would result in elimination or minimization of residual defects being passed on to the customer. This effectiveness can be measured using the defect density in the delivered product and is also called as the sigma value. This aspect is covered in greater detail in the chapter on quality metrics and measurement. Use the sigma value of the organization to derive effectiveness of organizational QA activities rating. Table 3.4 shows the derivation of this rating. Table 3.4 Derivation of effectiveness of organizational QA activities rating (EQAR) Sigma value of organization Rating 6 sigma (3 defects per 1,000,000 opportunities) 5 5 sigma (3 defects per 100,000 opportunities) 4 4 sigma (3 defects per 10,000 opportunities) 3 3 sigma (3 defects per 1,000 opportunities) 2 2 sigma (3 defects per 100 opportunities) 1 For our computational purposes let us assume a value of 4-sigma for the present

Peer Review coverage of software artifacts
Subjecting all software (information as well as code) artifacts to 100% peer review and managerial review is a best practice. However, many organizations skip peer review in some cases under the assumption that managerial review would be adequate for the purpose. Now compute this value as – Percentage of artifacts covered by peer review = (Number of software artifacts covered by peer review / Total umber of software artifacts) X 100 Now using this percentage derive a rating for review coverage as shown in table 3.5 – Table 3.5 Derivation of peer review coverage rating (PRCR) Percentage of review Rating coverage 100% coverage 5 80% and above but less than 4 100% 70% and above but less than 3 80% 60% and above but less than 2 70% Less than 60% coverage 1 For our computational purposes let us assume a value of 90% coverage for the present

Unit testing coverage of code
32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
Here, “unit testing” means “independent unit testing”, that is carried out by a person who did not code the artifact that is being tested. Subjecting all software code artifacts to 100% independent unit testing is a best practice. However, many organizations skip unit testing in some cases under the assumption that self unit testing (that is unit testing by the programmer who coded the artifact) would be adequate for the purpose. Now compute this value as – Percentage of artifacts covered by unit testing = (Number of software artifacts covered by unit testing / Total umber of software artifacts) X 100 Now using this percentage derive a rating for unit testing coverage as shown in table 3.6 – Table 3.6 Derivation of unit testing coverage rating (UTCR) Percentage of unit testing Rating coverage 100% coverage 5 80% and above but less than 4 100% 70% and above but less than 3 80% 60% and above but less than 2 70% Less than 60% coverage 1 For our computational purposes let us assume a value of 75% coverage for the present Exhaustiveness of software testing Exhaustiveness of software testing on a software product has two aspects, namely, the number of tests that should have been conducted and the number of test cases that should have been executed. Let us look at the first aspect of number of tests that should have been conducted. Each software product, depending on its nature (number of tiers, functional domain. Development and target platforms) needs to be subjected certain types of tests. Some products, such as reservation systems, need more exhaustive testing for parallel and concurrent testing than other tests. Some products, such as financial systems need to be more vigorously tested for accuracy and precision. Some products such as online stores need to be tested more for their response times. Thus they types tests a software product needs depends on its nature, A more exhaustively tested software product has better chance at being a quality product than a product that is less exhaustively tested. The types of testing that should the software product be subjected to is normally is recorded in the test plan document of the project. Sometimes, the types of tests to be carried out, is specified by the customer in the case of outsourced project. We need to contrast this specification with actual tests that are carried out. Sometimes, due to various reasons, such as – 1. Lack of time, all the planned or specified tests are not carried out 2. Based on the results obtained in the peer reviews and initial tests, a decision may be taken that further testing is not really necessary 3. Delivery and schedule pressures may force a decision to abandon further testing 4. so on
32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
Now compute the percentage of tests conducted using the formula – (Number of tests that are actually conducted / Number of tests that should have been conducted) X 100 Now using this percentage derive a rating for exhaustiveness of tests conducted as shown in table 3.7 – Table 3.7 Derivation of exhaustiveness of tests conducted rating Percentage of tests Rating conducted 100% conducted 5 80% and above but less than 4 100% 70% and above but less than 3 80% 60% and above but less than 2 70% Less than 60% conducted 1

Similarly all the planned test cases may not have been executed. Now compute the percentage of test cases executed, using the formula – (Number of test cases that are actually executed / Number of test cases that should have been conducted) X 100 Now using this percentage derive a rating for exhaustiveness of test cases executed as shown in table 3.8 – Table 3.8 Derivation of exhaustiveness of test cases executed rating Percentage of test cases Rating executed 100% executed 5 80% and above but less than 4 100% 70% and above but less than 3 80% 60% and above but less than 2 70% Less than 60% executed 1

Now let us assign weights to each of these two ratings. A weight of 0.6 is assigned to the exhaustiveness of tests conducted as missing a test altogether is much more serious than missing a few test cases. A weight of 0.4 is assigned to the exhaustiveness of test cases executed. Now let us compute the rating for exhaustiveness of testing as –

32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
Exhaustiveness of testing rating = (Exhaustiveness of tests conducted X weight) + (Exhaustiveness of test cases executed X weight) Let us assume a rating of 4 for exhaustiveness of tests conducted and a rating of 5 for exhaustiveness of test cases executed. Now. Let us compute the rating for exhaustiveness of testing, thus – Exhaustiveness of testing rating (ETR) = (4 X 0.6) + (5 X 0.4) = 2.4 + 2 = 4.4 Now we are ready to compute the composite software product quality rating. Composite Product Quality Rating The first activity is to assign appropriate weights to each of the product quality parameters. Table 3.9 shows the suggested weights. Table 3.9 – Weights for product quality parameters Product quality parameter Organizational environment that fosters product quality (OER) Effectiveness of the organizational QA activities (EQAR) Peer review coverage of software artifacts (PRCR) Unit testing coverage of code (UTCR) Exhaustiveness of software testing (ETR) Total Weight

Weight 0.35 (W 1) 0.25 (W 2) 0.15 (W 3) 0.15 (W 4) 0.10 (W 5) 1.00

If the organizational environment were one that fosters product quality, then all other parameters would automatically get the highest importance in the organization. That is the reason for assigning the highest weight for this parameter. Effectiveness of organizational QA activities would be higher if the organization fosters product quality, which is dependent on peer review and unit testing. If peer review and independent unit testing are carried out effectively and exhaustively, then the software testing be automatically taken care of.. Now let us compute composite product quality rating (CPQR), thus – CPQR = (OER X W 1) + EQAR X W 2) + (PRCR X W 3) + (ETCR X W 4) + (ETR X W 5) Let us substitute values computed above by us in the formula and arrive at the composite product quality rating, thus – CPQR = (4 X 0.35) + (3 X 0.25) + (4 X 0.15) + (3 X 0.15) + (4.4 X 0.10) CPQR = 1.4 + 0.75 + 0.60 + 0.45 + 0.44) CPQR = 3.64 Thus the CPQR is 3.64 on a 5-point scale. Now I would like to note the following points –

32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
1. It is possible that some organizations may feel the need to add or delete some of the parameters mentioned above. If their unique conditions dictate so, they may add or delete some of the parameters. 2. Some organizations may assign different importance and hence would like to assign different weights to the parameters. They may do so. 3. While a 5-point scale is suggested, if some organizations so desire, they may use a different scale too. Now, do we really need to measure the quality of the product that is developed against specified user requirements and against an order to do so? It is an accepted fact that there would be some residual defects and this is measured by the sigma level of the organization that developed the software product. After all, it is the end-product that we are really interested in and not in the organizational environment and other factors that are included in the above described measure. Sure, the sigma level of the organization is alone adequate to measure the quality of the product iff (this term is used in mathematics and it means “if and only if”) the sigma level is diligently and accurately derived. Most software development organizations that are engaged in customdevelopment of software do not have internal mechanisms to diligently collate all customer defectreports, analyze, and derive the organizational sigma level. Those organizations support the customer only during the warranty period and the focus is on resolving the defect than on diligently recording and analyzing it. Normally organizations designate a PL (Project Leader) or an equivalent person to provide support to the customer during the warranty period and the customer communicates with this person. Organizational level mechanisms for deriving organizational sigma level are more frequently absent than being present. After warranty, the customer is likely to take on the software maintenance in-house or may entrust it to the original developer or to a third party. The defects that are unearthed during software maintenance are never considered for inclusion in the data for computing organizational sigma level. Therefore, the sigma level alone is inadequate for measuring the quality of the software product that is produced through custom-software development. In the above example, while the sigma level of the organization is 5, the product quality turns out to be 3.64 only! It is also very easy to collect the data to compute this metric. The information needed for deriving the organizational environment is most likely available on the web site itself except perhaps for the existence of a quality department. This information cannot be classified as confidential by any stretch of imagination and therefore easy to obtain. The effectiveness of organizational QA activities depends on the sigma level of the company and the organization would be glad to share with you the information, if it derives it at all. If the organization, does not derive ande maintain its sigma value, you can consider it as 4-sigma level (that is 3 defects 10,000 opportunities) and proceed further. Peer review coverage information can be derived from the review records of the project. Unit testing coverage information can be derived from the unit test plans and unit test logs of the project. Exhaustiveness of testing information can be obtained from the product test plans and product test logs of the project. Thus the information needed to derive this CPQR is easy to obtain and the methodology is fairly simple.
32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
Now how do we interpret this CPQR metric? Table 3.10 shows the maximum possible values for each of the factors that go into the derivation of CPQR. Table 3.10 – Maximum possible values for CPQR factors Product quality parameter Maximum possible value Organizational environment that fosters product quality 1.75 (OER) Effectiveness of the organizational QA activities (EQAR) 1.25 Peer review coverage of software artifacts (PRCR) 0.75 Unit testing coverage of code (UTCR) 0.75 Exhaustiveness of software testing (ETR) 0.50 Maximum possible CPQR Value 5.00 Obviously, a value tending towards 5 is most desirable. To determine what is the minimum acceptable CPQR value, let us find out what minimum values can be expected from each of these factors. Even if we assume that the organization does not have a quality department, and no rewards and recognition system, it normally has a software development process, which contributes (0.40 X 1.75 = 0.70). if there is a process in existence, at least a sketchy process conformance would be in place which on an average level contributes (0.2 X 1.75 = 0.35). So the minimum expected from organizational environment factor in a process-driven organization would be 1.05. A process-driven organization would be at least at 4-sigma level (that is 3-defects per 10,000 opportunities). Effectiveness or organizational QA activities factor would thus be (0.25 X 3 = 0.75). Peer review coverage factor accepting a rating of 4, can be a minimum of (4 X 0.15 = 0.60). Unit testing coverage factor accepting a rating of 4, can be a minimum of (4 X 0.15 = 0.60). Exhaustiveness of testing factor accepting a rating of 4, can be a minimum of (4 X 0.10 = 0.40). Now the minimum acceptable CPQR would be thus (1.05 + 0.75 + 0.60 + 0.60 + 0.40 = 3.4). Now we have the range of 3.4 at a minimum to 5 at a maximum for CPQR for a software product that is developed in a process-driven organization. Out of these factors, the two factors, namely, organizational environment and effectiveness of organizational QA activities are beyond the purview of the current project. If those values are low, it is a concern for the development organization to improve. If the CPQR is low, we can inspect the other three values and insist on improving them. While the CPQR value of 5 is desirable, a value of 4 may be good augury. This metric may be computed before going in for acceptance testing. Then it would be possible for insisting on improvements based on CPQR. In conclusion of the discussion about product quality measurement using CPQR, I would like to place on record that it is desirable to compute this metric as it is accepted that 100% defect removal is not a practical possibility. CPQR would provide a quick, reliable and objectively derived
32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com

Chemuturi Consultants – Do it well or not at all
measure that assists the customer in judging the quality of the product being delivered and to ensure that the product is as defect-free as practically possible. ************************************************************************************************** Your feedback is gratefully acknowledged. Please send it to – murali@chemuturi.com ***************************************************************************************************

32-78/3 Sainik Nagar Rama Krishna Puram, Secunderabad - 500056 +91-40-2722 0771 - www.chemuturi.com - murali@chemuturi.com


				
DOCUMENT INFO
Shared By:
Stats:
views:198
posted:6/10/2009
language:English
pages:10
Description: This white paper shows ideas for measurement of software product quality