Docstoc

Survey-for-defect-prediction-and-best-practices---Software-Reliability

Document Sample
Survey-for-defect-prediction-and-best-practices---Software-Reliability Powered By Docstoc
					In general you will be filling in information that has a purple box A. Complete the survey Make sure that there is physical evidence to support all yes and sometimes answers B. Complete the instructions in the defect data worksheet. These are defects found exclusively after delivery. C. Complete the instructions in the size worksheet D. Optionally, you can provide defect data during systems testing to help us develop a model to predict total defects during testing. If you wish to do this, complete the optional defect data worksheet. The optional defect data must be from the same project as all other data.

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA ORGANIZATION Is software engineering visible to upper management? Is software engineering visible to engineering management? Is there a defined team structure for software engineers? (an example would be that some design, some code, some test or that they all design, code and test) Software engineers are located near other engineers (as opposed to another city or country) Software engineers are located near target equipment Software engineers consider themselves to be part of engineering The software testers do not code There is a software Quality Assurance organization How many software engineers are there not including testing or QA?

At least someone on software team has domain knowledge of product and end user requirements How many people have more then 3 years of experience with development environment? (i.e. Visual net is a development environment) How many people have more then 3 years of experience with the operating system? How many people have more then 3 years of experience with software engineering other then for academia? More then 3 month time to market There is a software manager that does not code (this person can do any other task other then actively code) Your SEI CMM level (between 1 and 5). Do not enter a 2,3,4, or 5 unless you have been certified by the SEI. If turnover rate is <20% per year enter a 1, if <40% per year enter a .5 and if >= 40% per year enter 0

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA

Your organization has control and management over it's subcontractors and vendors Enter the ratio of software testers to all software people here Do you have offshore programmers? If so, what is the ratio of these to the other programmers? Does this group regularly make enhancements after the coding phase has ended? Does this group regularly make enhancements after the testing phase has ended? Does this group have a formal means by which to filter, assign priority and schedule customer requests? What is your organization's philosophy on interruptions for software engineers? A) interrupt for any reason b) interrupt when the interruption will determine if the programmer understands what needs to be done or prevent a loss of time due to having the ladder against the wrong wall or C) never interrupt the programmer because they know what needs to be done better then anyone else. Does the average software engineer have a private workspace? Does the average software engineer have a reasonably quiet workspace? Do you have ISO certification? If so, what level and when? Has your organization had an SSQA audit (this is for semiconductor companies)? If so, when and please attach final results. What is the longest phase of the lifecycle - requirements, design, code or test? PROCESSS MEASURES Analysis Requirements analysis is part of the schedule (there are milestones and schedule time dedicated to analyzing the requirements) We have procedures for doing analysis We get the customer involved in analysis The customer requirements are translated to software requirements

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA Conflicts discovered in the customer and software requirements are resolved prior to entering the next phase We review the requirements document before entering the next phase We use prototyping as a tool to model the end user requirements We get the testers involved during the requirements phase We start/refine the test plan during requirements We have requirements software tools Design Design is part of schedule (there are milestones and schedule time dedicated for architectural design) We have design procedures We trace requirements to design We do top level design (behavior, states, architecture) We do detailed design (psuedocode) Conflicts discovered in this design are resolved prior to entering the next phase We review the design document before entering the next phase We use prototyping as a tool to model the design and architecture We get testers involved during design We start/refine the test plan during design We have design software tools CODE We have coding procedures and standards We trace code to design and requirements

Conflicts discovered in coding are resolved prior to entering the next phase We have exception handling standards

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA We use debuggers during coding We have coding tools (code analyzers, indenters, etc.) We get system testers involved prior to systems testing We start/refine the test plan prior to the start of systems testing We have code reviews consistently and regularly and to a defined set of criteria Unit test Unit testing is part of schedule (there is time allowed in the schedule for this) We have procedures for unit testing Unit tests are reviewed by a knowledgeable person such as a supervisor, lead engineer or manager We map the requirements and design to our unit tests Conflicts discovered in this unit testing are resolved prior to entering the next phase We use debuggers during unit testing We execute functional tests We execute path tests We execute tests for inputs and ranges and domains We test for algorithm underflow and overflow We test for complex logic (more then 2 anded or ored conditions) We test for module level error handling System testers are involved prior to systems testing We start/refine the test plan prior to systems testing

We have unit testing tools If so, have all software engineers on this project been trained to use the tool? Do all software engineers on the project use the tool? System test System test is part of schedule We have procedures for how to do systems testings (template test plans, etc.) We map the requirements and design to our systems tests

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA Conflicts discovered in systems testings are resolved prior to delivery We test the critical path We test system level error handling (I.e. error that are interprogram or hardware related) We do behavior tests such as state models We do install tests We have test beds We do performance testing when applicable We do security testing when applicable We do multi-user testing when applicable We do platform tests when applicable We test the user manual, documentation, release notes We test for time domain issues such as Y2K We have tools for systems testing (screen capture and replay for example) If yes, the appropriate people have received training on how to use them If yes, the tools are used for tedious testing and not necessarily as a substitute for all testing We have simulation tools REGRESSION TEST Regression test is part of schedule We have procedures for doing this We retest all corrective actions since last regression test We retest all upgrades since last regression test We retest all modifications since last regression test We retest all features that are highly visible to most customers regardless of whether it has been changed We retest all parts of the code that are historically buggy regardless of whether it has been changed CORRECTIVE ACT We have procedures for doing corrective action The software engineer retests all changed units prior to a regression test The software engineer retests all changed paths prior to a regression test We have source control procedures for corrective action The software engineer retests all new features added since the last regression test prior to the next regression test Defect tracking is used to document ALL corrective actions made after a baseline

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA The software engineer retests all global variables affected by this change Defect tracking system We have one We have procedures for defect tracking It is used by all software individuals We have it automated Testers use it to find out what bugs to retest Customers have access to it (it does not have to be complete access) The field uses it The systems engineers use it The software engineers use it to document all changes made to reported defects There is a physical link between the defect tracking and the source control system so that any time a change is made to the source a record is created in the defect tracking system. It is on an intranet or internet MISCELLANEOUS PROCESS We have Quality Assurance procedures for software We have delivery procedures for software We use project management tools for generating and maintaining the schedule We use object oriented (OO) methodologies Our people understand the 4 things that make OO different then non-OO We have regular walkthroughs on our product We use product metrics to measure things such as size, complexity, etc. Scheduling is part of the software development process We use process metrics to measure things such as actual time to completion, software maturity. (This spreadsheet is a process metric by the way). We use fault metrics to measure defect counts, severities or densities or failure rate We have reuse libraries and we use them to cut down on copy and paste code We do a root cause analysis on our bugs on a periodic basis to see what kinds of bugs we are delivering We proactively make improvements based on the results of the above root cause analysis We have a programmers workbench which is a fragment of hardware that resembles the target hardware and is available to software engineers and testers Our life cycle model is either incremental, waterfall, spiral or some combination thereof. Our programming language is supported by the original manufacturing as well as third party vendors

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA Our Operating System is supported by the original manufacturer as well as third party vendors CONFIGURATION MANAGEMENT Configuration management exists We have procedures for CM All software engineers use our CM and version control system Our system makes it difficult to have concurrent versions (more then one version built from the same baseline) Our system allows for a stop shipment by appropriate personnel Our CM system is used for requirements, design, test plans as well as code We have this CM system automated Enter 1 if you make <= 4 releases per year. Enter .5 if you make 5-8 releases per year. Enter 0 if you make more then 8 releases per year of the SAME software product line. We avoid concurrent releases (releases of slightly different features based from the same baselines) Industry and application characteristics Is this software or part of it, real time? Does this software have safety considerations? Is this software part of an experimental system? (this means that the engineering concepts are not known to be physically achievable such as software that supports a means for a human being to fly, etc.) What is the age in years of this software? Include the oldest part of the existing code. How many many versions of this software have been released prior to now? How many minor versions of this software have been released prior to now? Does this software have security considerations? Does this software have multi-user or multi-tasking considerations? How many different customers (as in organizations) will use this software ultimately? One, a few, dozens, hundreds, thousands, etc. How many end users will use this software utlimately? (this means the actual number of people) One, a few, dozens, hundreds, thousands, etc. What educational level represents the least experienced end user? Grade school, high school, some college, bachelors, masters, phd What educational level represents the most experienced end user? Grade school, high school, some college, bachelors, masters, phd Does this software have significant database interfaces? Does this software have significant mathematical requirements?

Answer These questions are to be answered for ONE specific product or project. Do not answer globally for Y, N, S or entire company. For parts of the lifecycle that have not taken place yet, answer for what is PLANNED. NA Does this software interface to hardware that is not yet defined or is continually changing itself? Describe. Does this software perform a process that is continually evolving? Describe. Does this software have internet capabilities? Does this software have requirements that require specialized knowledge to define that cannot be learned from college, seminars, etc? Is this software and/or it's processes susceptible to review by a government agency such as FDA, DOD, etc? On average, what percentage of defects found during testing or delivery are caused by fixes to other defects? How many third party vendors will supply software for this product? How many total executables and DLLs are part of this system? How many hardware subsystems are there for this product? Describe the typical fielded duty cycle for this product as 1) infrequent (i.e. missile, security systems) 2) occasional 3) frequent 4) round the clock Describe the mission time for this product as measurable in 1) minutes (dishwashers or missiles) 2) hours (medical therapies, flights) 3) days or longer Are all applicable communication protocols developed according to an industry standard? Are all applicalbe hardware boards developed according to an industry standard? Are all drivers developed according to an industry standard? Is the operating system home grown or purchased? Is the operating system multi-tasking? Product characteristics If you don't know the answers to the below then state so. Is the design currently undergoing reconstruction (say to object oriented from non-object oriented)? Is the code currently undergoing reconstruction? Do you know the average mccabe's complexity of this product. If so enter it and describe the means by which is was computed such as the tool that computed it. Do you know the average functional complexity of each function, procedure of code? (i.e. the number of unique tasks performed by each unit) Does this code have any irregular exits such as goto statements? Does this code have any irregular exits such as multiple return statements? Does this code have any depths of logic nesting > 3? What percentage of the code has been copied and pasted? (an estimate is OK)

Criteria Evidence supporting a yes, sometimes or no answer The level above engineering is upper management. Is the software budget, milestones, etc. visible at this level? The level above software engineering is the engineering level. Is the software schedule, budget, etc. visible at this level? A defined team structure means that each person has a defined role. Each person may have more then one role but these roles are defined and predictable. Do you have software engineers distributed across the country or outside the country? If the target equipment is anything other then a PC, are ALL software engineers co-located geographically near it? This can be determined via an org chart or via asking them While software coders must test, there must be some testers that don't code

This could be a testers, manager or software engineer who is available to the rest of the group on a continuous and consistent basis for clarifying end user domain perspectives

Time to market is from start of design to deployment

This includes both voluntary and involuntary turnover

Criteria 1. Are subcontractors/short term contractors used only for software development that is not in the company's line of business? 2. Are the deliverables of short term contractors/vendors measured with the same acceptance procedures as required for internally developed software?

Private means that other people aren't easily visible (such as with a bullpen)

Enter n/a if not in the semiconductor industry Measure this in terms of calendar time and not resources

This is a procedure for how the requirements will be developed, translated, approved, etc. For software with many customers, this could be a customer that reflects the customer domain

Criteria If there are ambiguities in the requirements are they cleared up prior to designing and coding? This is a formal review A prototype can be a piece of paper, picture, drawing, software prototype, etc. Any 2 or 3 D model. This means that testers must review and accept the requirements document The test plan does not have to be complete during this phase, just evolving. These would be parsers, special tools for finding ambiquities in software requirements.

This would be a checklist or document that defines what an acceptable design is. (i.e. Object oriented, etc.) This is an actual table tracing each to the other one by one

If there are issues requiring clarification by a subject matter expert, they are escalated and resolved prior to coding. This is a formal review If applicable. (i.e. timing analyses) The testers don't design, however, they are workign on the test plans and procedures while the software engineers are designing. The test plan is evolving during this phase. i.e. UML, etc. These are usually language specific. This is literally a table that traces one to the other If a software engineer uncovers a potential requirements, design or coding issue, the software engineer will escalate so that it can be resolved. This is the opposite of keeping big issues quiet. These are conventions for how to implement error handling at the module level.

Criteria Some development environments do not support debuggers (such as firmware). Generally, these should be available and used. Examples are spy, etc. Testers come on board prior to the start of systems testing. The test plan is evolving in this phase. The defined set of criteria must be written and available for each review. It could be a simple checklist or the entire coding standard. Unit testing is done exclusively the software engineers. These are simple how to instructions for what must be unit tested. See comments for previous sections for each of the below.

Making sure that the requirements are met by each unit Making sure that every line of code is tested at least once. Making sure that intersections of domains (i.e. <=, <, >, >= are tested There are 2^n test paths for each logic that is anded or ored together. Are all tested? Making sure that the error handling on the module works as required Systems testers don't unit test, however, they should be working on the systems test planning while the software engineers are unit testing. The test plan should be evolving prior to the start of systems testing These would be tools to identify paths, logic, domains, etc. You would know it if you have them because they are not cheap and require training to use.

These are instructions for how to determine what will and will not be tested, when, etc. This is not the test plan itself. See the above items for comments on how to answer the below.

Criteria The critical path has no error conditions The system level error handling is making sure that the software works when hardware is unavailable, etc.

A test bed is a protocol in which the answer is known in advance. If not applicable then answer n/a If not applicable then answer n/a If not applicable then answer n/a If not applicable then answer n/a If not applicable then answer n/a If not applicable then answer n/a

Regression tests are performed by testing

Criteria

The 4 things are inheritance, encapsulation, polymorphism, abstraction

Name your language(s)

Criteria Name your operating systems

Criteria

List the agencies that are applicable

Example, robotics, process chamber, material handling system, etc.

For just one unit and one applicable, how long does the typical usage last? If there are no special boards, then enter n/a

There are cheap tools that can calculate each of the below

The mccabes complexity can be measured by a variety of very cheap tools

This data shown in this sheet must be from the same project as the data shown in the size page and the survey pages. 1. We need to know the counts for unique defects for some time intervals after delivery. This is an example of such data:

Number of unique defects found after delivery Week this week 27-Oct 3 You can also give us the counts per day if you have them 3-Nov 2 You can also provide this data in cumulative format also 10-Nov 4 17-Nov 1 24-Nov 2 Do not count defects found prior to release. Since the goal is to predict all defects that will have to be fixed after delivery, you should count all defects that were visible both internally and externally AND ultimately required a fix. So, if a customer did not see a defect, but your internal groups did and it was serious to require fixing, it should be added. If you like, you can segregate those found by customer in one batch and those found internally after release in another batch.

2. Make sure that all defects counted are A. unique B. valid (they really are defects) c. not enhancements d. not defects caused by enhancements made after delivery e. are all "escapes" meaning that these are things that ultimately require fixing. f. not defects found and fixed prior to delivery

3. Optional information that is quite useful for predicting defects by severity

Week 27-Oct 27-Oct 3-Nov 3-Nov 10-Nov 10-Nov 17-Nov 24-Nov

Number of unique defects found this week 1 2 1 1 1 3 1 2

Severity type Catastrophic Critical moderate Critical moderate Critical Catastrophic moderate

4. What is the date that this version was first deployed? 5. What is the date that this version was first started in development? 6. Can you approximate the duty cycle of the software in the field? How many individual installs? How many hours of operation per month?

If you are measuring size in KSLOC then follow these instructions Do the below for each language such as C, C++, assembly, java, VB, etc. Language 1 Language 2 1. Describe the language 2. Count up all new lines of code added for this release 3. Count up all modified lines If you don't know the modified lines or new lines then compute just a simple delta in size from the start to the end of the release If you are measuring size in function points then follow these instructions 1. Describe the language 2. Count up all function points from new code 3. Count up all function points from modification Even though you are using function points, we still need to know this

Language 3 Language 4

eed to know this

This data shown in this sheet must be from the same project as the data shown in the defect data page, the size page and the survey pages. 1. We need to know the counts for unique defects for some time intervals during final systems testing. This is an example of such data:

Week 27-Oct 3-Nov 10-Nov 17-Nov 24-Nov

Number of unique defects found this week 3 You can also give us the counts per day if you have them 2 You can also provide this data in cumulative format also 4 1 2

2. Make sure that all defects counted are A. unique B. valid (they really are defects) c. not enhancements d. not defects caused by enhancements made after the start of testing e. are all potential "escapes" meaning that all of these defects required fixing 3. Optional information that is quite useful for predicting defects by severity

Week 27-Oct 27-Oct 3-Nov 3-Nov 10-Nov

Number of unique defects found this week 1 2 1 1 1

Severity type Catastrophic Critical moderate Critical moderate

10-Nov 17-Nov 24-Nov

3 Critical 1 Catastrophic 2 moderate

4. What is the date of the start of final systems testing? 5. What is the date that final systems testing ended? 6. Can you approximate the duty cycle of the software during systems testing? How many individual installs? How many hours of operation/testing per day or week?


				
DOCUMENT INFO
Shared By:
Tags: Surve, y-for
Stats:
views:20
posted:11/28/2009
language:English
pages:23
Description: Survey-for-defect-prediction-and-best-practices---Software-Reliability