How to perform good Test Estimation Problem Statement: In this document, I outline some guidelines about performing a Test Estimation exercise. For the purpose of this document, I mean “Test Development Cost” when I say “Test Estimation”. In other words, the document is focused on estimating Test Development and not Test Execution. Test Estimation is not a perfect science and there are many factors that affect this. There are no silver bullets to a successful estimation. In most cases, understanding and thinking through the factors (parameters) that I mention below significantly enhances the possibility of a successful estimation. Goals and Non Goals: Goals: To understand various factors that play a vital role in Test Estimation Process. To be able to understand how these factors can be measured. To discuss about some possible pitfalls in the process and some tricks of the trade suggestions. Non Goals: This document is limited to test development estimation and does not talk about test execution. As I mentioned above, Test Estimation is not a perfect science and hence this document does not intend to provide silver bullets. Instead, this focuses more on a generic methodology and suggestive guidelines. Theory: First and foremost, Test Estimation should always be “realistic” and should be “backed by information and description of the factors that influence the estimate.” The key is to understand the factors those affect estimation. I categorize these factors into 4 buckets: Source Factor Names Engineering and clearly defined hand-offs between testing and the rest of the organization processes timely and reliable bug fixes timely arrival of high-quality dev code proper execution of early test phases (unit, component, and integration) Clarification of ownership between the between the various features being tested so that there is a clear demarcation to make sure nothing is left to interpretation.(UI testing/API testing, ownership of the different integration points) Tools at hand, Resources Existing vs new development of a test case Quality of the test system which includes the test tools, test environment and so forth An adequate dedicated and secure test environment Goal of Automation – how much is planned automation vs manual Running the test pass with code coverage offers a good understanding of the state of testing available, high-quality (clear, concise, accurate, etc.) project documentation like requirements, designs, plans, and so forth Clarity of understandin of functional specs and Dev Design. reusable test systems and documentation from previous, similar release the similarity of the project and the testing to be performed to previous endeavors Not in schedule Quick fixes and Hotfixes and kind of testing performed for them People and Team honesty, commitment, transparency, and open, shared agendas among the individual contributors, the managers, and the project stakeholders realistic expectations across all participants, including the individual contributors, the managers, and the project stakeholders stability of the project team, especially the absence of turnover competent, responsive test environment support use of skilled contractors and consultants to fill gaps Outsourcing should account for ramp up time, possible geographical remoteness, 6 hour work week and leveraging on their experience for previous releases the need to ramp up, train, and orient a growing test or project team Required and sufficient information is supplied by the PM,Dev and leaders of the project Other Complication high complexity of the process, project, technology, organization, or test environment Rapidly changing PM spec and/or dev code. In those cases the start date and end dates change based on the churn Additions of new DCR’s the need to assimilate or develop new tools, techniques, or technologies at the testing or project levels any requirement for new test systems, especially automated test ware (harness and methodologies), as part of the testing effort any requirement to develop highly detailed, unambiguous test cases, especially to an unfamiliar or incomplete standard of documentation tricky timing of component arrival, especially for integration testing and test development fragile test data, for example, data that is time sensitive Buffer time incorporated in the test schedule after discussion with lead/manager Generically speaking, prioritizing these factors is a difficult task because: (a) Prioritization depends and varies case to case, feature to feature, product to product (b) Factors have some dependencies with each other. However, I would suggest the following as the FIVE most important factors for Estimation: (a) Specs: The Functional specs, Dev Design need to be frozen before a successful Test Estimation can happen. Understanding these specs is also a key to successful test estimation as that ensures nothing’s falling through the cracks and chances are nothing’s going to be discovered late. (b) Freezing the Test Spec: Trying to estimate test automation without scoping and freezing the test documentation doesn’t yield a lot of value. (c) State of Test Tools and Test Environment: Reliable, Performant and Robust Test Tools (including harness) and Test Environment (for example: Test Clusters, Shared Test Code, Test Library etc.) go in a long way to positively effect Test Estimation. (d) Automation vs Manual Trade-Off: In general, More automation means more development cost. However, test automation significantly reduces test execution cost (e) Re-Use: If existing test tools, cases and code can be re-used, that helps significantly reduce the test development cost. Methodology: Step 1. First of all, focus on the top 5 factors I mention above. Also, look at the list above to pick out the most important factors that affect test estimation in your case. List down the current state and risks involved for these factors. Step 2: Analyze the throughput / rate of your test development. In other words, how long does it take for the concerned Test Developer to develop one unit of test code. This has the potential to become a significant problem when the person estimating and the person actually developing are different people or geographically distant. Step 3: Start with T-Shirt size estimation for testing. In other words, divide the estimate into one of “S” (let’s say < 10 man days), “M” (let’s say 10 – 20 man days), “L” (20 – 32 man days) and “XL” ( > 32 man days). Step 4: Now, break the total work into work items (tasks). Each task should typically be less than 1 man week (i.e. < 5 man days). In other words, making the tasks granular enough so that they can fit in a week. Step 5: Add up all the work items (tasks). Add 30-50% buffer to that schedule (depending on your team abilities and experience). Account for Vacation days. Step 6: Set up a good Tracking Mechanism to track progress week-over-week. Product Studio, xls Worksheet or Project can be used to set this up. Step 7: Keep updating your estimation week-over-week.