Case Study: Community Testing for Agile-Development Web Applications Second Rotation Inc.
uTest Inc. 190 Front Street, Suite 101 Ashland, MA 01721 T: 1-800-445-3914 F: 1-617-203-9005 info@uTest.com www.utest.com
190 Front Street, Suite 101, Ashland, MA 01721 • Tel: 1-800-445-3914 • Fax: 1-617-203-9005 Email: info@uTest.com
Overview
Second Rotation, a web-based company providing a site for people to sell consumer electronics needed a solid and scalable solution to their growing software testing needs. As a proponent of the Agile development methodology, they release new software versions as often as possible, typically on a two week sprint schedule. Second Rotation selected uTest to meet their need for professional QA testing to accommodate fast turnaround schedules, on-demand access to professional testers, and flexibility to accommodate both budgetary and resource constraints uTest, the Software-as-a-Service (SaaS) marketplace for software application testing, provides these services via Pay-for-Performance model, thus answering Second Rotation’s needs in a cost-effective manner, allowing for easy management and full visibility of all testing activity.
About Second Rotation
Second Rotation was founded in 2006 with the goal of building the easiest destination on the web for people to sell consumer electronics and gadgets without the risks and challenges that come with traditional peer to peer selling online. The experience is designed to be as easy as possible for the seller. After the seller answers a few questions about the condition of an item, Second Rotation makes an offer of what they will pay for the item, and automatically arranges for shipment. By buying unwanted gadgets and putting them into the hands of people who still value them, Second Rotation also impacts the problem of ewaste – by extending the useful life of items that would otherwise end up in landfills. If an item has no market value, Second Rotation will ensure it is recycled responsibly.
Second Rotation is privately-held, with headquarters just outside Boston, MA.
190 Front Street, Suite 101, Ashland, MA 01721 • Tel: 1-800-445-3914 • Fax: 1-617-203-9005 Email: info@uTest.com
Agile Development at Second Rotation
“We are an Agile shop, completely,” stated James McElhiney, Second Rotation’s Chief Technology Officer. “From the very beginning we developed according to Agile. We didn’t call it Agile back then, it was just the natural way that we worked.” With a development team that includes in-house software developers as well as an off-shore development team, Mr. McElhiney schedules two-week release cycles, or sprints, allowing for a continuous update of their website. Almost all of their codebase is built using the Ruby on Rails development framework. Second Rotation keeps all their development – and most of their releases – in this two-week structure. By keeping to a strong Agile methodology, all new features have been able to be added on this schedule, without the need for any long-cycle architectural rework. On a few occasions, the company chooses to collect multiple features to be released concurrently, for marketing purposes. However, even in these cases the development process maintains their sprint schedules, with only the actual release to production delayed.
The Testing Challenge: Deliver on Promise of Easy Selling Process
The most important characteristic of Second Rotation’s offering is the absolute ease by which a first-time seller can execute a sales transaction. After all, it is this implicit promise that they offer, and this promise is what makes them stand out. To truly deliver on this promise, Second Rotation identified early that they need a robust testing process, and that the testers must represent true consumers, in their wide spectrum of background and experience. This requirement still had to be balanced with the need to fit into the company’s Agile sprint cycles. Any testing process, no matter how detailed and how wide the coverage, would be of no use if it required long set-up and execution times, thus stretching or even breaking the healthy Agile process already in place.
190 Front Street, Suite 101, Ashland, MA 01721 • Tel: 1-800-445-3914 • Fax: 1-617-203-9005 Email: info@uTest.com
Solution: Community-Based Testing
Second Rotation bootstrapped the company without a formal QA infrastructure. “At the time, we had no dedicated QA in place. The Product Manager would test each release, and would grab whoever was available to help out. It was effective for ‘smoke-test’ issues, but it was not sufficient for fulfilling on our vision of a smooth and easy user experience.” Second Rotation chose uTest as their primary means of software quality testing. This decision was easy to make because it represented a potential for QA efficiency, but did not require a long term commitment, thus keeping the option open to grow an in-house QA team to complement the community in the future. Once starting with uTest, Second Rotation immediately had access to a vast testing community. With the two-week cycles, Second Rotation found that a typical testing cycle was best handled with 15-20 testers per release – which offered a coverage map much wider than they would have been able to build out in-house. Although uTest offers the option of targeting certain tester profiles (by experience, environment, platform, geographic location, etc.), Second Rotation intentionally chose not to implement these limitations, thus creating a testing community that better reflected their “true” user base. “I really like the uTest model. It lets us scale correctly. I don’t have to hire and manage a whole QA staff.”
Community Testing in the Agile Sprint Cycle
uTest serves as a full virtual QA team at Second Rotation. The two-week development cycles include two testing phases. The first week, which consists of the main development, ends on a Friday. At the end of the day, the release is posted and the uTest community starts working. By early Monday morning, the testing is complete, with a typical
190 Front Street, Suite 101, Ashland, MA 01721 • Tel: 1-800-445-3914 • Fax: 1-617-203-9005 Email: info@uTest.com
release generating 40-50 verifiable bugs. The development team then takes the second Monday-to-Friday workweek for bug fixing and code maintenance. Again, the workweek ends with a release to uTest, for a final pre-release test. Although the Agile process results in a consistent release cycle, Second Rotation enjoys the fact that the team is available and utilized on-demand, allowing for shifts in schedule with no financial impact. Today, the uTest process is overseen by a part time QA manager, who is responsible for test plan creation and reviewing the Monday-morning test reports. “uTest works really well with the fast pace of Agile. We never lose a step, since we release when we leave for the weekend and results are waiting for us when we come back in on Monday.”
ROI of Community Testing
Second Rotation has identified Return on Investment (ROI) across numerous metrics – software quality/bug quantity, direct manpower costs, overall QA budget, and time-to-market revenue benefits. The ROI on manpower has had a noteworthy affect on developer productivity. "Prior to uTest, we had our business people and developers stepping into the inhouse testing effort. While this can be healthy once in a while, it definitely isn’t sustainable – both in terms of budget as well as morale and productivity.” With uTest delivering complete tests on 24-48 hour turns, Second Rotation productivity has jumped dramatically. In addition to the direct ROI of productivity and budget efficiency, Second Rotation can also be confident that each new feature launch is not risking a loss of customers that would result from missed bugs. “Our version releases suffered from poor quality and insufficient testing prior to engaging uTest. uTest has improved our product quality and helped us eliminate critical, show-stopper bugs. We’re now seeing a noticeable increase in product quality – prior to customer deployment – in part because uTest has allowed us to increase testing coverage, while decreasing the amount of bugs, manpower, time and money spent on testing.”
190 Front Street, Suite 101, Ashland, MA 01721 • Tel: 1-800-445-3914 • Fax: 1-617-203-9005 Email: info@uTest.com
Conclusion
With a short-cycle software release policy, built on the Agile development methodology, Second Rotation needed a testing process that ensured extremely high software quality, fast testing turnarounds with flexible scheduling, and ondemand utilization to meet special peak time tests. By teaming with uTest, Second Rotation achieved each of these goals. A virtual team of professional testers delivers high-value QA test results. This virtual team executes full test plans on a weekend schedule, starting work on Friday evening with a full test report complete by the time Second Rotation returns to work Monday morning. QA resources can be expanded or eliminated according to specific peak-time and down-time needs. Since teaming with uTest, product launches that contain undetected show-stopping behaviors has been eliminated. By reducing test times and increasing product quality with an improved QA and testing process, product release cycles have been reduced significantly. Furthermore, the solution has proven to be extremely costeffective. This is apparent not only in terms of total budget, but also in the per-bug pricing structure that assures both transparency as well as value.
About uTest
uTest™ , headquartered in Massachusetts, is the SaaS marketplace for software application testing – providing software companies with on-demand testing services by a global community of professional testers. Coupled with its unique payfor-performance business model, uTest delivers a powerful “real-world” costeffective solution providing a full range of testing services – functional, usability, load, and performance testing – throughout the product development life cycle. For more information about uTest, please visit www.utest.com
190 Front Street, Suite 101, Ashland, MA 01721 • Tel: 1-800-445-3914 • Fax: 1-617-203-9005 Email: info@uTest.com