The Internet Services Disruption and Going “Live”

Document Sample
The Internet Services Disruption and Going “Live” Powered By Docstoc
					How Microsoft does “Software Engineering”

Myung Ho Kim National Technology Officer (NTO) Microsoft Korea

 Product

Group Organization  Engineering Software Products  Hard Decisions and Wisdom

Product Group Organization

Group Division Feature-set Product Unit Program Management Build Test BVT
Testing Quality assurance

Product Unit Marketing Dev QFE

User Ed & Localization
Architecture Implementation

A „Typical‟ Product Group
Developers  45% Testers  10% Program Management  10% User Education / Localization  7% Marketing  3% Overhead
 25%

The Developer Division (DevDev)

Has about 2,200 people Visual Studio / .NET Framework: 30+ Product Units
.NET Client ClickOnce MSDN Setup TS: Architect Visual Studio for Office VC++ VS Data WebPT xM .NET SDK DDUE NCL DataWorks TS: Dev/Test VB VJ# VSDE WMI Localization

.NET CF CLR DDCPX PSDK Team System (Overall) TS: Team Foundation VC# VS Core WebData XML XES DirectoryServices

VS 2005 & .NET Framework 2.0
Over 8 million lines of code  Over 15,000 files  Over 150 binaries

Typical Project Cycle

Vision statement

Feature complete





Release Testing

Technology Preview

Beta 1

Beta 2


Planning Phase
Marketing gathered requirements.  Senior leadership provided an initial vision.  A cross discipline team created a vision statement and elaborated on product themes.  The same team created an overall schedule.  Product Units began a detailed planning of what features would get into the release.

Scheduling *

Milestone = unit of product cycle progress
– Enable accurate assessment of progress and distance left


Milestone features are scheduled in priority order
– Enable flexible scheduling to respond to feedback later

Milestone is done when quality exit criteria is met
– Forcing function to ensure team doesn’t go fast and loose – Ensure very stable, usable, and complete product at exits

Key Feature Milestone Events
Milestone Event
Spec Complete Feature Coding Code Complete (CC) Test Plan Complete Test Case Code Complete ZBB Test Pass (ZBB TP) Zero Bug Bounce (ZBB) Zero Resolved Bugs (ZRB) Test Sign-Off

Date design specs for milestone features written & reviewed Typically 8-9 weeks in length for feature milestones Date all features scheduled for the milestone are finished Test plans for all milestone features written and reviewed Date all test cases in the Test Plan are completed All existing functional tests are run on current builds # of milestone bugs older than 48 hours = 0 # of resolved milestone bugs to be verified before closing = 0 Final verification and media sign-off on the milestone build

Design Specs

The feature-set is driven by Program Managers
– Own the customer experience and making the right thing happen – Drive the product unit team during the design process – Own writing a design specification for each feature


Developers use a variety of tools for coding.
– Visual Studio, Emacs (huh!), vi (huh!!), Source Insight… – No tool-specific files or code-injection allowed to be checked-in


Maintain single source tree for entire product line.
– Can type “build -c” from root to build from scratch – Can type “build -c” in any sub-directory of source tree


Each team maintains a private branch off main tree.
– Use internal source control system optimized for branching/merging – Minimizes check-ins into main tree and avoids breaks

C++ & C# are the 2 common languages used.  The goal is efficient, clean, and maintainable code.

– Code we ship lives at least 10 years guaranteed support

Documentation and resource strings are stored externally from source code.
– Avoid documentation writers colliding with developers – Localization work handled centrally in Ireland & Japan – .NET Fx is localized into 34 languages, VS 2005 into 8

All code check-ins must be reviewed by another developer on the team prior to submitting any changes to the source tree.  Dev is responsible for “check-in suites” to exercise implemented features.  All product check-in suites must run and pass 100% before any check-in can be submitted into source tree (2-3 hour process)  “Check-in mail” is sent to the team summarizing changes made, bugs fixed, and files modified.

Producing Daily Builds *

A new “build” is built and released every day.


Central build lab produces daily builds for entire division

– Forces engineering discipline, critical in the end-game – Build number indicates build date (40730: Jul. 30th)


Build process:
– – – – – –

– Staffed 24 hours a day throughout the week

Build team syncs source code tree (~mid-night) Kick off end-to-end build (~4:00am) Build complete (~1:00pm) Perform BVT run to verify build works (~4:00pm) Pick up hot-fixes & re-build for BVT failures Repeat hot-fix/BVT cycle until the build passes clean


Test team is staffed by Devs
– Designs test plans, writes tests, and builds infrastructure


Target goal with tests is to reach >70% product code coverage by end of the product cycle
– We measure this statistic throughout the product – In Visual Studio 2005 - 62% (native) and 70% (managed)


The Visual Studio 2005 test status:
– 10,339,207 functional test cases – ~9,000 servers in test lab to run them in automated way – A full test pass takes 21 days


Test management system manages test plans/runs

“Mad-dog” Test Automation System

Bug Tracking

Bugs & work-items are tracked within an internal bug-system “Product Studio”.
– Enables rich reporting and change history tracking

 

Bugs triaged by feature leads and assigned priority
– Bugs fixed in priority order of range from 0 to 4

Daily status mails sent to entire division tracking bug status
– After the final feature-milestone is completed, life of the product team revolves around driving it to zero.

Ship Mode: Coasting In

Large projects “coast in” r.t. ending suddenly.
– You can’t dock a large ship abruptly.

Key set of steps along the “coasting path”
– – – – – – Lock feature-set and stop adding/changing design Run a full test pass to find all bugs of the locked design. Push for a zero bug bounce (ZBB) on the locked design. Absorb bug-bounce over few weeks from ZBB. Triage all bugs out of the system that are not “must fix”. Enter End-Game mode & start minimizing code churn.

End-Game Mode

“War Team” takes over to the End-Game mode.
– Made up of most senior team members, all with ship experience. – Guides the ship in into port. – Meets at least once a day,  twice a day at the very end.


Over the span of the final weeks before a release
– War team steadily raise the triage bar, and towards the end only allow “showstopper” bugs to be fixed. – If product is ready to ship, the number of showstopper bugs will slowly decline as check-in rates drop.

End-Game Mode

Shipping is more of an art r.t. a science
– Senior leaders need to feel that the product is right, have confidence in test coverage, customer scenarios…


Once the rate of incoming bugs gets to zero, put the build in escrow for a few days, and wait & see.
– If it holds, ship it. – If not, repeat until we are finally there.

Milestone ends  Rinse and repeat (Iterative process!)

– Visual Studio 2005 went through M0 (planning), M1, M2, Beta 1, M3.a, M3.b, M3.c, Beta 2, RTM

From Bugs to Builds
Component triage Dev coding & Unit testing Buddy testing Code check-in

Bug found or escalated

War Team

Daily build

Build available


BVT Dogfood

Customer Connections

We realized that our customers need to be part of our development.
– We had to break the pattern that let us think that “we know what’s best”.


Four areas of focus:
– – – – Product Feedback Center (Ladybug) Self-sustaining Communities (Forums) Community Technology Previews (CTP) Technical Assistance Program (TAP)


Refer to site

Hard Decisions

The Golden Rule
– Features + Quality = Time ‫ ‏‬You can only control two from these three!

 

Development focus had (always) been Q & F.
– Don’t worry too much about the schedule

Front Loading Quality
– Maintain a very high quality standard throughout the project to mitigate F/Q/T tradeoffs – No feature work items get checked-in until they are completely passing tests; there is always next release

More Wisdom of the Ages

Products ship with bugs...
– The last bug is found when the last customer stops using your product


People will always screw up a good project
– Don’t add them unless you really have to – “Mythical man month”


The world is not so static from specs thru RTM
– Resources, priorities, markets, features… will change – Focus on what ultimately matters
• Quality • Compelling to customers • Solving business problems

A Decade of Advances in Software Construction and Microsoft‟s Scorecard
2. 3. 4. 5. 6. 7.

9. 10.

Design has been raised a level Daily build and Smoke test Standard libraries Visual programming innovation Nice community of programmers The Web, for research Widespread use of incremental development Test-first development Refactoring as a discipline Faster computers
-- Source: Construx, Code Complete 2: Realities of Modern Software Construction, 2004

Software is built by People
 Understand

what’s really bad for morale

– – – –

Long hours and high stress Unrealistic schedules Feeling of low productivity Working on poor-quality software

Software is built by People

Watch out workplace demons!
– – – – – – – – – – Dishonesty Sloppiness Backbiting Intolerance Jealousy Sexism Carelessness Inability to speak to power Lack of respect for the other Blaming


Please come out from the color of the night…

Closing Remark
 Factors

that impact any successful project

– Technology