Dan Olsen: Product Development 101: Designing & Optimizing the DNA of a Killer App

Designing & Optimizing the DNA of a Killer App Dan Olsen, CEO & Founder, YourVersion Startonomics · San Francisco · Oct 2, 2008 Copyright © 2008 Olsen Solutions LLC Building the DNA of a Killer App with Metrics and Optimization How to integrate metrics & optimization  Using metrics to optimize value creation  Value  Customer Value  UI Design   Business Illustrate with real world examples Copyright © 2008 Olsen Solutions LLC Adding Metrics and Optimization to your Product Process Plan Business Objectives Product Objectives Prioritized Feature List Site Level Scoping Feature Level Design Develop Optimize Requirements & Design Code Test Launch Metrics & User Feedback Copyright © 2008 Olsen Solutions LLC Optimization through Iteration: Continuous Improvement Measure the metric Analyze the metric Identify top opportunities to improve Design & develop the enhancement Learning Gaining knowledge: • Market • Customer • Domain • Usability Launch the enhancement Copyright © 2008 Olsen Solutions LLC Optimizing Business Value Creation with Metrics Copyright © 2008 Olsen Solutions LLC Approaching Business as an Optimization Exercise Given reality as it exists today, optimize our business results subject to our resource constraints. Copyright © 2008 Olsen Solutions LLC Define the Equation of your Business “Peeling the Onion” Profit = Revenue - Cost Unique Visitors x Ad Revenue per Visitor Impressions/Visitor x Effective CPM / 1000 Visits/Visitor x Pageviews/Visit x Impressions/PV New Visitors + Returning Visitors Invited Visitors + Uninvited Visitors # of Users Sending Invites x Invites Sent/User x Invite Click-through Rate Copyright © 2008 Olsen Solutions LLC How to Track Your Metrics  Track each metric as daily time series Date Unique Visitors Page views Ad Revenue New User Sign-ups … 4/24/08 4/27/08 … 10,100 10,500 29,600 27,100 25 24 490 480  Create ratios from primary metrics: X / Y Example: How good is your registration page?  Okay: # of registered users per day  Better: registration conversion rate = # registered users / # uniques to reg page  Copyright © 2008 Olsen Solutions LLC Sample Signup Page Yield Data Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Changed messaging Added questions to signup page Started requiring registration Daily Signup Page Yield 0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0 Copyright © 2008 Olsen Solutions LLC Using Metrics to Optimize the Equation of your Business     What are the metrics for your business? Where is current value for each metric? How many resources to “move” each metric?  Developer-hours, time, money Which metrics have highest ROI opportunities? Metric A Good ROI Return Return Metric B Bad ROI Return Metric C Great ROI Investment Investment Investment Copyright © 2008 Olsen Solutions LLC Case Study: Optimizing Viral Loop Metrics % of users sending = 15% invites Invites per sender = 2.3 Invite click-through rate Active Users % of users who are active Invite Prospective Users Don’t Click Click Registration Process Fail Succeed Conversion = 85% rate Users • Multiplied together, these metrics determine your viral ratio • Which metric has highest ROI opportunity? Copyright © 2008 Olsen Solutions LLC Maximum Upside Potential of a Metric ? 100% 100% 85% 15% 0 0 % of users sending invitations 2.3 0 Avg # of invites sent per sender Registration conversion rate Max possible improvement 0.15 / 0.85 = 18% 0.85 / 0.15 = 570% ? / 2.3 = ?% Copyright © 2008 Olsen Solutions LLC Doubling Number of Invitations Sent per Sender by Adding Address Book Importer Copyright © 2008 Olsen Solutions LLC Optimizing Customer Value Creation with Metrics Copyright © 2008 Olsen Solutions LLC The Fuzzy Math of Customer Value  You create customer value with a product that Satisfies customers’ needs  Is easy to use  Has a good price  Is better than other alternatives     Customer value = customer satisfaction Not fuzzy: # of users and frequency of use Understanding user needs and satisfaction Quantitative: site analytics, usage metrics, surveys  Qualitative: support emails, usability sessions (best)  Copyright © 2008 Olsen Solutions LLC What is Your Value Proposition? Which user benefits are most important?  How well does your product deliver?  User Benefit Benefit 1 Benefit 2 Benefit 3 Benefit 4 Benefit 5 Benefit 6 Doesn’t matter Importance to User Low High Current User Satisfaction – + Upside Potential High High opportunity Low ? Med opportunity Low ? Low Copyright © 2008 Olsen Solutions LLC Analyzing Product Ideas by ROI Return (Value Created) 4 3 Idea A ? Idea D Idea B Idea C 2 1 Idea F 1 2 3 4 Investment (developer-weeks) Copyright © 2008 Olsen Solutions LLC Prioritizing Product Ideas by ROI Return (Value Created) Idea C 8 7 6 5 4 3 2 1 Idea B Idea A 1 2 3 4 5 Investment (developer-weeks) Copyright © 2008 Olsen Solutions LLC The UI Design Iceberg What most people see and react to Visual Design Interaction Design What good PMs and Designers think about Information Architecture Conceptual Design Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net Copyright © 2008 Olsen Solutions LLC Approaching UI Design Analytically  Typical UI design question: “When using web pages, do users scroll down?” - Yes - No    UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics Impact can be positive, negative, small, large  Seek high-ROI UI changes  Copyright © 2008 Olsen Solutions LLC Put Key Conversion Actions Above The Fold Landing Page A Landing Page B Key conversion action is above the fold The Fold Key conversion action is below the fold Copyright © 2008 Olsen Solutions LLC The Fold Isn’t Binary Either 768 px % of Users The chrome steals about 170 pixels 600 px 1024 px Data courtesy of ClickTale Free trial at www.clicktale.com Copyright © 2008 Olsen Solutions LLC Case Study: Account Signup Process Redesign 100% 100% 80% Biggest drop 62.3% 58.8% 50.9% % of Users 60% 40% 34.4% 32.7% 20% 0% Sign in / Account Type Cash vs. Registration Margin 5 Partner Pages 3 Partner Pages Copyright © 2008 Olsen Solutions LLC Analysis of Sign In/Registration Flow Open Account 44% 55% (24% of Total) Register Registration Process 45% drop off (20% of total) 64% of Total 36% overall drop off for this step Account Selection 56% Sign in 83% (46% of Total) 30% (14% of Total) Forget Password 17% drop off (10% of total) 70% (32% of Total) Change Password 20% drop off (6% of total) 80% (26% of Total) Copyright © 2008 Olsen Solutions LLC Redesigned User Flow Improved Registration Conversion Rate Abandonment Rate (7 Day Moving Average) Steps 1-2 80% 70% Abandon ment Rate (7 Day Moving Averag e) 60% 50% 40% 30% 20% 10% 0% 1/6/03 1/13/03 10/14/02 10/21/02 10/28/02 11/11/02 11/18/02 11/25/02 12/16/02 12/23/02 12/30/02 1/20/03 10/7/02 11/4/02 12/2/02 12/9/02 Released New Design 37% improvement in conversion rate Copyright © 2008 Olsen Solutions LLC Summary: Building the DNA of a Killer App  Define what success means  Customer value proposition  Equation of your business Instrument your site and track key metrics  Identify opportunities and prioritize by ROI  Launch, learn, and iterate  Copyright © 2008 Olsen Solutions LLC Questions? www.yourversion.com dan@yourversion.com Copyright © 2008 Olsen Solutions LLC Appendix Copyright © 2008 Olsen Solutions LLC Taking Optimization to The Next Level Multivariate testing, aka A/B testing  New challenges emerge   Are we at local vs. global optimum?  How do we identify top new ideas to test? (creativity and innovation)  Are there assumptions we should challenge?  Are we missing the forest for the trees? Copyright © 2008 Olsen Solutions LLC Olsen’s Hierarchy of Web User Needs (adapted from Maslow) Customer’s Perspective Increasing Satisfaction What does it mean to us? Usability & Design Feature Set How compelling and easy to use is the functionality? Does the functionality work? Absence of Bugs Decreasing Dissatisfaction Is the site fast enough? Page Load Time Is the site up when I want to use it? Uptime Copyright © 2008 Olsen Solutions LLC Kano Model: User Needs & Satisfaction User Satisfaction Delighter (wow) Performance (more is better) Need not met Must Have Needs & features migrate over time Need fully met User Dissatisfaction Copyright © 2008 Olsen Solutions LLC Importance vs. Satisfaction Ask Users to Rate for Each Feature 100 95 90 85 Great 84 87 98 Bad 55 70 72 75 79 80 80 86 84 Importance 80 75 70 65 60 55 50 40 50 41 60 70 Satisfaction 80 90 100 Recommended reading: “What Customers Want” by Anthony Ulwick Copyright © 2008 Olsen Solutions LLC Scalability for Startups: How to GROW Up Without BLOWing Up Frank Mashraqi http://Mashraqi.com