Designing & Optimizing the DNA of a Killer App
Dan Olsen, CEO & Founder, YourVersion
Startonomics ∙ San Francisco ∙ Oct 2, 2008
Copyright © 2008 Olsen Solutions LLC
Building the DNA of a Killer App with Metrics and Optimization
How to integrate metrics & optimization Using metrics to optimize value creation
Business Value Customer Value UI Design
Illustrate with real world examples
Copyright © 2008 Olsen Solutions LLC
Adding Metrics and Optimization to your Product Process
Plan
Business Objectives Product Objectives Prioritized Feature List
Site Level
Scoping
Feature Level
Design Develop Optimize
Requirements & Design Code Test Launch
Metrics & User Feedback
Copyright © 2008 Olsen Solutions LLC
Optimization through Iteration: Continuous Improvement
Measure the metric Analyze the metric Identify top opportunities to improve Design & develop the enhancement
Learning
Gaining knowledge: • Market • Customer • Domain • Usability
Launch the enhancement
Copyright © 2008 Olsen Solutions LLC
Optimizing Business Value Creation with Metrics
Copyright © 2008 Olsen Solutions LLC
Approaching Business as an Optimization Exercise
Given reality as it exists today, optimize our business results subject to our resource constraints.
Copyright © 2008 Olsen Solutions LLC
Define the Equation of your Business “Peeling the Onion”
Profit = Revenue ‐ Cost Unique Visitors x Ad Revenue per Visitor Impressions/Visitor x Effective CPM / 1000 Visits/Visitor x Pageviews/Visit x Impressions/PV New Visitors + Returning Visitors Invited Visitors + Uninvited Visitors # of Users Sending Invites x Invites Sent/User x Invite Click‐through Rate
Copyright © 2008 Olsen Solutions LLC
How to Track Your Metrics
Track each metric as daily time series
Date 4/24/08 4/27/08 … Unique Visitors 10,100 10,500 Page views 29,600 27,100 Ad Revenue 25 24 New User Sign‐ups 490 480 …
Create ratios from primary metrics: X / Y
Example: How good is your registration page? Okay: # of registered users per day Better: registration conversion rate = # registered users / # uniques to reg page
Copyright © 2008 Olsen Solutions LLC
Sample Signup Page Yield Data
Daily Signup Page Yield vs. Time New Registered Users divided by Unique Visitors to Signup Page
100% 90% 80% 70% 60% 50% 40% 30% 20% 10%
Changed messaging Added questions to signup page Started requiring registration
Daily Signup Page Yield
0% 1/31 2/14 2/28 3/14 3/28 4/11 4/25 5/9 5/23 6/6 6/20 7/4 7/18 8/1 8/15 8/29 9/12 9/26 10/1 0
Copyright © 2008 Olsen Solutions LLC
Using Metrics to Optimize the Equation of your Business
What are the metrics for your business? Where is current value for each metric? How many resources to “move” each metric?
Developer‐hours, time, money
Which metrics have highest ROI opportunities?
Metric A Good ROI Return Return Metric B Bad ROI Return Metric C Great ROI
Investment
Investment
Investment
Copyright © 2008 Olsen Solutions LLC
Case Study: Optimizing Viral Loop Metrics
% of users sending = 15% invites Invites per sender = 2.3 Invite click-through rate
Active Users
% of users who are active
Invite
Prospective Users Don’t Click
Click
Registration Process
Fail
Succeed
Conversion = 85% rate
Users
• Multiplied together, these metrics determine your viral ratio • Which metric has highest ROI opportunity?
Copyright © 2008 Olsen Solutions LLC
Maximum Upside Potential of a Metric
?
100% 85%
100%
15% 0 Registration conversion rate Max possible improvement 0.15 / 0.85 = 18% 0 % of users sending invitations 0.85 / 0.15 = 570%
2.3 0 Avg # of invites sent per sender ? / 2.3 = ?%
Copyright © 2008 Olsen Solutions LLC
Doubling Number of Invitations Sent per Sender by Adding Address Book Importer
Copyright © 2008 Olsen Solutions LLC
Optimizing Customer Value Creation with Metrics
Copyright © 2008 Olsen Solutions LLC
The Fuzzy Math of Customer Value
You create customer value with a product that
Satisfies customers’ needs Is easy to use Has a good price Is better than other alternatives
Customer value = customer satisfaction Not fuzzy: # of users and frequency of use Understanding user needs and satisfaction
Quantitative: site analytics, usage metrics, surveys Qualitative: support emails, usability sessions (best)
Copyright © 2008 Olsen Solutions LLC
What is Your Value Proposition?
Which user benefits are most important? How well does your product deliver?
User Benefit
Benefit 1 Benefit 2 Benefit 3 Benefit 4 Benefit 5 Benefit 6
Doesn’t matter
Importance to User
Low High
Current User Satisfaction
–
+
Upside Potential
High High
opportunity
Low ? Med
opportunity
Low ? Low
Copyright © 2008 Olsen Solutions LLC
Analyzing Product Ideas by ROI
Return (Value Created) 4 3
Idea A Idea B Idea C ? Idea D
2 1
Idea F
1 2 3 4 Investment (developer-weeks)
Copyright © 2008 Olsen Solutions LLC
Prioritizing Product Ideas by ROI
Return (Value Created)
Idea C
8 7 6 5 4 3 2 1
Idea B
Idea A
1 2 3 4 5 Investment (developer-weeks)
Copyright © 2008 Olsen Solutions LLC
The UI Design Iceberg
What most people see and react to
Visual Design
Interaction Design Information Architecture Conceptual Design
Recommended reading: Jesse James Garrett’s “Elements of User Experience” chart, free at www.jjg.net
What good PMs and Designers think about
Copyright © 2008 Olsen Solutions LLC
Approaching UI Design Analytically
Typical UI design question: “When using web pages, do users scroll down?”
‐ Yes ‐ No
UI questions are never yes/no! (not binary) Should ask: “What percentage of users …?” UI changes impact your metrics
Impact can be positive, negative, small, large Seek high‐ROI UI changes
Copyright © 2008 Olsen Solutions LLC
Put Key Conversion Actions Above The Fold
Landing Page A Landing Page B
Key conversion action is above the fold
The Fold
Key conversion action is below the fold
Copyright © 2008 Olsen Solutions LLC
The Fold Isn’t Binary Either
768 px
% of Users
The chrome steals about 170 pixels
600 px
1024 px
Data courtesy of ClickTale Free trial at www.clicktale.com
Copyright © 2008 Olsen Solutions LLC
Case Study: Account Signup Process Redesign
100%
100%
80%
Biggest drop
62.3% 58.8% 50.9%
% of Users
60%
40%
34.4%
32.7%
20%
0%
Sign in / Registration
Account Type
Cash vs. Margin
5 Partner Pages
3 Partner Pages
Copyright © 2008 Olsen Solutions LLC
Analysis of Sign In/Registration Flow
Open Account
44% 55% (24% of Total)
Register
Registration Process
45% drop off (20% of total)
Account 36% overall Selection
30% (14% of Total) drop off for this step
64% of Total
56%
Sign in
83% (46% of Total)
Forget Password
17% drop off (10% of total)
70% (32% of Total)
Change Password
20% drop off (6% of total)
80% (26% of Total)
Copyright © 2008 Olsen Solutions LLC
Redesigned User Flow Improved Registration Conversion Rate
Abandonment Rate (7 Day Moving Average)
Steps 1-2
80% 70%
Abandonment Rate (7 Day Moving Average)
60% 50% 40% 30% 20% 10% 0%
10/14/02 10/21/02 10/28/02 11/11/02 11/18/02 11/25/02 12/16/02 12/23/02 12/30/02 10/7/02 11/4/02 12/2/02 12/9/02 1/13/03 1/20/03 1/6/03
Released New Design
37% improvement in conversion rate
Copyright © 2008 Olsen Solutions LLC
Summary: Building the DNA of a Killer App
Define what success means
Customer value proposition Equation of your business
Instrument your site and track key metrics Identify opportunities and prioritize by ROI Launch, learn, and iterate
Copyright © 2008 Olsen Solutions LLC
Questions?
www.yourversion.com dan@yourversion.com
Copyright © 2008 Olsen Solutions LLC