Internet Marketing Training - PowerPoint by jpg12539


Internet Marketing Training document sample

More Info
									Multivariate Web Page
Testing & Optimization

SDMA – November 2, 2005
Presentation Objectives
   Introduce You to Online Multivariate Testing
   Discuss When Multivariate Testing is Better
    than A/B Testing
   Outline the Process and Steps Required to
    Execute a Successful Multivariate Test
   Walk Through a Case Study          Internet Marketing Training & Consulting
The Players
                       RealNetworks – Consumer Products,
                       Website/Landing Pages, Audience, Paid
                       Search Program, Creative for Most Tests

                       Brand Digital, Inc. – Testing Strategy,
                       Matrix Creation, Search Program
                       Management, Analysis & Wrap-Up

                       Optimost – Multivariate Testing Platform,
                       Randomization, Online Reporting,
                       Significance Modeling & Testing Expertise

                       Peak Systems – Creative for First
                       Landing Page Test                       Internet Marketing Training & Consulting
 RealNetworks – Consumer Products
Rhapsody                             SuperPass
-Purchased in 2003        -Video News
-Subscription Music Service          -Music Downloads
-“All You Can Eat” Pricing           -Premium Player
-Discounted Downloads                -Commercial & Ad Free Radio
-Manage Entire Catalog on PC         -10 Free Songs/Mo
-Burn CDs, Create Playlists          -1 Free Game/Mo
-Personalized, Ad-Free Radio
-Music Recommendations

Arcade/GamePass                     MediaPlayer
-Casual and Arcade Games            -Premium Audio/Video
-Free & Pay per Download            -Advanced CD Burning
-GamePass is Subscription Service   -News & Entertainment Content
                                    -Music Store ($.99 downloads)               Internet Marketing Training & Consulting
 Testing, Testing, A-B-C…
Traditional A/B Testing Tells You Which One Converts Better

But doesn’t tell you why…                  Internet Marketing Training & Consulting
Multivariate Testing Tells You Why…
                       Test Each Element Individually and in
                       Conjunction with the Rest of the Page
                       - Page Layouts
                       - Headlines and Subheads
                       - Location & Color of Buttons
                       -Text & Shape of Buttons
                       - Navigation yes or no
                       - Images: size, quality, placement
                       - Bullets vs Paragraph
                       - Offers
                       - Analysis by Audience Segment

                       Instead of running hundreds of
                       sequential A/B tests to understand
                       the impact, you run these variables
                       using experimental design principles.      Internet Marketing Training & Consulting
Improvements Happen Through Iterations    Internet Marketing Training & Consulting
Three Major Components to Multivariate Testing
   Methodology
       Determines the depth of the tests that can be run and the
        robustness of results.
       What is the experimental design behind the test.
   Technology
       Determines the breadth of testing and the ease of which it
        can be performed.
       Who hosts the page, what code is used to trigger the
        variables, how is the code integrated, etc.
   Experience
       Both the experience with testing and the experience behind
        the testing solution impacts the way the test is defined,
        produced, analyzed and communicated.                     Internet Marketing Training & Consulting
The Multivariate Test Process
Planning               Development     Execution                    Roll Out
Business                               Pre-Wave Test
Objectives                                                           Move Pages to
                           Creative                                  Full Production
                         Development     Wave One
                                       Regression Analysis
 Test Plan                                                              Send Full
 Creation                                                                Traffic
                         Technology      Wave Two
                           Review      Regression Analysis

  Matrix                                                              Confirmation
 Creation                                Wave Three                     Metrics
                          Template     Regression Analysis
                          Creation                                    Case Study &
  Traffic                                                             Presentation
                                       Regression Analysis                    Internet Marketing Training & Consulting
Rhapsody Landing Page Case Study
                       Quiz: How much better did this landing
                       page work than the control?           Internet Marketing Training & Consulting
Landing Page Testing Objectives

1.     Create a landing page to boost click to trial
       conversions by at least 20%.
2.     Understand which elements of the landing
       page are the most important in terms of
       driving results.
3.     Understand the differences between the
       behavior profile of the search audiences:
       online music, artists and genres.
4.     Try a third party testing platform (Optimost)
       to understand the value and performance.             Internet Marketing Training & Consulting
Business Challenge
   The Pay Per Click (PPC) Search Program had
    grown 200% since previous summer in terms of
    Rhapsody trials while decreasing CPA 20% through
    increasing terms, managing bids more effectively
    and optimizing copy.
   The next step to growing the program was to
    improve the landing pages we used for search and
   We needed to test 1,000s of variable combinations
    to get a significantly better landing page based on a
    click to trial conversion rate.               Internet Marketing Training & Consulting
The Audience
   Traffic to the test pages came from two search
    publishers: Google and Yahoo!
   Three major groups of terms pointed to three
    primary templates: Music, Artists & Genres
   A subset of groups were within each primary:
       Music: Download, MP3, Rhapsody, Radio, Music
       Artists: Beyonce, Eminem, Kenny Chesney, etc.
       Genres: Rock/Pop, R&B/Soul, Rap/Hip-Hop, etc.
   We used a source code structure to send the right
    keyword group to the right template.
       ?src=ppc_gg_artist                  Internet Marketing Training & Consulting
The Control Page – Elements Tested
                       Top Navigation
                       Primary Image

                       Offer Copy

                       Button Color and Call to Action
                       Supporting Points
                       Additional test items:
                       • Page Layout
                       • Flash Tour
                       • Artist List
                       • Screenshots
                       • Move Proof Points Above the Fold
                       • Separate Offer Box         Internet Marketing Training & Consulting
  Example of a Test Matrix (first two rows only)

• Each value in each variable gets its own cell
• Include the text or the absolute link to the creative
• Go through it one more time with executives                               Internet Marketing Training & Consulting
                        Pages Optimosted:
            Artist, Online and Genre Landing Pages

                       •   Test conducted from
                           5/12/04 to 07/01/04
                       •   5 layout templates
                       •   23 variables tested
                       •   124 values tested
                       •   1,093,680 template
                           permutations analyzed                      Internet Marketing Training & Consulting
Testing Methodology
1.        Divided each landing page layout into buckets
2.        Assign up to 7 variables to each bucket.
3.        Tested in three waves:
     1.      Wave 1 – Focused on same variables with different page layouts
             AND a single new layout with extreme variables to discern which
             sections created the biggest differences.
     2.      Wave 2 – Focused on the winning layout vs control and
             additional values for variables
     3.      Wave 3 – Narrowed down to top performing values for each
             variable in validation wave against control.
4.        Statistical significance was targeted at the 95% level inter-wave
5.        Success metrics were defined at two levels:
     •       Order Path Entry Rate (the first page after the landing page) – Used
             when not enough traffic made it to the trial thank you page.
     •       Trial Rate – reaching the trial thank you page – the PRIMARY
             success event.                            Internet Marketing Training & Consulting
   Optimost Reporting Interface
Browser based reporting shows avgs in almost real time include clicks, conversions, creative and traffic
Inter-wave reports are more succinct and detailed with specific wins/losses and recommendations                                         Internet Marketing Training & Consulting
     Topline Result Summary

    Winning creatives increased conversions to Order Path Entry
     pages by over 160% on all 3 pages (Online Music, Genre,

    Winning creatives increased actual registrations on Online
     Music page by 98.4% (increases on other pages were also
     very large, but sample size was too small to be significant)

    Winning elements were very consistent across Music, Online,
     and Genre pages

    Offer, Call-To-Action (text on the button), Testimonial
     Placement and Core Bullet Area are particularly critical to
     improvements in conversion                       Internet Marketing Training & Consulting
                 Winning layout were Significantly Better
                   than control and alternate layouts.


     Control Layout           Alternate Layouts          Template Baseline Layouts                          Internet Marketing Training & Consulting
            “Over 600,000 Songs” and “Ultimate Digital Music
                 Experience” outperformed the control


                 Control                      Internet Marketing Training & Consulting
Multivariate Testing Lessons Learned
   Get input from IT, Creative, Execs, Marketing early on to gain support and
    ideas for what should be tested.
   Test the widest conceivable options as well as the shades of gray.
   Make sure your test traffic is the same as your “live” traffic so that the results
    are consistent between the two.
   Make sure you have enough traffic to support the amount of testing you want.
   Think about starting smaller and building up to larger tests.
   Control for important elements like page weight in addition to the various
    messaging you want to test.
   Don’t be afraid to test audience segments
   Don’t be afraid to test shopping carts, forms or rich media content like flash or
    navigation elements.
   Be prepared to learn what doesn’t work as well as what does – especially in
    the first wave.
   Report back to the wider group with a case study, lessons learned and next
    testing ideas so that the lessons can be built into other products/programs.
   Keep a journal of tests and lessons learned. Centralize the information.                              Internet Marketing Training & Consulting
So, When Should You Multivariate Test?
1. If you have a lot of ideas or questions on how to sell a
   product or service
2. If you want to understand the relationship between
   variables as well as which ones perform better
3. If you need results that show the best performing
   combination of elements sooner than later (vs a series of
   A/B testing)
4. If you have a significant stream of traffic - the more traffic,
   the more variables and values you can test sooner.
5. If you want see performance by segment – the analytical
   and reporting tool must support this.
                   Multivariate Testing is NOT Always the Answer.
                   Use it as an Additional Tool in Your Testing Kit.                             Internet Marketing Training & Consulting
More Information
       Try Rhapsody for Free – 14 Day Trial
       Try SuperPass for Free – 14 Day Trial
       Workshops on Search Engine Marketing, Conversion and
        Customer Acquisition
       Multivariate testing platform, case studies, info
       Web creative and hosting                       Internet Marketing Training & Consulting

To top