Online Research Quality

Document Sample
Online Research Quality Powered By Docstoc
					 Update                 Online
                        Research
                        Quality
Advertising Research Foundation’s Online Research Quality Council
NEW LEADERSHIP
Stan Sthanunathan of Coca Cola Takes Charge
Findings from the FoQ study are already paying dividends, leading to a better understanding of ways to boost data qual-
ity and the improvement of online survey response quality, while enhancing the reliability of results.


W       e are proud to issue the second ORQC
        Newsletter to inform the industry on
progress being made against the issues and
                                                    impact the industry in a big way. The need
                                                    of the hour is an industry solution. But, we
                                                    do not have the luxury of waiting for a long
                                                                                                      ers and sellers to work together to bring the
                                                                                                      industry to a better place in terms of data
                                                                                                      quality, comparability, and the trustworthi-
opportunities addressed by the ARF’s Online         time to come up with a perfect solution. Let      ness of online research results.
Research Quality Council. As a reminder,            us set our respective agendas aside and work
we originally formed the ORQC in order to           together to resolve this challenge.”
create a positive and constructive forum to            Robert Tomei, Founder and Co-Chair of
bring together the leading buyers and sellers       the ORQC adds: “Since the very beginning
of online research to effectively address the       of the ORQC, Stan has been a tremendous
key quality-related issues and opportunities        advocate of developing a solution based on
being faced by the market research indus-           the facts, and now that we have achieved
try. Since the official launch of the council,      that with the ‘Foundation of Quality’ study,
we have achieved active participation across        we can move into identifying solutions that
the industry with 20 client companies, more         the Industry can adopt to address the real is-
than 20 research agencies, and seven other          sues around online research quality. Having
professional associations, all working toward       the right leaders and companies represented
a common goal.                                      on the ORQC will ensure that we achieve this        Stan Sthanunathan
                                                                                                        Co-Chair, Online Research Quality Council
    Taking the helm of the ARF’s Online Re-         objective”.                                         VP, Global Knowledge & Insights, Coca Cola
search Quality Council (ORQC) at a new                 Stan has instilled a sense of urgency that
confluence of insights, solutions, and metrics
is Stan Sthanunathan, Vice President of Mar-
                                                    respects the hard work of the ARF and indus-
                                                    try leaders who are contributing their time.
                                                                                                        “Stan has been a
keting Strategy & Insights at Coca-Cola. Stan       This collaborative commitment to a “90-Day           tremendous advocate
assumed this key role on June 9th, 2009 - an        plan” will culminate in a presentation to the
opportune time as definitive results from the       industry on September 29 of a Version 1.0            of developing a
ARF-led Foundations of Quality (FoQ) re-            Industry Solution for Online Data Quality.
search-on-research project became available.        With the cooperation of other industry as-           solution based on
    Stan, a proponent of an action agenda,          sociations, the ARF will ultimately provide
says: “Having lived through quality chal-           a program with recommendations regarding             the facts ...”
lenges, I have a very high level of passion for     metrics and business practices, templates,            Robert Tomei – IRI & Co-Chair, ORQC
this subject. I firmly believe that our work will   definitions, and training that will enable buy-
RESULTS
FoQ Project Begins To Bear Fruit
The ARF’s Foundations of Quality (FoQ) project begins to bear fruit with the creation of ARF Knowledge Briefs: executive
reports based on the FoQ research-on-research results.


I  n today’s world of swift and rapid change,
   knowledge is power. Just a few short weeks
after the presentation of the preliminary
                                                  The Knowledge Briefs are available to ARF
                                                  Members on MYARF and to non-members by
                                                  request and for a fee.
                                                                                                      the FoQ initiative moves into the next phase,
                                                                                                      ESPN, as users of online research, is excited
                                                                                                      to see that the ORQC has committed to mov-
results of the FoQ study at the ARF’s 2009            The Foundations of Quality study collect-       ing quickly to address the original study’s re-
Annual Re:Think Conference, topical papers        ed over 100,000 completed survey responses          sults and how they impact what we think we
based on the FoQ results were created by Bob      from 17 panels in a rigorously designed re-         know and what we do not know about online
Walker, President of Surveys & Forecasts,         search project. The question with the most          data quality.”
LLC, Joel Rubinson, Chief Research Officer        important implications for the industry was:
and Dr. Raymond Pettit, Senior Vice President     Why do survey results not replicate? Pan-
at the ARF. Following the careful participation   els appear to produce different answers, so
and vetting processes established at the out-     switching panels or changing the mix of pan-
set of this project, the briefs were reviewed     els can lead to results that are not compara-
by the ORQC leadership, committees, and           ble across studies. Also, panel practices vary
supporters of the project.                        widely, which influences results; for example,       FoQ Knowledge Briefs are available
    The ARF is now issuing three papers           in the FoQ study it was found that purchase          free to ARF members on My ARF and to
based on the FoQ results: (1). The Impact         intent is higher in panels that offer cash incen-    non-members for $399.
of Overlap, Multi Panel Membership, and           tives. Such panels also seem to attract more         (For questions, please contact Raymond
Duplication on Online Data Quality; (2). The      panelists who say they are motivated to take         Pettit, SVP, Research and Standards at
Response Quality of Online Surveys; and (3).      surveys for money.                                   rpettit@thearf.org).
The Consistency, Reliability, and Inter Study         Dr. Tom Evans of ESPN offers encourage-          Download or purchase at my.thearf.org.
Comparability of Online Research Results.         ment about the next steps of the project: “As



WORK IN PROGRESS
The Road to Version 1.0
The Foundations of Quality Research-on-Research Project has produced a definitive report on online research quality
improvement. Results from this study offer findings and guidelines on measuring and assessing the quality of online
research data.


T     he Online Research Quality Council’s
      newly formed Industry Solutions group,
led by Kim Dedeker, Kantar; Mark Berry, Syn-
                                                  lead a group focused on finding better ways
                                                  to manage survey delivery. Simon Hough,
                                                  Microsoft and Ashley Grace and Jeff Cox,
                                                                                                      Kraft, and Steve Segur, TNS will be coordi-
                                                                                                      nating and integrating their work on the de-
                                                                                                      velopment of key performance metrics and
ovate; and Tomas Emmers, Unilever has cre-        ARS Group will be exploring ways to improve         guidelines with the new committees.
ated three focus committees tasked to work        response quality and other related issues,             The groundbreaking work of the Industry
on the primary quality issues confronting         such as managing survey length.                     Solutions group will be blended into a Ver-
online panel research. Based on the results          In tandem, a Metrics Committee, led by           sion 1.0 Industry Solution presentation to the
of the ARF’s Foundations of Quality research      Jonathan Jephcott, Synovate, Stephen Oi,            industry on September 29.
study, extensive secondary research, and an
inspection of best practices in the industry,
these subcommittees are tackling the impor-
tant job of turning results into actions and
guidelines.
   John Willard, Bayer and Jissoon Barton,
Synovate will co-chair a group that will look
at online panel management issues with an
eye to improving consistency and reliability.
Justin Fromm, ABC and Deb Santus, TNS will
FOUR ISSUES ON THE TABLE
Our Shared Future Regarding Online Data Quality
Posted June 10, 2009 in CRO-ing About Research, ARF Chief Research Officer Joel Rubinson’s Blog.
blog.joelrubinson.net


O      n Tuesday, June 9th the ARF Online
       Research Quality Council presented
detailed findings from an unprecedented
                                                      We found two reasons worth noting for lack
                                                  of comparability but not powerful enough to
                                                  fully explain differences across panels:
US R&D project regarding online data qual-            • Panelist longevity: newer recruits are
ity, called “Foundations of Quality” (FoQ).             more likely to give positive purchase
Beyond the fact that it was about $1MM in               intent towards concepts. The longev-
research, it was unprecedented and quite                ity profile differed dramatically across
remarkable as 17 leading online panel com-              panels.
panies cooperated with each other and with            • Those motivated by monetary rewards
large buyers of marketing research in a col-            to take a survey appear to be more posi-
laborative and transparent way. We studied              tive to concepts and some panels stress
                                                                                                        Joel Rubinson
700,000 panelists and over 100,000 survey               such rewards while others do not.               Chief Research Officer

                                                  2
responses from 17 panels in a rigorously de-                                                            The Adversing Research Foundation
signed research project.                               Some feared that there is a small
    Competitors and trading partners were         group of “professional respondents”                 “Competitors and
drawn together at the ARF by a sense of ur-       on everyone’s panel, doing it for the
gency to reestablish the trustworthiness of       money, and gaming the system rather
                                                                                                       trading partners were
online research ... by our shared future.
    Let me discuss findings against four of the
                                                  than providing thoughtful answers
                                                  FoQ proves that this is not true. Overlap is less
                                                                                                       drawn together at the
big issues that were on the table.                than thought. Over 80% of e-mail addresses           ARF by a sense of
1
                                                  appear on only one panel and the collective
     Some high profile examples have              pool of unique e-mails in the US is estimated        urgency to reestablish
been reported of study results not                at something over 5.5 million. For historical
replicating. Why is that?                         comparison, this is probably 2-4 times greater       the trustworthiness of
FoQ data prove that, for each of the 17 pan-      than the pool of mail panelists that NFO, HTI,
els, results replicate within panel (within the   and Market Facts collectively had in the 80s-        online research...”
limits of sampling variation) but do not nec-     90s. Also, mostly people are motivated to join
essarily replicate across panels. This means      online research panels by a desire to share         damaging. The industry must adopt practices
that buyers will need to be cautious about        their opinions, rather than for the incentives.     that will address this.

                                                                                                      4
switching suppliers when data comparabil-         Perhaps the most telltale finding is that those
ity to other study results is a main consider-    who take MORE surveys per month (up to 10)               Are things going to get better in the
ation. Furthermore, because suppliers often       provide MORE thoughtful answers (i.e. less          near future?
draw on sample sources beyond their own           likely to straightline their answers or fail trap   Yes! We all feel the sense of urgency so The
panels, buyers must engage in conversations       questions.) In other words, if anything, being      ARF and industry leaders who have contrib-
with suppliers about any change in sample         “professional” is a GOOD thing as it relates to     uted their time and expertise will continue;
source.                                           respondent engagement with the activity of          we are committing to a 90 day plan. With the
                                                  survey-taking.                                      cooperation of other industry associations,
READ & DISCUSS: blog.joelrubinson.net                 In fact, as it turns out, the biggest cause     we will provide a program with recommen-
                                                  of not providing thoughtful answers is long         dations regarding metrics and business prac-
                                                  surveys!                                            tices, templates, definitions, and training that

                                                  3
                                                                                                      will enable buyers and sellers to work togeth-
                                                      Are people taking the same survey               er to bring the industry to a better place in
                                                  more than once?                                     terms of data quality, comparability, and the
                                                  On studies where qualification criteria are not     trustworthiness of online research results.
                                                  very restrictive, “duplication” is mostly a sin-
                                                  gle digit issue (depends on the panel or pair
                                                  of panels) but if the qualification criteria are
                                                  more specialized, it can more prevalent and
SAVE THE DATE – SEPTEMBER EVENT
The Foundations of Quality Version 1.0
Industry Solutions Plan
A    very important meeting is set for Tuesday, September 29th, 2009, from 2-4
     PM, at the Time Warner Center’s Screening Room. The culmination of nearly
two years of work on the Foundations of Quality initiative will be presented to the
                                                                                                        An ORQC Special Meeting
                                                                                                        Presentation of the FoQ Version 1.0 Industry
                                                                                                        Solutions Plan
industry. This meeting will be open to ARF members and non-members, but due
                                                                                                        Tuesday, September 29 • 2:00–4:00pm
to space limitations we will be requiring registration. Register at my.thearf.org.
                                                                                                        1 Time Warner Center
                                                                                                        (58th St., between 8th and 9th Avenues)
                                                                                                        Screening Room, 10th Floor
                                                                                                        New York, NY 10019




 ORQC Governing Board
 CO-CHAIR Stan Sthanunathan   VP, Global Knowledge & Insights                                                    Coca Cola
 CO-CHAIR Robert Tomei        President, Consumer & Shopper Insights                                             Information Resources, Inc.
 John Willard                 Director, Market Research & Consumer Relations                                     Bayer Consumer Care
 James Mendelsohn             Vice President, Corporate Strategy                                                 Capital One
 Bryan Mahlmeister            Global Consumer & Product Research; Senior Manager, Study Design & Fieldwork       General Motors
 Steve Oi                     Sr. Consumer Insights Manager                                                      Kraft
 Kim Dedeker                  Chair, North America                                                               Kantar
 Efrain Ribiero               COO                                                                                Lightspeed Research
 Steven Schwartz              Group Vendor and Quality Manager, Central Marketing Research and Insights          Microsoft
 Steve Coffey                 Chief Research Officer                                                             NPD
 Bob Walker                   Principal                                                                          Surveys & Forecasts
 Steve Segur                  Vice President, Panel Development & Research                                       TNS
 Tomas Emmers                 Consumer & Market Insights Director                                                Unilever Americas
 Tom Evans                    VP, Audio Research and Special Projects                                            ESPN
 Renee Smith                  VP Global Operations                                                               Ipsos
 Mark Berry                   EVP Consumer & Business Insights                                                   Synovate
 Jonathan Jephcott            Executive Vice President                                                           Synovate



 Industry Solutions Committee
 CO-CHAIR Kim Dedeker         Chair - North America                                                              Kantar
 CO-CHAIR Mark Berry          EVP Consumer & Business Insights                                                   Synovate
 CO-CHAIR Tomas Emmers        Consumer & Market Insights Director                                                Unilever
 Ashley Grace                 President, Strategy and Partnerships                                               ARS Group
 Birgi Martin                 Advertising Research Manager                                                       Google
 Debra Santus                 Senior VP, IIS NA Access Panel                                                     Ipsos
 Don Gloeckler                Manager, Next Gen Research                                                         Procter & Gamble
 Dyna Boen                    Vice President, TrueSample                                                         MarketTools, Inc.
 Jackie Lorch                 VP, Global Knowledge Management                                                    SSI
 James Mendelsohn             Capital One                                                                        Capital One
 Jeff Cox                     CEO                                                                                ARS Group
 Jisoon Barton                Senior Vice President                                                              Synovate
 Joe Giacobbe                 SVP, Operations & Services                                                         Vision Critical
 John Ouren                   EVP/GM - Zoom Business                                                             Market Tools
 John Willard                 Director, Market Research & Consumer Relations                                     Bayer - Consumer Care
 Jonathan Jephcott            Executive Vice President                                                           Synovate
 Kate Whittington             Group Manager, Guest Insights                                                      Target
 Kevin Umeh                   CEO                                                                                Cint
 Mainak Mazumdar              SVP, Global Measurement Science/Emerging Media                                     Nielsen
 Marjette Stark
 Matt Dusig                   CEO / co-founder                                                                   United Sample
 Simon Hough                  Senior Advertising Research Manager                                                Microsoft
 Simon Kooyman                CEO                                                                                Knowledge Networks
 Susan Frede                  Vice President Research & Panel Management                                         TNS
 Steven Oi                    Sr. Consumer Insights Manager                                                      Kraft
 Steve Segur                  VP, Panel Development & Research                                                   TNS
 Ted Christenson              Consumer Insights Manager                                                          General Mills
 Thomas Evans                 Vice President of Audio Research/Special Projects                                  ESPN