Lies_ Damned Lies_ Statistics.pdf by tongxiamy


                                                         Damned Lies,
Avoiding paths that lead to the ‘Dark Side’ of analytics
The famous quote, “There are three kinds of lies: lies, damned lies, and statistics” attributed to British Prime
Minister Benjamin Disraeli and popularized by Mark Twain, always brings a knowing smile to people’s
faces.Why? The reason is that almost everybody loves to find information that furthers their cause, but many
times people bend the truth or hide other facts that might put things in a different perspective.
                                                           BY   LOUIS WINOKUR

                                                                        assignments, I worked at the headquarters of a major pharma-

   n this article, we’ll review common analytic errors to avoid
   as we work through our busy days on ad hoc requests, regu-           ceutical company that marketed many over-the-counter
   larly scheduled reports and custom experiments for patient           brands.
relationship marketing programs or DTC drug campaigns. To                   After making strides at establishing relationships with vari-
be concise, we will focus on four areas where a small invest-           ous brand and category management teams, I was approached
ment of time and effort will pay big dividends on the back end          by a key contact and asked the above question. The question
in better data collection and interpretation, which leads to            isn’t a bad one per se, but my intuition told me even before
more insightful presentations.                                          pulling the data that the flagship brand’s key demographic base
Four common errors:                                                     of people ages 55-plus would not align with the NASCAR
  1. Force-fitting answers onto existing data                           profile. Sure enough, after a quick review, I saw that the
  2. Forgetting to frame significance against sample size               NASCAR audience was much younger and had a somewhat
  3. Jumping to conclusions based on initial findings                   rural skew. So I went back to my contact and said, “It would
  4. Creating poorly thought-out presentations with pretty              appear that the sponsorship is not the best use of company
     charts                                                             funds.”
                                                                            Here’s where it gets interesting.
Force-fitting answers                                                       About two months later, the same person came back to me
“Can you show that our flagship pain reliever brand is a good fit for   and said, “Perhaps we’ve taken the wrong approach. Maybe all of
our NASCAR sponsorship?”                                                NASCAR’s sponsors are a bad fit to its fan base’s demographic!”
   In the early days of my career, I served as an onsite consult-           So, in the spirit of being open minded, I tracked down a list
ant for a company that had a unique way of segmenting syndi-            of all of NASCAR’s sponsors and conducted an extensive
cated data from major suppliers such as AC Nielsen and Infor-           examination of how they matched up with those who watch
mation Resources (IRI). During one of my longer term                    on TV or attend races in person. And guess what? There are

26 | DTC Perspectives • December 2009
                                                                                                 DARK SIDE OF ANALYTICS

tons of sponsors who are great fits for NASCAR, the No. 1               pletely different layout, using icons to explain the value propo-
being Slim Jim Beef Jerky!                                              sition and eliminating all images.
    I later found out that somebody in the organization was the             A comprehensive test methodology was established with a
champion behind that six-figure sponsorship and my contact              distinct goal and target sample size that would provide a con-
was trying to find a way to end that association when the               clusion in a specified time period.
agreement expired.                                                          To our surprise, “Form B” had the worst performance of
    If I were to step back into that role today, I would add a fol-     any of the designs. “Form C” proved to be the winner, signifi-
low-up to my findings, and recommend that they evaluate alter-          cantly outperforming “Form B” over the course of the test and
native sponsorship opportunities such as horse racing or PGA            edging out “Form A” for the highest enrollment rate over the
events against their key consumer demographic. Alternatively, I         test period. (See related “Enrollment Rate” chart.) So, if we
would suggest that we analyze the dollar opportunity and                had run with the initial results, we would have done our client
spending power of various demographic segments. We could                a disservice. They would have a form that was less efficient than
then evaluate the benefit of targeting certain segments of people       the original, despite the fact that it incorporated all of the sug-
in their 40s and match them up with the right sponsorship.              gestions of the focus group.
Framing significance / sample size                                      Jumping to conclusions
“Why bother to run additional tests? We already have the results from     “E-mail metrics have been declining; we’d better rethink our subject
the focus group. Isn’t that enough?”                                    lines and redesign our content.”
    It’s important to remember that focus groups are a great                PRM programs are often based on multiple communica-
source of qualitative data. They can provide information on             tions with patients. The bread and butter metrics for the e-mail
what people think and feel about a particular subject. However,         portion of a program are opens, clicks and opt-out clicks. So, if
a sample of 10 to 20 people is almost never representative of           the open rate and click-through rates for a program decline
the entire population. In fact, if you surf the net for a sample        consistently over a six-month period, it must be an indicator
size calculator and plug in a sample N of 20, you will find that        that something is drastically wrong with subject line or concept
your survey will provide an accuracy of plus or minus 22% (a            design, right?
44-point spread!) at a 95% confidence level.                                Not necessarily. Numerous factors have an effect on metrics
    Back in 2008, we received excellent feedback on how to              on a month-to-month basis. Before starting emergency meet-
make improvements to the enrollment form for an ongoing                 ings with the creative team, be sure to step back and evaluate a
PRM campaign. A focus group study, incorporating eye-track-             few other factors that could be contributing to the decline.
ing technology provided specific data on strengths and weak-                One recent development that has touched numerous phar-
nesses of the existing form. Because this information was based         maceutical marketers has been the renewed focus on regulatory
on focus group results, it was decided to take things to the next       compliance led by DDMAC. In recent months, many e-mail
level and to conduct a live field-test to validate the results.         programs have been asked to shut down in order to make
    In preparation for the test, two designs were developed as an       adjustments that bring communications into compliance with
alternative to “Form A” (the original form). “Form B” was a             regulations. The only problem is patients have no idea why the
modified                                                                messages they have been receiving on a regular basis have sud-
version of                            Enrollment Rate                   denly stopped. Once the dust settles and the program goes back
the or ig i-                                                            “on the air,” many recipients have forgotten that they ever
nal based                                                               signed up in the first place – open rates decline and opt-outs
o n f o c u s 12.0%                         11.3%                       increase.
g r o u p                                                                   Fluctuations in responsiveness are also closely linked to
f e e d b a c k 8.0%                                                    media strategy. It’s no secret that leads generated by natural
featur ing                                                              search provide the most engaged and responsive patients. Peo-
new intro                                                               ple who are anxious to learn more about recent symptoms or
c o p y ,                                                               who have just returned home from a visit to the doctor are
rollover                      Form A        Form B         Form C       very likely to actively surf the net to find answers. When they
functional-                                                             fill out an enrollment form for a PRM program, you can bet
ity, a different color call to action box and revised images.           they will be checking their inbox to see what information is on
Rather than stop there, it was decided to push the envelope             the way. As a result, time after time, open and click rates for
and come up with a brand new approach for “Form C.” This                natural search leads outpace other sources. The downside of
version incorporated the spirit of the findings, but with a com-        paid media is the relatively high cost and smaller pool of

                                                                                                    DTC Perspectives • December 2009 | 27

                      PRM Engagement Rates                     respondents.   but it is also needlessly complex and harder to understand. In
                            23.4%                              (See related   fact, don’t be afraid to experiment and see if the same data can

                                    20.9%                      c h a r t ,    be presented in different ways. Sometimes a table is vastly supe-
                                                               “ P R M        rior to a chart, especially if there are multiple variables to
15%                                                            Engage-        review.
         10.7%              10.4%
                                                               m e n t            Lastly, never take people to the edge of the “promised land”
10%                                 9.6%
                                                               Rates.”)       and leave them
                                                                  Co-reg      without            a 1,400                                1,250
                                                         4.4%                                                   Avg. Monthly Visitors
                                                               a n d C PA     roadmap to get 1,200
 0%                                                            vendors, on    there. In other 1,000
       Jan     Feb        Mar     Apr          May     Jun
                                                               the other      words, a bunch          800
                                                                                                                            730                    740

                    Open Rates     Click-through
                                                               hand, usual-   of slides with lots     600    550

                                                               ly     can     of facts is ver y       400

 acquire leads at a fraction of the cost of paid search and cater to          nice. But a pres-       200

 a huge database of people who have opted to receive invita-                  entation with             0
                                                                                                           Alpha site      Beta site  Gamma site Epsilon site
 tions to various programs. The catch is that patients in this situ-          insights, findings
 ation are likely to receive invitations for multiple programs and            and recommen- 1,400                                       1,250
 ultimately will have lower responsiveness compared to paid                                                     Avg. Monthly Visitors
                                                                              dations is best. 1,200
 search.                                                                      Many years ago, 1,000
    Evaluating which media strategy is better is not the focus of             at another com-         800
                                                                                                                            730                    740

 this discussion, which is a question better suited for a media               p a n y, w e h a d      600    550

 expert to evaluate. Ultimately, that boils down to an evaluation             some excellent          400

 of CPA compared to the conversion rate and how that affects                  training on this        200
 the backend through an ROI model. The point is, if rates are                 very subject. The         0
                                                                                                           Alpha site      Beta site  Gamma site Epsilon site
 declining, make sure that the changes are “real.” If they are due            bu z z wo rd s t o
 to any of the above factors, wait a couple of months to establish            remember were
 a new baseline before taking drastic action.                                 “What,” “So What,” and “Now What.” Whenever I craft a
Poor presentations with pretty charts                                         presentation, I think about answering those three questions:
“Don’t worry, we have all of the data. I’ll just throw some charts into           • What happened?
PowerPoint and we’ll figure out what it all means.”                               • What does it mean to my audience?
    I’ve never actually heard anybody say this, but we’ve all seen                • What should they do about it?
presentations that have been put together this way. They have                 A few concluding thoughts
100 slides to present in about 30 minutes and every chart had                    This brings us to the end of the discussion. And taking my
lots of “pretty colors” that aren’t necessary. Oftentimes the for-            own advice, I’ll leave you with some follow-up thoughts.
mat and the charts have the opposite effect of what was intend-               Remember that the purpose of analytics is to answer questions.
ed: everybody ends up overwhelmed with data and confused                      The best analysis results from understanding what the key
about the meaning of the various items on the charts.                         question is and what reasonable conclusions can be made from
    Planning is the key to putting together insightful presenta-              the data at hand. If all of the desired answers are not available,
tions. Allow enough time to review the information and see                    then data should not be bent, twisted, altered, folded, spindled
where the story is before preparing any slides. Once the story                or mutilated – follow-up studies or experiments should be
becomes clear, take time to map out the three key insights that
the audience needs to know. One approach is to take blank
                                                                                 This way you can quote Disraeli, get a good laugh from the
sheets of paper and sketch out a rough template of the presen-
                                                                              audience and know they aren’t laughing at you, but at how
tation. I have also found that thinking of a “funnel” approach
                                                                              other people sometimes fall prey to temptation. DTC
helps. Start with the big picture, then drill down to the finer
points using more granular data.                                              Louis Winokur is director analytics at DKI Direct. Winokur has
    Data must be presented in a clear, easy to understand fash-               more than 15 years of marketing experience, working extensively on
ion. Don’t get blinded by the “pretty colors” available in Excel.             consumer segmentation and analysis of purchase patterns across various
The two charts below show the same information via bar                        products and categories. He can be contacted through e-mail at
charts. I’ll admit, the multi-color version is more eye-catching,   

28 | DTC Perspectives • December 2009

To top