Hard Choices

							Hard Choices for Individual Situations:
Selecting a Course Management System



By Bruce Landon, Ph.D

Douglas College &
Western Cooperative for Educational
 Telecommunications
Why choices are difficult
   The newness of the course
    management software as an
    educational innovation
   Too many Products
   Too many Products Features
   You could make a big mistake here
Too many products
   Commercial Products
       Not yet profitable
   Open Source Products
       Many Unknowns – OKI
   Training Products
       Related and similar, but larger market
   Institutional Products
       Many in-house proprietary systems
Too many product features
   50 plus features
   With each new version there are
    additional features
   Marketing often focuses on
    “special” features or “names of
    features” in the intense
    competition
Difficulties inside of the
Decision-Makers

   Overconfidence in historic
    strategies for making decisions
   Too little working memory for
    decision problems this large
   Too many Cognitive Illusions
cognitive illusions




are due to limited focus
How Many can you juggle?
Even a Mathematician is
limited by Memory Span
Some are more limited
Some can handle more
But Seven – Nine is Maximum
and Ten is only in your dreams
When there are too many
Some of them get dropped
Cognitive Illusions:
   Availability Heuristic
   Representativeness Heuristic
   The Framing Effect
Availability Heuristic
   Powerful cognitive distortion
   like Out of Sight - Out of Mind
   Believability is related to ease of recall
   if one cannot remember it
    then it must not be true
   Distorted by vividness of information
   Distorted by number of repetitions,
    like ads on the radio and television
Representativeness Heuristic
    Powerful cognitive distortion
    if it looks like one then it is one
    in spite of relevant base rate
    information
    (you can tell a book by its cover)
    (if looks good then it works well)
Does Monday look like this?
Or does Monday look like this?
      The Framing Effect
    Refers to effect of the frame of reference
    Risk Frame: people tend to take risks
    described in terms of loss – like losing
    enrolments
    Benefits Frame: People tend to avoid
    risks that are described in terms of
    benefits – gaining more enrolments
Framing is like context for the
size of the circle in the middle




      It only looks different in different contexts
    Effect of More Options –
    delaying and avoiding
   In a high conflict situation where there are
    many alternatives, the decision makers tend
    to postpone making a decision altogether
   The effect has been observed in physicians,
    when offered more alternative drugs fewer
    opted to try any new drug
   The effect has also been observed in
    members of the Ontario Provincial Parliament
    when they were requested to choose among
    different types of health care expenditures
The ultimate fallibility is Overconfidence
      The tendency to be more confident than is warranted by
      the evidence
      To overestimate the accuracy of one's beliefs and
      judgments (availability heuristic again)
      For example, the confidence of an eye witness in their
      testimony is unrelated to the accuracy of that testimony
      This overestimation of confidence enhances personal
      self-esteem and contributes to the resistance to being
      persuaded otherwise
     “Ironically, people often are most overconfident
      when most incompetent” Myers (2001, p 527)
How people normally Make Decisions:
     The 5 Basic Strategies
     One Reason strategy
     Elimination by aspect strategy
     Satisficing strategy
     Equal weight strategy
     Weighted averaging strategy
One Reason strategy
   (aka pick the best)
   Step 1: select most important feature
   Step 2: pick product best on that
    feature
   (no need to use numbers)
   Non-Compensatory in that other
    features can not make up for not being
    best on the most important feature
Elimination by aspect strategy
      (aka pick the last one standing)
      Step 1: set the requirements for each
       feature – the minimum criteria
      Step 2: eliminate options one at a time
       if any feature does not meet the
       minimum feature requirement
      (does not necessarily require numbers)
      Non-Compensatory in that once a
       product is rejected on any feature it is
       eliminated
      This strategy does not always select
       only one best option
Satisficing strategy
   (aka Bounded Rationality Model)
   Step 1: setting cutoff levels for each of the
    features or criteria
   Step 2: an option is examined until it fails on any
    criteria and then is eliminated
   Step 3: The first option that passes all feature
    criteria is the one selected
   Step 4: (optional) if none of the products pass
    then the cutoff requirements are reduced and the
    process is repeated
   Non-Compensatory and is effected by the order in
    which the options are considered
Equal weight strategy
   (aka scoring strategy)
   Step 1: Set pass/fail criteria for each feature
   Step 2: Assign suitability scores (1,0) to every
    feature on all of the products
   Step 3: Total the suitability scores for all products
   Step 4: The option with the highest total is
    selected (there may be a tie when only a few
    features are considered)
   Strategy is Compensatory in that some feature
    suitability scores can compensate for other
    missing or failing features
Weighted averaging strategy
    (aka weighted adding strategy, grading model)
    Step 1: Set importance weightings for each feature
    Step 2: Assign suitability scores for every feature
    Step 3: Multiply the scores times their weights
    Step 4: Sum the weighted feature subscores into a
     total score for each option
    Step 5: Select the option with the highest weighted
     average score or highest sum
    This is a compensatory strategy both in terms of the
     weights and in terms of the suitability scores
    This method is considered normatively rational
     decision process because it uses all of the
     information in a consistent manner
Screening tools on the web

   Finding products with specified
    features using:
    www.edutools.info/landonline/

   Selecting specified features to find
    products supporting those
    features.
    Making a Short List of
    Application Options
   The optional applications are more like
    business partners in a continuing
    arrangement so there are usually
    additional considerations required than
    mere product features
   In the end there is a trade-off in how
    many to consider and how much time is
    available for the decision
Evaluating product suitability


      The recommended approach is to use
       the most rational decision strategy –
       even though it is more work
      An crucial part of the work is to assign
       the relative importance weighting to
       each of the decision criteria
      This can be done in a way that
       facilitates political endorsements of the
       decision process involving stakeholders
Inviting the shortlist of vendors to make
competitive proposals (RFP model)
    This ensures that the most recent product
     information will be available in spite of rapid change
     in product evolution
    The RFP process shifts much of the information
     gathering task to the vendor rather than the buyer
    This process provides a preview of what it would be
     like to work with different vendors before making a
     expensive commitments
    Often vendors will provide on-site presentations that
     can be the basis for product comparison and
     suitability ratings – a product competition
    Judging product feature
    suitability to local situation
   The same judges do not have to judge all features –
    it is better to use judges with expertise in the specific
    feature to comparatively judge the products
   Persons with disabilities should be among the
    ranks of the judges or the accessibility of the
    products will likely be overestimated
   Consider the inclusion of “new users” whenever the
    system will become the “front door” to the institution
    for new students and faculty.
Demonstration model
   of the Comparative Analysis Decision
    Table with Three Options
   Using features:
       Discussion Forums
       Course Layout Templates
       Course Management
   Using the edutools.info tools
    Rechecking by doing
    Sensitivity Analyses

   You can change the weights and rescore
   You can change the ratings and rescore

   It is often comforting to know that the winner
    would not change even if the weights were a
    little off or if some of the suitability ratings
    had been a little different
Thank You for your Attention



    http://www.edutools.info

    Bruce_Landon@douglas.bc.ca