PowerPoint Presentation

							               Evaluating a Study


•   Macro Questions:
   Who’s doing the research?
   Who’s funding the research?
    Why are they asking the research question,
    within the framework of society, class,
    ethnicity, gender, & so on, as situated within a
    particular part of the world in a particular
    point in time?
   Methodological-Empirical Questions
   What problem does the research address?
   What is the degree of significance of the
    problem:
        Real world/practical?
        Theoretical? The research literature?
   Is the problem clearly/coherently described &
    appropriately conceptualized/discussed within
    the framework of theories/hypotheses in the
    relevant research literature? (see, e.g., King et
    al., Designing Social Inquiry).
 Is the problem conceptualized from one particular
perspective, or does it consider a range of competing
perspectives? Is the particular approach valid or not in
this regard?
 Does the author anticipate problems/objections &
adequately address them?
 Is the relevant literature adequately reviewed? Is
relevant literature left out?
 Does the discussion of the problem correspond to the
germane issues in the literature, & is the discussion
coherent & compelling?
 Does the discussion of the problem represent
significant advancement (theoretical &/or empirical)
relative to the relevant literature?
   Is the study geared to description or to examining
  theoretical questions/hypotheses?

   What research questions/hypotheses are
  formulated? Are relevant ones left out?

 What are the dependent and independent variables?
How are the variables defined & measured?

   Are the definitions & measurement of the
  variables valid, both practically speaking in terms of
  real-world constraints & in terms of both the
  conceptualization of the problem & the literature?
  Is the measurement of the variables likely to be
  reliable?
 Would alternative ways of defining &
measuring the variables be likely to
change the results? If so, how? Does the
author consider such possibilities?

 Are the research questions/hypotheses
falsifiable? That is, what evidence would
lead the author to change her/his
perspective/ argument?
 Are the research questions/hypotheses
consistent with each other?
 Are they appropriately conceptualized in terms of
the germane issues in the literature & the
discussion of the problem?
 Do they represent significant advancement
(theoretical &/or empirical) relative to the
literature?
 Do they represent significant contributions to
other literatures (see King et al.)?
 Who/what & how many are the observations
being studied? How were they selected? Are the
characteristics of the observations & their
method(s) of selection appropriately discussed?
 Does anything about the observations influence
their selection & potentially influence the study’s
findings (i.e. in regard to how, when, & where the
data were collected)? Thus, how & why might
data collected differently, in other times, & in
other places change the findings? (See the
previous questions about how the definition &
measurement of the variables might influence the
results.)
 Is the number of observations adequate for
the study’s purpose & the analytic techniques
to be used?

 Are the analytic techniques appropriate for
the purpose of the study in view of prevailing
standards? Are the techniques appropriately
discussed?
 Are the analytic techniques used competently?

 Do the empirical findings correspond to the
hypotheses or not?
 What are the results in terms of both the direction
of relationships & the magnitude of effects? Are the
results significant statistically, theoretically, &
practically?
 Are the findings (including the numbers)
consistent? Are they plausible (including are they
too good to be true)?
 Are the empirical findings clearly/accurately
described?
 Are the empirical findings appropriately
discussed in view of the theories & research
questions/hypotheses?
 Is relevant evidence not reported? If not,
then why?
 What are the author’s interpretations of the
results of the statistical, theoretical, & practical
significance of the results?
 Are they appropriate in regard to the sample,
questions/hypotheses, and analytic techniques?
 Are there meaningful alternative interpretations
that the discussion does not consider?
 To what degree was the study worthwhile in some
way or another: description, theory, activism/policy?
 Do the results have ramifications for other groups,
places, times, issues, & disciplines?
 From top to bottom, what could have, or should
have, been done differently, & why? Are there
practical reasons why these alternatives were not
carried out in this particular study?
 What are the study’s implications for the research
agenda, both for the particular topic & perhaps for
other topics?
 That is, does the author adequately
discuss the directions in which research
should head, based on the findings of this
study?


Selected References: King et al.; Babbie; Moore &
McCabe; Utts; Freedman et al.; Pyrczak.
   Short Version of Research-Evaluation Questions
       Adapted from a document provided by Dr. Ronald Lee,
                     College of Business, FIU



 What is the problem?

 Why should anyone read this?
 Who cares about the results (i.e. practical &
theoretical relevance)?
 What does the research claim about its findings, &
are the findings & claims valid & refutable?
 What’s left out in terms of evidence &
interpretations?
 Do the findings & claims represent significant
advancement in the topic, or even in other topics?
 Where do we go from here in regard to the topic?