Educational Research:
Study Methodology
Pamela M. Williams MD
Dept of Family Medicine Uniformed Services University
Educational Research
Broadly Defined
Any investigation related to the education of medical professionals
– Undergraduate (medical school) – Graduate (residency) – Continuing medical education
Basic (Educational) Research Steps
1. 2. 3.
4.
5. 6. 7.
Identifying a problem Examining relevant variables through a literature review Constructing a hypothesis Creating a research design to investigate a problem Collecting and analyzing data Drawing conclusions Writing and publishing the results
Session Topics
Basics of research paradigms & design Measurement & data collection Sampling Threats to validity IRB approval—is it needed?
Session Ground Rules
Interactive discussion
– Minimal lecture – Small group activities
Please
– Ask questions – Share knowledge and experiences
What is your research question?
Break into your small group Consider the problem you identified last evening Take 2-3 min to write a research question Articulate your question to peers & refine as needed
Optimally, your question is…
Based on literature/theory Includes sample description (e.g. 3rd year medical students) Includes study design (e.g. relationship, difference between groups, etc) Includes the independent & dependent variables Measurable Stated as a question or hypothesis
Selecting a Research Design
Research Paradigms
Inductive: Investigator begins with observations and attempts to explain what has been observed by generalizing. Deductive: Investigator begins with a theory and collects data to test it.
Research Design Decision Tree
Categorizing Predominantly qualitative
Exploratory
Relationships
Process
What type of question is being asked?
Descriptive/ Correlational
Many cases Single case Nonexperimental
Predominantly quantitative
Explanatory/ confirmatory
Experimental/ quasi-experimental
Small Group Exercise
Review your set of three papers As a group, answer the following for each study:
– What is the research question? – Is it a deductive or inductive research process? – What is the study design?
Be prepared to
– Briefly tell us about the study – Describe characteristics of the methodology used
Inductive Research
What is the phenomenon?
Use when
– Focus is meaning and context – In-depth recording, triangulation – Inductively derived interpretation
Methods
– Interview – Observation – Think aloud, stimulated recall – Chart review – Surveys
Uses in Medical Education
Needs assessment Program development Curriculum evaluation Performance evaluation And more!
Challenges of Qualitative Research Methods
Data overload Time demands of processing and coding Adequacy of sampling Generalizability of findings Researcher bias Credibility and quality of conclusions
Descriptive/Correlational Research
How are variables associated? How do we begin to make sense of what we observe?
Use when
– Data on target variables is available – Predictors can’t be randomized – Subjects &/or treatments not controllable – Control groups not available
Methods
– Surveys – Chart review – Archived data
Experimental & Quasiexperimental
Does the predictor cause the relationship?
Use when
– Temporal relationship – Feasible explanatory mechanism – No alternative explanations – Subjects & treatments controlled
Methods
– Control over treatment & measure – Randomization – Control groups
Non-experimental Designs
Case control Cross-sectional Time series (serial surveys) Cohort (Panel)
– Prospective – Retrospective
Research Design Decision Tree
Categorizing Predominantly qualitative
Exploratory
Relationships
Process Many cases
What type of question is being asked?
Descriptive
Single case Explanatory/ confirmatory Nonexperimental
Predominantly quantitative
Experimental/ quasi-experimental
Small Group Work
As a group identify 2 possible study designs for your project
Educational Research: Measures & Outcomes
Brian Smoley, MD, MPH Sandy Kimmer, MD, MPH
Review: Variables
Independent (IV): Variable that is manipulated,
measured, or selected to observe the relationship to some other observed variable (i.e. it is expected to influence some other variable)
Dependent (DV): Variable that is observed and
measured in response to an independent variable (i.e. it is expected to increase, decrease, or vary systematically as the IV changes
Control: Anything held constant
An Example
Do first-year interns who complete a resident run in-service review course score higher on the in-service exam than those who do not complete the course?
– What is the independent variable? – What is the dependent variable? – What are possible control variables?
Small Group Work
Take 10 minutes to consider the variables for your various projects.
What potential threats to validity for your projects?
Threats to Validity
Internal
– History – Maturation – Repeated measurement – Statistical regression – Selection – Loss of subjects – Investigator bias
External
– Is the sample representative of the population? Can the study be generalized? – Are the conditions the same as other environments? – Hawthorne Effect
Sampling
Who are your ―people‖?
– To whom would you like the results to be generalized? – How do you select your sample in away that enables you to generalize the results of this sample? – What do you wish to generalize to this sample? – What are some ways of selecting an appropriate population?
How and why are you sampling?
Random sampling Stratified sampling Cluster sampling Systematic sampling
Challenges to sampling
– – – – – – Sampling bias Size Population Self-selection Snowballing Available group use
Do I need IRB Approval?
Research is a systematic investigation including research, development, testing, and evaluation to develop or contribute to ―generalizable knowledge.‖
Questions to Consider…
Is the data normally collected in your course? Is the data being specifically collected for your study? Will the data be anonymous? Will you use any identifier? Is it linked? Do you want to publish or ―make public‖ your results? Do you think you need informed consent?
Potential Categories
Exempt Expedited Full review
Take home? Remember the Basic (Educational) Research Steps !!!
1. 2. 3.
4.
5.
6.
Identify a problem Review the literature Construct a hypothesis Identify a research design to investigate the problem Collect and analyze your data Draw conclusions
Questions?
Deductive Research Process
Define Research Question No gap in literature Conduct Literature Review Gap in literature Refine Question Collect & Analyze Data Obtain IRB Approval Design Study