The State of Eating Disorders Research

The State of Eating Disorders Research Publications 1980-2000. An Empirical Analysis Mary J. Markland, MA, AHIP (Southeast Clinical Campus Librarian); Stephen A. Wonderlich, PhD (Associate Chair/Professor Department of Neuroscience) James E. Mitchell, MD (Chair/Professor Department of Neuroscience); Ross D. Crosby, PhD (Director of Biomedical Statistics & Methodology, Department of Neuroscience/Neuropsychiatric Research Institute) Martina de Zwaan, MD (Research Scientist, Department of Neuroscience/Neuropsychiatric Research Institute) Research Questions • What is the quality of eating disorders publications? • Has the quality of eating disorders publications changed over time? • How does the quality of eating disorders literature compare to publications in anxiety? PsycInfo Search Strategies Eating Disorders (anorexia nervosa OR bulimia OR binge eating OR eating attitudes OR body image disturbances OR eating disorder*) AND (treatment* OR therap*) Limits: journal article, English (anorexia nervosa OR bulimia OR binge eating OR eating attitudes OR body image disturbances OR eating disorder*) NOT (treatment* OR therap*) Limits: journal article, English Population/Sample The interpretation of the results of any study depends on the characteristics of the population included for analysis. Define the population clearly. If control groups are part of the design, present how they are defined… Describe the sampling procedures and emphasize any inclusion or exclusion criteria. APA Task Force on Statistical Inference Clear Inclusion/Exclusion Criteria Eating Disorders Anxiety Disorders 80 77.9 85.5 70.9 84.2 Interval Estimates Confidence Intervals Presented 20 Eating Disorders Anxiety Disorders 13.2 10.4 6.6 1.2% 0.0% 3.9 85.7 88.2 Percent 60 40 20 0 Methods Raters • 8 raters with master’s or doctoral level training in psychology/psychiatry Percent Year Interval estimates should be 16 given for any effect sizes involving principal outcomes. 12 Provide intervals for correlations and other 8 coefficients of association or 4 variation whenever possible. APA Task Force on Statistical Inference 0 1980 1980 1990 2000 1990 2000 Anxiety (anxiety disorder* OR phobia* OR obsessive-compulsive disorder OR panic disorder OR panic OR post-traumatic stress disorder OR traumatic neurosis OR stress reactions OR ocd OR ptsd OR post-traumatic stress disorder) AND (anxiety management OR treatment* OR therap*) Limits: journal article, English (anxiety disorder* OR phobia* OR obsessive-compulsive disorder OR panic disorder OR panic OR posttraumatic stress disorder OR traumatic neurosis OR stress reactions OR ocd OR ptsd OR post-traumatic stress disorder) NOT (anxiety management OR treatment* OR therap*) Limits: journal article, English Interpretation Random Assignment of Subjects 30 20 25.6 Clinical Significance Data Presented 20 16 Percent 12 8 6.5 9.2 5.8 3.9 1.3% 7.9 Training • 21 practice articles were reviewed jointly Assignment Random assignment allows for the strongest possible causal inferences free of extraneous assumptions…provide enough information to show that the process for making the actual assignments is random. (If) random assignment is not feasible, describe methods used to attenuate sources of bias, including plans for minimizing dropouts, noncompliance, and missing data. APA Task Force on Statistical Inference Procedure • Citations were stored in EndNote libraries by subject, treatment vs. nontreatment and year • Citations were randomly selected from EndNote and articles obtained • Article assignments were balanced by field, year, content, and rater pairs • All articles rated independently by 2 raters • Raters were blind to journal, authors, and author affiliation • Disagreements resolved via consensus When you interpret effects, think of credibility, generalizability, and robustness. APA Task Force on Statistical Inference Eating Disorders Anxiety Disorders Eating Disorders Anxiety Disorders Percent 4 0 15.8 15.8 9.2 9.1 10 7.8 0 1980 1990 2000 Statistical Analysis • Power Analysis: n=456 articles to detect small effect size for ED vs. Anxiety • Interrater reliability assessed using kappa coefficients • Statistical analysis performed using hierarchical log linear analysis Methods Rating Instrument 75-item rating based on recommendations from APA Task Force on Statistical Inference American Psychologist 1999 54(8): 594-604. • Funding source • Description of measures • Research question • Psychometrics of measures • Theory-based hypothesis • Blind rating of outcome • Sample size • Research Design • Completion rates • Control/Comparison group • Follow-up rates • Effect size • Clear inclusion criteria • Clinical significance • Method of determining inclusion • Constraint of alpha • Recruitment strategies • Multivariate tests • Demographic information • Post hoc tests • IRB approval • Reporting of statistics • Informed consent • Confidence intervals • Description of procedures • A priori power analysis • Random assignment • Methodological limits 1980 1990 2000 Conclusions Note the shortcomings of your study…acknowledging limitations if for the pupose of qualifying results and avoiding pitfalls in future research. APA Task Force on Statistical Inference Methodological Limits Discussed 80 Percent 60 40 34.9 51.3 Eating Disorders Anxiety Disorders 63.2 67.5 73.7 PubMed Search Strategies Eating Disorders (eating disorder* OR anorexia nervosa OR bulimia) AND (therapy OR therapies OR treatment) Limits: human, journal article, English (eating disorder* OR anorexia nervosa OR bulimia) NOT (therapy OR therapies OR treatment) Limits: human, journal article, English Procedures Clearly describe the conditions under which the measurements are taken (e.g. format, time, place, personnel who collected data). Describe the specific methods used to deal with experimental bias… APA Task Force on Statistical Inference Assessor Blind to Condition Eating Disorders Anxiety Disorders 60 Percent 40 30.8 62.5 54.5 45.0 62.5 20 0 22.1 1980 1990 2000 20 0 25.0 Conclusions • Eating Disorder publications tend to be less methodologically rigorous than Anxiety Disorder publications in many important areas. Structured Interviews Prospective Longitudinal Designs Random Assignment Blind Outcome Assessment • Both Eating Disorder and Anxiety Disorder publications demonstrate improvement in methodological rigor over the last 20 years. Sample Size Alpha constraint Effect Size Psychometrics • The majority of Eating Disorder and Anxiety publications do not include many of the APA Task Force recommendations. Confidence Intervals Clinical Significance A priori power analysis Alpha constraint Anxiety Disorders (anxiety disorder* OR obsessive compulsive disorder OR panic disorder OR phobia* OR agoraphobia OR stress, disorders, post-traumatic OR posttraumatic stress disorder* OR ptsd) AND (therapy OR therapies OR treatment) Limits: human, journal article, English (anxiety disorder* OR obsessive compulsive disorder OR panic disorder OR phobia* OR agoraphobia OR stress, disorders, post-traumatic OR posttraumatic stress disorder* OR ptsd) NOT (therapy OR therapies OR treatment) Limits: human, journal article, English 1980 1990 2000 Power and Sample Size A Priori Power Analysis Mentioned 100 80 Percent 60 40 20 3.9% Study Design Eating Disorders Anxiety Disorders Total 1980 76 (38/38) 76 (38/38) 152 (76/76) 1990 76 (38/38) 76 (38/38) 152 (76/76) 2000 76 (38/38) 76 (38/38) 152 (76/76) Overall 228 (114/114) 228 (114/114) 456 (228/228) Provide information on sample size and the process that led to sample size decisions. Document the effect sizes, sampling and measuring assumptions, as well as analytic procedures used in power analysis. APA Task Force on Statistical Inference Eating Disorders Anxiety Disorders 0.0% 0.0% 0.0% 0.0% 0.0% 0 1980 1990 2000 Results Design Make clear at the outset what type of study you are doing. Do not cloak a study in one guise to try to give it the assumed reputation of another…be sure to define and prioritize (the) goals. APA Task Force on Statistical Inference Prospective Longitudinal Design Research Question/Hypothesis Stated 100 Percent 80 60 40 20 3.1 66.2 89.6 76.7 78.9 77.6 92.1 Eating Disorders Anxiety Disorders 60 57.1 57.0 56.6 Effect Sizes Always present effect sizes for primary outcomes. Effect Size Data Presented 30 Eating Disorders Anxiety Disorders 20.8 16.4 Limitations 25.7 Percent Percent Eating Disorders Anxiety Disorders 40 35.5 28.6 43.4 20 APA Task Force on Statistical Inference 20 10 10.0 8.3 • • • • • Literature searches may not have found all the articles Excludes animal/basic science studies Excludes non-English articles Low interrater agreement on some ratings Raters could deduce field & year from articles 0 Year: 1980 1990 2000 0 0 1980 1990 2000 1980 1990 2000