2007 AcademyHealth Research Meeting Section of Assessing Patient Experiences with Care: Measurement Issues & New Approaches
Meaningful comparisons between the ratings of adolescent’s QOL by parent and adolescent
I-Chan Huang, PhD1, Elizabeth A. Shenkman1, PhD Caprice A. Knapp1, PhD, Walter Leite2, PhD for Child Health Policy, College of Medicine 2Department of Educational Psychology, College of Education University of Florida
1Institute
Background
2
Children’s HRQOL ratings by parents
HRQOL is an important indicator, beyond clinical variables, to evaluate pediatric health outcomes
Parent ratings of pediatric HRQOL are used when children are too young or too sick to respond to the questions There is a great interests in examining the agreement / discrepancy between dyadic ratings
– Direct group comparisons using conventional methods (e.g., t-test, regression) are limited
– Discrepancy in observed scores may be due to lack of measurement invariance in HRQOL ratings
3
An example …
Consider one item measuring pediatric physical functioning…
“In the past month, how much of a problem has you [or your child] had with participating in sports activity or exercise?” (score: 1-5) – Children endorse: almost never (score: 4)
– Parents endorse: sometimes
(score: 3)
4
Measurement invariance (MI)
How might bias arise?
Different construct
5
Different item-domain relationships (slopes)
Child
Different origins of the ratings (intercepts)
5
Child
4
3
2
1
e1 p e2 p e3 p
e1 s
e2 s
Latent HRQOL
1
2
Y1p Y2p Y3p
Y1s Y2s
Parent
3
4
Physical
Social
Parent
Latent HRQOL
Meaningful group comparisons
– Assure the comparisons are based on the same measurement metric and reflect true differences
5
Aims
Main purpose
– To demonstrate meaningful comparisons between ratings of pediatric HRQOL by parents and children
Specific aims
– To test whether MI exists between the ratings of pediatric HRQOL by parents and children – To examine, after adjusting non-invariant items, whether discrepancy still exists between dyadic ratings of pediatric HRQOL – To determine the correlates of discrepancy in dyadic ratings of pediatric HRQOL
6
Methods
7
Data Collection and sample
Data collection
– A satisfaction survey for parents & children who enrolled in the Florida’s Children's Medical Services (CMS) Program – Telephone interview between 12/2005 to 03/2006
Sample
– Random samples of 364 dyads (children and their parents) who completed HRQOL measures – Adolescents between 15 to 18 years old – White: 37%, Black: 40%, Hispanic: 18%, others: 6%
8
The instrument – PedsQL 4.0
Four domains (23 items)
– Physical functioning (8 items) – Emotional functioning (5 items) – Social functioning (5 items) – School functioning (5 items)
Response category
– 5 categories with the range from 0 (never have a problem) to 4 (almost always have a problem)
9
A measurement model
A means & covariance structure (MACS) model Yij = ζj + λj*ηi + rij
Where, i=subject, j=item, Y=item score, η=factor score, ζ=intercept of item score, λ=factor loading, r=residuals.
Adolescents
λ1a
Physical ηa
λ2a λ3a λ1p
Physical ηp
λ2p λ3p
Parents
Y1
Y2
Y3
Y1
Y2
Y3
r1a ζ1a r2a ζ2a r3a ζ3a
r1p ζ1P r2p ζ2p r3p ζ3p
10
A framework for MI tests (Meredith 1993)
Type Configural invariance Metric invariance1 Scalar Intuitive meaning Similar common factor Statistical meaning Item clusters are identical
Invariance2 Residual
invariance3
+ Comparable relationships between items and common factor (i.e., the same unit of measurement) + Similar origin (or baseline) of the item response
+ Similar degree of measurement error for item response
+ Factor loading (λ) of an items is identical
+ Item intercept (ζ) is identical + Item residual (r) is identical
Note: known as 1: weak factorial invariance; 2: strong factorial invariance; 3: strict factorial invariance
11
Configural invariance test
Adolescents
Physical η
Physical η
Parents
Y1
Y2
Y3
Y1
Y2
Y3
12
Metric invariance test
Adolescents
λ1
Physical η
λ2 λ3 λ1
Physical η
λ2 λ3
Parents
Y1
Y2
Y3
Y1
Y2
Y3
13
Scalar invariance test
Adolescents
λ1
Physical η
λ2 λ3 λ1
Physical η
λ2 λ3
Parents
Y1
Y2
Y3
Y1
Y2
Y3
ζ1
ζ2
ζ3
ζ1
ζ2
ζ3
14
Residual invariance test
Adolescents
λ1
Physical η
λ2 λ3 λ1
Physical η
λ2 λ3
Parents
Y1
Y2
Y3
Y1
Y2
Y3
r1
ζ1
r2
ζ2
r3
ζ3
r1
ζ1
r2
ζ2
r3
ζ3
• Multi-Group Confirmatory Factor Analysis (MG-CFA) • Fit indices: Satorra-Bentler Χ2 with P>0.05; RMSEA <0.06 • Software: Mplus
15
Score calibrations & group comparisons
Partial measurement invariance (PMI)
– Allow parameters of some (but not all) items in a domain acting differently between groups – Calibrate domain scores given the PMI
Group comparisons & covariates of discrepancy
– Compare domain scores between groups using regression analysis – Demographics: child & parent’s age, sex, race, and parent’s level of education – Health status: parent’s rating of adolescent’s health and the clinical risk group (CRG)
16
Results
17
Items with measurement invariance
Domain Type Configural Metric Scalar Residual Physical
(8 items)
Emotional
(5 items)
Social
(5 items)
School
(5 items)
All All 3 items (38%) 4 items (50%)
All All 3 items (60%) 4 items (80%)
All All 2 items (40%) 3 items (60%)
All All 3 items (60%) All
18
Group means and discrepancy (covariate un-adjustment)
Observed scores Parent Child Difference ES‡ Physical 66.5 77.9 11.5 0.53 Emotional 66.7 74.1 7.4 0.33 Social 66.1 81.0 14.7 0.72 School 59.9 65.9 5.7 0.31 True scores Parent Child Difference ES Physical 73.7 82.3 8.6 0.40 Emotional 77.2 85.3 8.1 0.53 Social 70.5 83.0 12.3 0.77 School 60.5 74.1 13.3 0.70 † Not adjust for covariates. ‡ Effect size: small (0.2~0.49), moderate (0.5~0.79), and large (≥0.8)
19
Discrepancy with covariate adjustment
Group
†
Child age Child sex ‡ Child race Black Others Health $ Very good Good Fair Poor Disease group # Acute Mild CDs Moderate CDs Severe CDs
†
Physical 8.12*** (0.38)& -0.86 6.48** 7.38** 3.84 -1.21 -9.77** -13.93*** -18.51*** 3.18 2.59 -3.13 -11.76***
Emotional 7.71*** (0.50)& -0.44 4.43** 2.40 0.26 -3.07 -6.16*** -9.58*** -15.12*** 3.89 -3.26 -1.56 -2.05
Social 11.84*** (0.73)& 0.62 4.02** 4.45** -1.47 -2.67 -6.04* -6.42* -8.29* 4.86 -5.53 -2.86 -1.37
School 12.94*** (0.66)& 1.43 -1.97 2.84 2.15 -4.22 -7.89* -9.69* -10.98* 8.33 -9.41* -2.59 -2.55
Parent, ‡Girl, $Excellent health, and #Healthy groups are the references, respectively. & Effect size: ignorable (<0.02), small (0.2~0.49), moderate (0.5~0.79), and large (≥0.8). * P<0.05; **P<0.01; ***P<0.001
20
Discrepancy by level of health status
0.25
Effect size (unit: SD)
0.20
Poor
0.15
Fair Good
0.10
Very Good Excellent
0.05
0.00 Physical Emotional Social School
Effect size: ignorable (<0.2), small (0.2-0.5), moderate (0.5-0.8), & large (>0.8)
21
Discussions & Implications
22
Significant discrepancy between dyads
Major findings
– Higher HRQOL scores reported by adolescents vs. parents – Magnitudes were small (in ES) for physical domain, but moderate for emotional, social & school domains – Health status was the most significant variable associated with the discrepancy
Possible interpretations
– Assign different meanings to the same items
– Use different response sets (e.g., expectation)
23
Implications for clinical practice
Use dyadic reports b/c parents’ and adolescents’ HRQOL ratings are complementary
– Significant discrepancy between dyads – Greater reliability by parents’ ratings, but greater validity by adolescents’ ratings [Matza, 2004]
Important for children with life-limiting conditions
– Parents play a critical role in shared decision-making – Discrepancy was largest for severe health conditions
– Parents emphasize the future consequence, while children focus on immediate cancer impact [Vance, 2001]
24
Implications for HSR
Assuring measurement invariance before conducting groups comparisons
– MI testing is useful not only for HRQOL research, but also for other measurement studies, e.g., patient satisfaction, health behavior, etc.
Conducting qualitative studies to confirm our quantitative findings of the discrepancy in the HRQOL ratings
25
Thank you
I-Chan Huang ichuang@ufl.edu Elizabeth A. Shenkman eas@ichp.ufl.edu