Computational Radiology Laboratory Children’s Hospital
Harvard Medical School Department of Radiology
www.crl.med.harvard.edu Boston Massachusetts
A Survey of Validation Techniques
for Image Segmentation and
Registration, with a focus on the
STAPLE algorithm
Simon K. Warfield, Ph.D.
Associate Professor of Radiology
Harvard Medical School
Outline
• Validation of image segmentation
– Overview of approaches
– STAPLE
• Validation of image registration
• STAPLE algorithm available as open
source software from:
– http://www.nitrc.org/projects/staple
– http://crl.med.harvard.edu/
Computational Radiology Laboratory.
Slide 2
Segmentation
• Goal: identify or label structures
present in the image.
• Many methods:
– Interactive or manual delineation,
– Supervised approaches with user
initialization,
– Alignment with a template,
– Statistical pattern recognition.
• Applications:
– Quantitative measurement of
volume, shape or location of
structures,
– Provides boundary for visualization Newborn MRI
by surface rendering. Segmentation.
Computational Radiology Laboratory.
Slide 3
Validation of Image Segmentation
• Spectrum of accuracy versus realism in
reference standard.
• Digital phantoms.
– Ground truth known accurately.
– Not so realistic.
• Acquisitions and careful segmentation.
– Some uncertainty in ground truth.
– More realistic.
• Autopsy/histopathology.
– Addresses pathology directly; resolution.
• Clinical data ?
– Hard to know ground truth.
– Most realistic model.
Computational Radiology Laboratory.
Slide 4
Validation of Image Segmentation
• Comparison to digital and physical
phantoms:
– Excellent for testing the anatomy, noise and
artifact which is modeled.
– Typically lacks range of normal or
pathological variability encountered in
practice.
MRI of brain
phantom from
Styner et al. IEEE
TMI 2000
Computational Radiology Laboratory.
Slide 5
Comparison To Higher Resolution
MRI Photograph MRI
Provided by Peter Ratiu and Florin Talos.
Computational Radiology Laboratory.
Slide 6
Comparison To Higher Resolution
Photograph MRI Photograph Microscopy
Provided by Peter Ratiu and Florin Talos.
Computational Radiology Laboratory.
Slide 7
Comparison to Autopsy Data
• Neonate gyrification index
– Ratio of length of cortical boundary to length
of smooth contour enclosing brain surface
Computational Radiology Laboratory.
Slide 8
Staging
Stage 3: at 28 w GA
shallow indentations of inf. frontal
and sup. Temp. gyrus
(1 infant at 30.6 w GA,
normal range: 28.6 ± 0.5 w GA)
Stage 4: at 30 w GA
2 indentations divide front. lobe into
3 areas, sup. temp.gyrus clearly
detectable
Stage 3 (3 infants, 30.6 w GA ± 0.4 w, Stage
normal range: 29.9 ± 0.3 w GA)
Stage 5: at 32 w GA
frontal lobe clearly divided into three
parts: sup., middle and inf. Frontal gyrus
(4 infants, 32.1 w GA ± 0.7 w,
normal range: 31.6 ± 0.6 w GA)
Stage 6: at 34 w GA
temporal lobe clearly divided into
3 parts: sup., middle and inf. temporal
gyrus
(8 infants, 33.5 w GA ± 0.5 w
normal range: 33.8 ± 0.7 w GA)
Stage 4 Stage
“Assessment of cortical gyrus and sulcus
formation using MR images in normal Computational Radiology Laboratory.
fetuses”, Abe S. et al., Prenatal Diagn 2003 Slide 9
Neonate GI: MRI Vs Autopsy
Gyrification Index versus age in days
3
2.5
2
GI
1.5
1
0.5
0
200 220 240 260 280 300 320 340
Post-conceptional age in days
MRI Scan 2 MRI Scan 1 Armstrong
Computational Radiology Laboratory.
Slide 10
GI Increase Is Proportional to Change in Age.
'change in GI' versus 'days of growth before final scan'
0.8
0.7
0.6
change of GI
0.5
0.4
0.3
0.2
0.1
0
50 55 60 65 70 75 80 85 90
time interval between scans in days
Change of Total Brain GI Linear (Change of Total Brain GI)
Computational Radiology Laboratory.
Slide 11
GI Versus Qualitative Staging
Staging versus GI
2.4
2.2
2
Total Brain GI
1.8
1.6
1.4
1.2
1
3 4 5 6 7 8 9
Staging Grade
MRI scan 1 MRI scan 2
Computational Radiology Laboratory.
Slide 12
Neonate Gyrification
GI : interactive versus automatic segmentation.
5
GI - automatic segmentation
4.5
4 y = 1.2241x + 0.4443
3.5
3
2.5
2
1.5
1
0.5
0
-1 0 1 2 3 4 5
GI - hand segmentation
Linear (line of equality)
Computational Radiology Laboratory.
Slide 13
Validation of Image Segmentation
• Comparison to expert performance; to other
algorithms.
• Why compare to experts ?
– Experts are currently doing the segmentation tasks
that we seek algorithms for.
– Surgical planning.
– Neuroscience research.
• What is the appropriate measure for such
comparisons ?
Computational Radiology Laboratory.
Slide 14
Measures of Expert Performance
• Repeated measures of volume
– Intra-class correlation coefficient
• Spatial overlap
– Jaccard: Area of intersection over union.
– Dice: increased weight of intersection.
– Vote counting: majority rule, etc.
• Boundary measures
– Hausdorff, 95% Hausdorff.
• Bland-Altman methodology:
– Requires a reference standard.
• Measures of correct classification rate:
– Sensitivity, specificity ( Pr(D=1|T=1), Pr(D=0|T=0) )
– Positive predictive value and negative predictive value
(posterior probabilities Pr(T=1|D=1), Pr(T=0|D=0) )
Computational Radiology Laboratory.
Slide 15
Validation of Image Segmentation
• STAPLE (Simultaneous Truth and
Performance Level Estimation):
– An algorithm for estimating performance
and ground truth from a collection of
independent segmentations.
Computational Radiology Laboratory.
Slide 16
STAPLE papers
– Image segmentation with labels:
• Warfield, Zou, Wells ISBI 2002
• Warfield, Zou, Wells MICCAI 2002.
• Warfield, Zou, Wells, IEEE TMI 2004.
• Commowick and Warfield IPMI 2009
– Image segmentation with boundaries:
• Warfield, Zou, Wells MICCAI 2006.
• Warfield, Zou, Wells PTRSA 2008.
– Diffusion data and vector fields:
• Commowick and Warfield IEEE TMI 2009
Computational Radiology Laboratory.
Slide 17
STAPLE: Estimation Problem
• Complete data density: f (D, T | p, q)
• Binary ground truth Ti for each voxel i.
• Expert j makes segmentation decisions Dij.
• Expert performance characterized by sensitivity
p and specificity q.
– We observe expert decisions D. If we knew
ground truth T, we could construct
maximum likelihood estimates for each
expert’s sensitivity (true positive fraction)
and specificity (true negative fraction):
p, q arg max ln f (D, T | p, q)
ˆ ˆ
p, q
Computational Radiology Laboratory.
Slide 18
Expectation-Maximization
• Since we don’t know ground truth T, treat T as a
random variable, and solve for the expert performance
parameters that maximize:
Q( | (t1)
) E f (D,T | ) | D,
ln (t1)
• Parameter values θj=[pj qj]T that maximize the
conditional expectation of the log-likelihood function
are found by iterating two steps:
– E-step: Estimate probability of hidden ground truth T given a
previous estimate of the expert quality parameters, and take
expectation.
– M-step: Estimate expert performance parameters by
comparing D to the current estimate of T.
Computational Radiology Laboratory.
Slide 19
Probability Estimate of True Labels
Estimate probability of tissue class in reference standard:
W f (Ti s | Di , )
k
si
k
f (Ti s) f (Dij | Ti s, ) k
j
f (T s) f (D
i ij
| Ti s, ) k
s j
Computational Radiology Laboratory.
Slide 20
Binary Input: True Segmentation
Wi k f (Ti 1| Di ,p k , q k )
f (D ij | Ti 1, p q ) f (Ti 1)
k
j,
k
j
j
Ti f ( Dij | Ti, p j , q j ) f (Ti )
k k
j
k
k
k
k f (Ti 1) j:D
ij
p k j :D
1 j ij 0
(1 p k )
j
k f (Ti 0) j:D
ij
q k j :D
0 j ij 1
(1 q k )
j
f (Ti 1) : prior probability true label at voxel i is 1.
Wi k : conditional probability that true label is 1.
Computational Radiology Laboratory.
Slide 21
Expert Performance Estimate
k 1
i:Dij 1
Wi k
pj
i:Dij 1
Wi i:D 0 Wi
k
ij
k
k 1
i:Dij 0
(1 Wi )
k
qj
i:Dij 1
(1 Wi k ) i:D
ij 0
(1 Wi k )
p (sensitivity, true positive fraction) : ratio of expert
identified class 1 to total class 1 in the image.
i:Dij s
k
Wsi
k 1
q (specificity, true negative fraction) : ratio of expert
jss k
identified class 0 to total class 0 in the image. Wsi
i
Computational Radiology Laboratory.
Slide 22
Newborn MRI Segmentation
Computational Radiology Laboratory.
Slide 23
Newborn MRI Segmentation
Summary of segmentation quality (posterior probability
Pr(T=t|D=t) ) for each tissue type for repeated manual
segmentations.
Indicates limits of accuracy of interactive segmentation.
Computational Radiology Laboratory.
Slide 24
Expert and Student Segmentations
Test image Expert consensus Student 1
Student 2 Student 3
Computational Radiology Laboratory.
Slide 25
Phantom Segmentation
Image Expert Student
segmentation segmentations
Image Expert Students Voting STAPLE
Computational Radiology Laboratory.
Slide 26
STAPLE Summary
• Key advantages of STAPLE:
– Estimates ``true’’ segmentation.
– Assesses expert performance.
• Principled mechanism which enables:
– Comparison of different experts.
– Comparison of algorithm and experts.
• Extensions for the future:
– Prior distribution or extended models for
expert performance characteristics.
– Estimate bounds on parameters.
Computational Radiology Laboratory.
Slide 27
Image registration
• A metric: measures similarity of images
given an estimate of the transformation.
• Best metric depends on nature of the
images.
• Alignment quality ultimately possible
depends on model of transformation.
• The transformation is identified by
solving an optimization problem.
– Seek the transform parameters that
maximize the metric of image similarity
Computational Radiology Laboratory.
Slide 28
Validation of Registration
• Compare transformations
– Take some images, apply a transformation
to them.
– Estimate the transform using registration
– How well does the estimated transformation
match the applied transform?
• Check alignment of key image features
– Fiducial alignment
– Spatial overlap
• Segment structures, assess overlap after
alignment.
Computational Radiology Laboratory.
Slide 29
Intraoperative Nonrigid Registration
• Fast: it should not take more than 1 min to make the
registration.
• Robust: the registration should work with poor quality
image, artifacts, tumor...
• Physics based: we are not only concerned in the
intensity matching, but also interested in recovering the
physical (mechanical) deformation of the brain.
• Accurate: neuro-surgery needs a precise knowledge of
the position of the structures.
• Archip et al. NeuroImage 2007
Computational Radiology Laboratory.
Slide 30
Block Matching Algorithm
Similarity measure: coefficient of correlation [0 : 1]
Divide a global optimization problem in many simple local ones
Highly parallelizable, as blocks can be matched independently.
Computational Radiology Laboratory.
Slide 31
Block Matching Algorithm
Displacement
estimates are
noisy.
Computational Radiology Laboratory.
Slide 32
Patient-specific Biomechanical Model
Pre-operative Automatic Brain finite
image brain segmentation element model
(linear elastic)
Computational Radiology Laboratory.
Slide 33
Registration Validation
• Landmark matching assessment in six cases
• Parallel version runs in 35 seconds on a 10 dual 2GHz
PC cluster
– 7x7x7 block size
– 11x11x25 window Registration Error Evaluation Using Landmarks
– 1x1x1 step Correspondences
– 50 000 blocks 3
– 10 000 tetrahedra 2,5 Patient 1
Measured Error
Patient 2
2
Patient 3
1,5 Patient 4
Patient 5
1
Patient 6
• 60 landmarks: 0,5
0
– Average error = 0.75mm 0 5 10 15
– Maximum error = 2.5mm Displacement
– Data voxel size 0.8x0.8x2.5 mm3
Computational Radiology Laboratory.
Slide 34
Registration Validation
• 11 prospective consecutive cases,
• Alignment computed during the surgery.
• Estimate of the registration accuracy –
95% Hausdorff distance of the edges of
the registered preoperative MRI and the
intraoperative MRI.
Computational Radiology Laboratory.
Slide 35
Automatic selection of fiducials
(1)Non-rigid alignment of Contours extracted from (1)
preoperative MPRAGE. with the Canny edge
detector
95% Hausdorff
metric
Contours extracted from (2) computed
(2) Intraoperative whole
brain SPGR at 0.5T with the Canny edge
detector
Computational Radiology Laboratory.
Slide 36
Alignment improvement
Non-rigid registration – preop to intraop scans
(95% Hausdorff distance)
Max Displacement Rigid registration Non-Rigid Ratio
Tumor position Tumor pathology measured accuracy – preop to registration Rigid/Non-
(mm) intraop accuracy – preop to Rigid
(mm) intraop
(mm)
Case 1 right posterior frontal oligoastrocytoma Grade II 10.68 5.95 1.90 3.13
Case 2 left posterior temporal glioblastoma Grade IV 21.03 10.71 2.90 3.69
Case 3 left medial temporal glioblastoma Grade IV 15.27 7.65 1.70 4.50
Case 4 left temporal anaplastic oligoastrocytoma 10.00 6.80 0.85 8.00
Grade III
Case 5 right frontal oligoastrocytoma Grade II 9.87 5.10 1.27 4.01
Case 6 left frontal anaplastic astrocytoma Grade 17.48 10.20 3.57 2.85
III
Case 7 right medial temporal anaplastic astrocytoma Grade 19.96 9.35 2.55 3.66
III
Case 8 right frontal oligoastrocytoma Grade II 17.44 8.33 1.19 7.00
Case 9 right frontotemporal oligoastrocytoma Grade II 15.08 7.14 1.87 3.81
Case 10 right occipital anaplastic oligodendroglioma 9.48 5.95 1.44 4.13
Grade III
Case 11 left frontotemporal oligodendroglioma Grade II 10.74 4.76 0.85 5.60
AVG 14.27 7.44 1.82 4.58
Computational Radiology Laboratory.
Slide 37
Visualization of aligned data
• Matched preoperative fMRI and DT-MRI
aligned with intraoperative MRI.
Tensor alignment: Ruiz et al. 2000
Computational Radiology Laboratory.
Slide 38
Conclusion
• Validation strategies for registration:
– Comparison of transformations.
– Fiducials
• Manual, automatic.
– Overlap statistics – as for segmentation.
• Validation strategies for segmentation:
– Digital and physical phantoms.
– Comparison to domain experts.
– STAPLE.
Computational Radiology Laboratory.
Slide 39
Acknowledgements
Collaborators
• Neil Weisenfeld. • William Wells.
• Andrea Mewes. • Kelly H. Zou.
• Richard Robertson. • Frank Duffy.
• Joseph Madsen. • Arne Hans.
• Olivier Commowick.
• Karol Miller.
• Alexandra Golby.
• Michael Scott.
• Vicente Grau.
This study was supported by:
R01 RR021885, R01 EB008015, R01 GM074068
Computational Radiology Laboratory.
Slide 40