Ultrasound Obstet Gynecol 2010; 36: 32–36
Published online 2 June 2010 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/uog.7577
Learning curve for lung area to head circumference ratio
measurement in fetuses with congenital diaphragmatic hernia
R. CRUZ-MARTINEZ, F. FIGUERAS, O. MORENO-ALVAREZ, J. M. MARTINEZ, O. GOMEZ,
E. HERNANDEZ-ANDRADE and E. GRATACOS
Fetal and Perinatal Medicine Research Group, Department of Maternal–Fetal Medicine, Hospital Clinic-IDIBAPS, University of Barcelona
and Centre for Biomedical Research on Rare Diseases (CIBER-ER), Barcelona, Spain
K E Y W O R D S: congenital diaphragmatic hernia; CUSUM; learning curve; O/E-LHR
ABSTRACT
Objective To assess the learning curve for the fetal lung
area to head circumference ratio (LHR) calculation in
fetuses with congenital diaphragmatic hernia (CDH).
Methods Three trainees with the theoretical knowledge,
but without prior experience in the LHR measurement,
were selected. Each trainee and one experienced examiner
measured the observed to expected (O/E)-LHR in
the lung contralateral to the side of the hernia by
two methods – manual tracing of lung borders and
multiplication of the longest diameters – in a cohort of
95 consecutive CDH fetuses. The average difference
between the three trainees and the expert in the O/E-LHR
measurement was calculated. A difference below 10%
was considered to indicate an accurate measurement. The
average learning curve was delineated using cumulative
sum analysis (CUSUM).
Results The CUSUM plots demonstrate that the learning
curve was achieved by 77 and 72 tests performed for the
area obtained by the manual-tracing and multiplicationof-the-longest-diameter methods, respectively.
Conclusion The minimum number of scans required
for an inexperienced trainee to become competent in
examining the LHR is on average 70. Copyright  2010
ISUOG. Published by John Wiley & Sons, Ltd.
INTRODUCTION
Congenital diaphragmatic hernia is associated with a high
postnatal mortality rate owing to pulmonary hypoplasia1 .
Lung area to head circumference ratio (LHR) measured by
ultrasonography provides prediction of postnatal survival,
with an improvement in postnatal survival with increasing
prenatal values2,3 . However, the reported results on LHR
performance lack consistency, probably because of different criteria being used for patient selection and different
methodologies being used in the estimation of the lung
area4 – 8 . Recently, it has been reported that area tracing rather than diameter-derived area is a more reliable
parameter9 . Furthermore, the LHR changes with changes
in gestational age and fetal size, increasing the variability
of this estimation10,11 . Attempts to account for this variability have been made by adjusting the observed LHR
against that expected for gestational age, in the observed
to expected LHR ratio (O/E-LHR). Clinical studies suggest that O/E-LHR yields more accurate prediction of
pulmonary hypoplasia12,13 . LHR is a demanding measurement owing to the difficult delineation of the borders
of the lung in relation to its surrounding tissues, and concerns exist as to whether the operator’s competence could
have influenced the results of some published studies8 . No
previous paper has evaluated the learning curve of these
parameters in order to set the minimum number of cases
required for an operator to yield reliable measurements.
Cumulative sum (CUSUM) analysis is a graphic method
that has been used for quality control in monitoring
doctors’ performance on a case-by-case basis, showing
changes in competence over time14 – 17 and to assess
learning curves of trainees in different diagnostic methods and surgical operations as well as fetal ultrasound
measurements18 – 21 .
The aim of this study was to determine the number of
ultrasound examinations that are necessary for an inexperienced sonographer to perform in order to achieve
competence in performing the O/E-LHR measurement
in fetuses with isolated congenital diaphragmatic hernia
(CDH).
Correspondence to: Dr F. Figueras, Fetal and Perinatal Medicine Research Group, Maternal–Fetal Medicine Department, Hospital Clinic,
University of Barcelona, Barcelona, Spain (e-mail: ffiguera@clinic.ub.es)
Accepted: 29 June 2009
Copyright  2010 ISUOG. Published by John Wiley & Sons, Ltd.
ORIGINAL PAPER
Learning curve for LHR
METHODS
Subjects
Three fetal medicine fellows with more than 1 year
of experience in ultrasonography but who had never
previously measured the LHR in fetuses with CDH
were instructed on this measurement by one experienced
operator (E.G. or J.M.M.) by means of offline video
clips. Between October 2005 and February 2009, 95
consecutive fetuses with isolated CDH referred to our
hospital for CDH were included. In order to estimate
the O/E-LHR in each case, the head circumference was
measured by one experienced examiner and the lung area
was then estimated firstly by the experienced examiner
and secondly by each of the trainees, blinded to previous
measurements. The average difference between the three
trainees and the expert in the O/E-LHR measurement was
calculated. The measurements were considered accurate
when the individual measurements, or the mean value
of the three trainees when considered together, were less
than 10% different from the expert measurement. When
the difference, or the mean difference, was ≥ 10% it was
considered a failure. Similarly, lack of success in obtaining
the required ultrasound plane was also considered for the
analysis as a failure, with the mean of the remaining two
trainees used for the group analysis in such cases. This
study was approved by the hospital ethics committee, and
patients provided written informed consent.
33
both for each individual trainee and for their averaged
measurements. In short, the CUSUM values are plotted
on the y-axis and the number of examinations on the
x-axis. Horizontal lines are plotted at regular intervals on
the y-axis, defining h0 and h1 for the spacing between
unacceptable and acceptable boundary lines, respectively.
The CUSUM graph is the running sum of a series
of increments (with each failure) and decrements (with
each success). When the number of failures exceeds the
unacceptable failure rate or is lower than the acceptable
failure rate, the graph shows a positive or a negative slope,
respectively. The acceptable and unacceptable failure rates
were set at 10% (i.e. p0 = 0.1) and 25% (p1 = 0.25),
respectively, and Type 1 (α) and Type 2 (β) error rates
were set at 0.1. According to the standard formulae,
the decrement with each success was calculated as (s) =
Q/(P+Q) = 0.18/(0.92 + 0.18) = 0.17, the increment
of each failure as (1 − s) = 1 − 0.17 = 0.83, and the
spacing between the unacceptable (h0) and the acceptable
(h1) boundaries as a/(P+Q) = 2.2/(0.92 + 0.18) = 215 ,
where P = ln(p1 /p0 ), Q = ln((1 − p0 )/(1 − p1 )), and
a = ln((1 − β)/α). This means that for each failure the
line will go up by 0.83 units and for each success
it will go down by 0.17, with competence declared
when the plot falls below the lower boundary of
h1 , i.e. dropping below two consecutive boundary
lines17 .
RESULTS
Lung to head ratio measurements
Fetal ultrasound examinations were performed using a
Siemens Sonoline Antares (Siemens Medical Systems,
Malvern, PA, USA) ultrasound machine equipped with
a 6–2-MHz linear curved-array transducer. The lung
contralateral to the side of the hernia was evaluated in
a cross-sectional view of the fetal thorax at the level of
the cardiac four-chamber view. The LHR was estimated
firstly by multiplication of the longest diameter by its
widest perpendicular diameter and secondly by manual
tracing of the lung borders as described previously11,22 .
The expected LHR values were calculated using normal
reference ranges according to gestational age and the side
of the hernia for both the longest-diameter and the tracing
methods11,13 . The observed LHR value was compared
with the expected LHR in order to calculate the O/ELHR as (LHR observed/LHR expected) × 100. Only one
set of measurements for each patient was included in the
analysis.
Statistical analysis
CUSUM analysis was performed for each of the two
methods of lung area estimation, and for each component
measurement of the longest diameter method, according
to published standard methodology14 – 21 , using Excel
for Windows 2007 (Microsoft Corp., Redmond, WA,
USA) statistical software. CUSUM charts were generated
Copyright  2010 ISUOG. Published by John Wiley & Sons, Ltd.
Among the 95 included cases, 80 had left-sided CDH
and 15 right-sided CDH. In 71 (74.7%) cases there was
intrathoracic herniation of the liver. Mean gestational
age at evaluation was 30.3 ± 4.3 weeks. The O/E-LHR as
measured by the experienced examiner was less than 26%
in 25 (26.3%), 26–30% in 18 (18.9%), 30–45% in 28
(29.5%) and above 45% in 24 (25.3%) fetuses.
Among the total 285 O/E-LHR measurements performed by the three trainees, the number of failed examinations (≥ 10% difference with the experienced operator)
was 38 (40.0%) and 41 (43.2%) for the manual-tracing
and the two-longest-diameters methods, respectively, and
32 (33.7%) and 40 (42.1) for the measurement of the
longest and widest diameters, respectively. These include
13 failures to obtain an adequate image owing to the fetal
position.
The CUSUM plots obtained using the average measurements of the trainees demonstrated that with
the manual tracing method competence was achieved
by 77 cases performed (Figure 1a), and with the
two-longest-diameters approach it required 72 cases
(Figure 1b). The CUSUM plot was also delineated for
each component of the two-longest-diameters method.
The number of attempts of the learning curve was
slightly lower for the longest (Figure 1c) than for
the widest diameter (Figure 1d) (84 vs. 96, respectively). Similar findings were observed when individual CUSUM charts were generated for each of the
trainees.
Ultrasound Obstet Gynecol 2010; 36: 32–36.
Cruz-Martinez et al.
34
(b) 30
28
26
24
22
20
18
16
14
12
10
8
6
4
2
0
CUSUM values
CUSUM values
(a) 30
28
26
24
22
20
18
16
14
12
10
8
6
4
2
0
1
11
21
31
41
51
61
71
81
91
1
11
21
41
51
61
71
81
91
(d) 30
28
26
24
22
20
18
16
14
12
10
8
6
4
2
0
71
81
91
CUSUM values
CUSUM values
(c) 30
28
26
24
22
20
18
16
14
12
10
8
6
4
2
0
31
Number of attempts
Number of attempts
1
11
21
31
41
51
61
71
81
91
Number of attempts
1
11
21
31
41
51
61
Number of attempts
Figure 1 Cumulative sum (CUSUM) graphs of the observed to expected lung area to head circumference ratio, measured by the manual
tracing method (a) and the longest diameter approach (b), and of measurements of the pulmonary longest diameter (c) and the pulmonary
widest diameter (d). The black graph in each chart shows the CUSUM graph of the average measurement of the three trainees, with the
arrows indicating the number of attempts necessary to achieve statistically significant competence. The gray graphs show the individual
CUSUM charts of the three trainees. Horizontal lines show acceptable/unacceptable boundary lines of the CUSUM score.
DISCUSSION
This study evaluated the experience required to achieve an
acceptable competence on measuring the O/E-LHR. Our
findings suggest that to obtain a reliable calculation of the
O/E-LHR requires considerable experience. Differences
in operators’ competence might contribute to explaining
the variability among studies and controversies observed
in the literature2 – 8 . Our findings are consistent with the
notion that some fetal ultrasound measurements require
formal training before achieving competence and comparability between observers23 – 25 . In addition, taking into
account the rarity of this disease and the number of scans
necessary to achieve competence, this training must be
performed in referral units with experience in order to
accelerate the learning process.
In this study, the a priori acceptable rate of disagreement > 10% between two competent examiners was set
at 10% and the unacceptable rate was set at 25%; as
O/E-LHR is a percentage with a mean of around 30%
Copyright  2010 ISUOG. Published by John Wiley & Sons, Ltd.
in our population, the 10% limit would mean a variability of ±3% in absolute values. The CUSUM analysis
demonstrated that the number of scans necessary to
achieve competence under these requirements was similar
between the two methods evaluated, tracing and twodiameters measurement. This finding is in line with the
clinical observation that one of the main challenges in the
calculation of LHR in CDH is to obtain a cross-sectional
view of the fetal thorax in a reproducible fashion. While
the LHR is supposed to be measured in a four-chamber
view of the heart, this view is not easy to obtain because
the heart has usually lost its horizontal normal disposition. Another interesting finding is that the transverse
lung diameter required a higher number of ultrasound
examinations than the longitudinal diameter to achieve
competence. To estimate the limits of the lung tissue at
the time of calculating the transverse diameter can also be
challenging, and care must be taken not to include other
viscera, such as the spleen, which in very severe cases is
located very close to the contralateral lung.
Ultrasound Obstet Gynecol 2010; 36: 32–36.
Learning curve for LHR
CUSUM analysis has been used in several studies as
a quality control procedure to continuously monitor
doctors’ performance among different specialties16 and
in assessing imaging processes in fetal medicine, such
as biometric measurements17 . Clinical studies20,21 have
demonstrated that CUSUM analysis is a useful tool for
quantifying the duration of a training regimen, providing
an early indication of performance as well as individual
difficulties or failure to achieve competence. The method
is practical, simple to apply, easy to introduce and
has proved popular with trainees in some medical and
surgical specialties19,26 . This method determines whether
trainees have reached the acceptable predefined level of
performance.
The strength of this study is that each operator was
blinded to other measurements and they were also
blinded to their performance until the end of the study.
In addition, each individual component of the O/ELHR and two different methods of calculation were
addressed. However, the study also has some limitations.
Firstly, it could be argued that both the expert and
the trainees were not representative of their respective
populations. Secondly, the head circumference was only
estimated by the experienced examiner. It could be argued
that our LHR learning curve neglected this component
but we recognize that, as the trainees had 1 year’s
experience with ultrasonography at the beginning of the
study, the impact of head circumference measurement
on the LHR learning curve could only be marginal.
Thirdly, although estimation of the LHR is likely to
be more demanding in cases with right-sided CDH
and those with herniation of the liver, sample size
was limited and did not allow the construction of a
learning curve for these subgroups. In addition, the study
design did not allow monitoring the competence once
acquired and, therefore, loss of competence could not be
evaluated.
In conclusion, this study provides evidence that the
O/E-LHR estimation requires a substantial number of
measurements for a non-experienced examiner to achieve
competence. Studies on CDH where O/E-LHR is used
should report on the experience of the observers to
facilitate comparability of studies.
ACKNOWLEDGMENTS
The study was supported by grants from the
Fondo de Investigación Sanitaria (FIS) of the Ministerio de Sanidad y Consumo with the grant PI
06/0585 and by EuroSTEC LSHB-CT-2006-037409.
R.C.M. is supported by Marie Curie Host Fellowships
for Early Stage Researchers, FETAL-MED-019707-2.
E.H.A. was supported by a Juan de la Cierva post-doctoral
fellowship, Fondo de Investigaciones Sanitarias, Spain.
R.C.M. and O.M.A. wish to thank the Mexican
National Council for Science and Technology (CONACyT), in Mexico City, for supporting their predoctoral
stay at the Hospital Clinic in Barcelona, Spain.
Copyright  2010 ISUOG. Published by John Wiley & Sons, Ltd.
35
REFERENCES
1. Stege G, Fenton A, Jaffray B. Nihilism in the 1990s: the true
mortality of congenital diaphragmatic hernia. Pediatrics 2003;
112: 532–535.
2. Jani JC, Nicolaides KH, Gratacos E, Vandecruys H, Deprest JA.
Fetal lung-to-head ratio in the prediction of survival in severe
left-sided diaphragmatic hernia treated by fetal endoscopic tracheal occlusion (FETO). Am J Obstet Gynecol 2006; 195:
1646–1650.
3. Laudy JA, Van Gucht M, Van Dooren MF, Wladimiroff JW,
Tibboel D. Congenital diaphragmatic hernia: an evaluation of
the prognostic value of the lung-to-head ratio and other prenatal
parameters. Prenat Diagn 2003; 23: 634–639.
4. Ba’ath ME, Jesudason EC, Losty PD. How useful is the lungto-head ratio in predicting outcome in the fetus with congenital
diaphragmatic hernia? A systematic review and meta-analysis.
Ultrasound Obstet Gynecol 2007; 30: 897–906.
5. Hedrick HL, Danzer E, Merchant A, Bebbington MW, Zhao H,
Flake AW, Johnson MP, Liechty KW, Howell LJ, Wilson RD,
Adzick NS. Liver position and lung-to-head ratio for prediction
of extracorporeal membrane oxygenation and survival in
isolated left congenital diaphragmatic hernia. Am J Obstet
Gynecol 2007; 197: 422 e.1–4.
6. Sbragia L, Paek BW, Filly RA, Harrison MR, Farrell JA,
Farmer DL, Albanese CT. Congenital diaphragmatic hernia
without herniation of the liver: does the lung-to-head ratio
predict survival? J Ultrasound Med 2000; 19: 845–848.
7. Bretelle F, Mazouni C, D’Ercole C, Chaumoitre K, Chau C,
Desbrieres R. Fetal lung-head ratio measurement in the
evaluation of congenital diaphragmatic hernia. J Pediatr Surg
2007; 42: 1312–1313; author reply 1313–1314.
8. Arkovitz MS, Russo M, Devine P, Budhorick N, Stolar CJ. Fetal
lung-head ratio is not related to outcome for antenatal diagnosed
congenital diaphragmatic hernia. J Pediatr Surg 2007; 42:
107–110; discussion 110–111.
9. Jani J, Peralta CF, Benachi A, Deprest J, Nicolaides KH. Assessment of lung area in fetuses with congenital diaphragmatic
hernia. Ultrasound Obstet Gynecol 2007; 30: 72–76.
10. Yang SH, Nobuhara KK, Keller RL, Ball RH, Goldstein RB,
Feldstein VA, Callen PW, Filly RA, Farmer DL, Harrison MR,
Lee H. Reliability of the lung-to-head ratio as a predictor of
outcome in fetuses with isolated left congenital diaphragmatic
hernia at gestation outside 24–26 weeks. Am J Obstet Gynecol
2007; 197: 30 e.1–7.
11. Peralta CF, Cavoretto P, Csapo B, Vandecruys H, Nicolaides KH. Assessment of lung area in normal fetuses at
12–32 weeks. Ultrasound Obstet Gynecol 2005; 26: 718–724.
12. Jani J, Nicolaides KH, Benachi A, Moreno O, Favre R, Gratacos E, Deprest J. Timing of lung size assessment in the prediction
of survival in fetuses with diaphragmatic hernia. Ultrasound
Obstet Gynecol 2008; 31: 37–40.
13. Jani J, Nicolaides KH, Keller RL, Benachi A, Peralta CF,
Favre R, Moreno O, Tibboel D, Lipitz S, Eggink A, Vaast P,
Allegaert K, Harrison M, Deprest J. Observed to expected lung
area to head circumference ratio in the prediction of survival in
fetuses with isolated diaphragmatic hernia. Ultrasound Obstet
Gynecol 2007; 30: 67–71.
14. Biau DJ, Porcher R, Salomon LJ. CUSUM: a tool for ongoing
assessment of performance. Ultrasound Obstet Gynecol 2008;
31: 252–255.
15. Bolsin S, Colson M. The use of the Cusum technique in the
assessment of trainee competence in new procedures. Int J Qual
Health Care 2000; 12: 433–438.
16. Lim TO, Soraya A, Ding LM, Morad Z. Assessing doctors’
competence: application of CUSUM technique in monitoring
doctors’ performance. Int J Qual Health Care 2002; 14:
251–258.
17. Weerasinghe S, Mirghani H, Revel A, Abu-Zidan FM. Cumulative sum (CUSUM) analysis in the assessment of trainee
Ultrasound Obstet Gynecol 2010; 36: 32–36.
Cruz-Martinez et al.
36
18.
19.
20.
21.
competence in fetal biometry measurement. Ultrasound Obstet
Gynecol 2006; 28: 199–203.
Biau DJ, Williams SM, Schlup MM, Nizard RS, Porcher R.
Quantitative and individualized assessment of the learning curve
using LC-CUSUM. Br J Surg 2008; 95: 925–929.
McCarter FD, Luchette FA, Molloy M, Hurst JM, Davis K
Jr, Johannigman JA, Frame SB, Fischer JE. Institutional and
individual learning curves for focused abdominal ultrasound
for trauma: cumulative sum analysis. Ann Surg 2000; 231:
689–700.
Rozenberg P, Porcher R, Salomon LJ, Boirot F, Morin C,
Ville Y. Comparison of the learning curves of digital examination and transabdominal sonography for the determination
of fetal head position during labor. Ultrasound Obstet Gynecol
2008; 31: 332–337.
Young A, Miller JP, Azarow K. Establishing learning curves
for surgical residents using Cumulative Summation (CUSUM)
Analysis. Curr Surg 2005; 62: 330–334.
Copyright  2010 ISUOG. Published by John Wiley & Sons, Ltd.
22. Metkus AP, Filly RA, Stringer MD, Harrison MR, Adzick NS.
Sonographic predictors of survival in fetal diaphragmatic hernia.
J Pediatr Surg 1996; 31: 148–151; discussion 151–152.
23. Maiz N, Kagan KO, Milovanovic Z, Celik E, Nicolaides KH.
Learning curve for Doppler assessment of ductus venosus flow at
11 + 0 to 13 + 6 weeks’ gestation. Ultrasound Obstet Gynecol
2008; 31: 503–506.
24. Cicero S, Dezerega V, Andrade E, Scheier M, Nicolaides KH.
Learning curve for sonographic examination of the fetal nasal
bone at 11–14 weeks. Ultrasound Obstet Gynecol 2003; 22:
135–137.
25. Braithwaite JM, Kadir RA, Pepera TA, Morris RW, Thompson PJ, Economides DL. Nuchal translucency measurement:
training of potential examiners. Ultrasound Obstet Gynecol
1996; 8: 192–195.
26. Toker A, Tanju S, Ziyade S, Kaya S, Dilege S. Learning curve in
videothoracoscopic thymectomy: how many operations and in
which situations? Eur J Cardiothorac Surg 2008; 34: 155–158.
Ultrasound Obstet Gynecol 2010; 36: 32–36.