475

1988 ARRS Detection of Pneumothorax:

Executive Council Comparison of Digital and Conventional
Award Chest Imaging

Laurie L. Fajardo1 To investigate radiologists performance at interpreting digital radiographic images,

Bruce J. HiIIman1 we compared the detectability of pneumothoraces on computed radiographic chest
Gerald D. Pond1 images with 0.2-mm pixel size (2.5 lp/mm) with their detectability on matched conven-
tional screen-film images (5 lp/mm). Eight radiologists reviewed 50 computed and 50
Raymond F. Carmody1
screen-film chest radiographs from 25 patients with pneumothoraces and 25 patients
American Journal of Roentgenology 1989.152:475-480.

James E. Johnson1
with other (or no) abnormalities. Four of the readers who best detected pneumothoraces
William R. Ferrell2 on screen-film examinations performed worse when interpreting computed radiographic
studies; the other four readers detected pneumothoraces similarly with both techniques.
No relationship was found between the size of a pneumothorax and its likelihood of
detection by either technique.
These results raise concerns about implementing computed radiography for compre-
hensive chest imaging.

Despite the emergence of numerous computerized imaging techniques, most

images are still obtained in an analog fashion and displayed on film [1 ]. However,
computed radiography systems based on photostimulable plates provide excellent
imaging for excretory urography and for some facets of musculoskeletal and chest
radiography [2-8]. As a result, investigators are now focusing further on how well
radiologists interpret digital images, particularly in comparison with their ability to
detect specific abnormalities on conventional screen-film radiographs [2-7, 9].
To evaluate further the clinical suitability of digital chest imaging, we investigated
whether radiologists could adequately detect pneumothoraces on images gener-
ated by a state-of-the-art commercial computed radiography system (TCR, model
21 0; Toshiba, Tustin, CA) with 0.2-mm image pixel size. Our rationale for conducting
this experiment was that pneumothorax is an important clinical condition that is
radiographically manifest as a fine, linear density. As such, its visualization is
representative of subtle, high-frequency pulmonary structures, the identification of
which may be adversely affected both by the spatial resolution constraints imposed
by computed radiographys pixel size and by psychophysical influences that
Received March 1, 1988; accepted after revision accompany the interpretation of new types of images. To evaluate the usefulness
November 25, 1988.
of computed radiography in depicting pneumothoraces, we conducted an observer
Presented at the annual meeting of the American
Roentgen Ray Society, San Francisco, CA, May detection experiment in which we used matched screen-film and computed images.
1988. Winner of the Executive Council Award at
that meeting.
Department of Radiology, The University of
Subjects and Methods
Arizona and University Medical Center, Tucson, AZ
85724. Address reprint requests to L. L. Fajardo, We prospectively obtained matched screen-film and computed posteroanterior chest
Dept. of Radiology, University Medical Center, 1501 radiographs on 50 patients at the Tucson Veterans Administration Medical Center between
N. Campbell Ave., Tucson, AZ 85724.
2 Department of Systems and Industrial Engi- July 1986 and March 1987. Among these were 25 cases of pneumothorax. These were
derived mainly from patients who suffered pneumothorax during needle biopsy or placement
neering, The University of Arizona, Tucson, AZ
85724. of a central venous catheter; some cases were spontaneous. Preceding and subsequent
radiographs verified the diagnosis. There were also 25 age-matched control subjects. Each
AJR 152:475-480, March 1989
0361 -803x/89/1 523-0475 control image manifested from zero to five significant chest radiographic abnormalities (i.e.,
0 American Roentgen Ray Society potentially affecting patient care) (Table 1). We chose to include as controls abnormalities
 476 FAJARDO ET AL. AJA:152, March 1989

TABLE 1: Abnormalities Present on Screen-Film and Computed for the 36 x 36 cm imaging plate is 2.5 lp/mm (approximately half
Chest Radiographs the conventional screen-film system), corresponding to a pixel size
of 0.2 mm (Fig. 2).
Number of Cases

Diagnosis Patients with

Control Subjects Review of Examinations
Pneumothorax
(n = 25)
(n = 25)
A panel of three radiologists reviewed the screen-film and com-
Pulmonary infiltrate 4 8 puted images and determined by consensus up to five correct diag-
Emphysema 4 3 noses for each case. The 1 00 films were randomly assigned to four
Pleural effusion 6 5 groups of 25; each group had a similar number of screen-film and
Pulmonary mass 3 5
computed radiographs. The groups of films were organized so that
Pleural mass 4 3
Hilar mass 4 1 the experimental readers did not view both the screen-film and
Interstitial disease 1 3 computed images on any patient during the same session. Eight
Cardiomegaly 2 3 board-certified radiologists, who regularly interpret computed images
Congestive heart failure 1 2 as part of their routine clinical work, independently reviewed each
Pulmonary edema 1 1 group of images on separate occasions, using standard light boxes.
No acute disease 0 5 The test radiologists (readers) were not told that the purpose of the
Total 30 39 study was to evaluate the detection of pneumothorax.
For each image, readers indicated the presence or absence of
radiographic abnormalities from those listed in Table 1 and the degree
of certainty for each reported finding, on the basis of a six-point
other than pneumothorax, so that readers would not divine the confidence scale(1 = definitely not present, 2 = probably not present,
purpose of the study and direct their searches specifically toward
3 = possibly not present, 4 = possibly present, 5 = probably present,
American Journal of Roentgenology 1989.152:475-480.

finding pneumothoraces. The mean age for both experimental and

6 = definitely present).
control groups was 58.5 years (age range, 24-72 years). Informed
consent was obtained from all patients.
Screen-film exposures were produced at 105 kVp with an OPTI- Analysis
150 tube on a Sireskop (Siemens, Iselin, NJ) X-ray unit; a three-
phase, 1 2-pulse, thyristor-controlled, high-frequency generator (880 Consistent with the focus of the study, we analyzed only readers
kW) was used. Focal-spot size of the unit is 0.6-1 .0 mm and detection of pneumothoraces. Readers confidence ratings were cat-
exposures are terminated by an lontomat (Siemens) automatic ex- egonzed as either negative (confidence ratings 1-3) or positive (con-
posure control positioned behind the left lung field. Quanta Ill inten- fidence ratings 4-6) responses concerning whether a pneumothorax
sifying screens were used with WDR-SR334 film (Dupont, Wilming- was present. We used the negative and positive designations to
ton, DE) (Fig. 1A). The estimated spatial resolution of this screen-film calculate individual readers sensitivity and specificity for detecting
system is at least 5 lp/mm. pneumothoraces on screen-film and computed radiography images.
Computed radiographs were exposed on the same unit with the Receiver-operating-characteristic (ROC) curves were drawn to
same exposure parameters and geometry, but at 36 x 36 cm imaging characterize oath radiologists performance with screen-film and
plate was used. The imaging plates contain photostimulable phos- computed chest images. We did this by fitting binomial curves to
phors that store energy in quasi-stable states after excitation by oath radiologists confidence ratings by maximum likelihood estima-
electromagnetic radiation in the X-ray spectrum. Luminescent radia- tion [1 1 -1 3]. We then calculated the area under the screen-film and
tion is reemitted as blue phosphorescence when the imaging plate is computed radiography curves for each radiologist [1 2, 14].
scanned by the computed radiography systems 633-nm helium- To evaluate the hypothesis that screen-film and computed radio-
neon laser. Emitted light is recorded by a photomultiplier tube; the graphic imaging provided equivalent detectability of pneumothoraces,
electronic analog signal is subsequently digitized into 10 bits or 1024 we used two separate statistical analyses of the ROC data. We first
gray levels. For each exposure, two digital images (1 760 x 1760 performed analysis comparing the areas under the eight
a traditional
matrix) are produced. One image is generated with an algorithm that individual readers screen-film and computed radiography ROC
simulates the contrast scale of a conventional screen-film system; a curves, using a one-tailed, paired t test.
nonlinear gradient curve and a spatial frequency enhancement of 1.5 Our second analysis better reflects salient features of the distil-
times normal, centered at a spatial frequency of 0.35 cycles/mm, is bution of our ROC data. In this analysis, we tested the hypothesis
used (FIg. 1 B). The second image is produced by using a linear that if computed and film-screen radiography were equivalent tech-
gradient and spatial frequency of six times normal, also centered at nologies for detecting pneumothoraces, similar numbers of radiolo-
0.35 cycles/mm (FIg. 1 C). Unsharp-masking, high-boost filtering is gists clearly would perform better with one technology than with the
used in the frequency enhancement process of both computed im- other, whereas a number of readers would perform about the same
ages. High-boost filtering amplifies the high spatial frequencies while with both technologists. Thus, we categorized each readers corn-
passing the low spatial frequencies unattenuated. The two computed puted radiography ROC performance as being better, about the
radiographs were printed and displayed on a single 1 0 x 1 4 in. (25.4 same, or worse than that readers ROC performance with screen-
x 35.7 cm) single-emulsion film (Fuji CR-type 633; Tokyo). The actual film chest images. Readers were categorized as having performed
size of each image is 7 x 8.5 in. (1 7.8 x 21 .6 cm). Details of this better with either computed or screen-film radiography when the area
computed radiography system have been described [2, 10]. under the ROC curve for one technology was at least 3% greater
The spatial resolution of the computed radiography system was than it was for the other. Lesser differences resulted in the reader
determined by measuring the modulation transfer function of an being categorized as having performed the same with both. We then
exposed imaging plate by analyzing the fundamental-sine-wave fre- used four models of an ad hoc trinornial distribution, specific to our
quencies for periodic bar patterns with frequencies between 0.05 and data. With these models, we determined the probability that at least
3.0 lp/mm. The calculated maximal spatial resolution of the system as many of our readers would perform worse with computed than
 AJR:152, March 1989 DIGITAL VS CONVENTIONAL IMAGING OF PNEUMOTHORAX 477

Fig. 1.-Conventional screen-film (A), computed (B), and frequency-modified computed (C) radiographs show a right apical pneumothorax (arrows).

10 obtained, we averaged the concurring results. Pearson coefficient

American Journal of Roentgenology 1989.152:475-480.

correlation analysis was used to evaluate the relationship between

C
0 the size of pneumothorax and the probability of detection by corn-
08
0 puted radiography and screen film.
U-

a) 06
U)
C
Results

Sensitivity for the eight individual readers in detecting pneu-

04 mothoraces ranged from 0.76 to 0.92 for screen film, and
02 0.64 to 0.84 for computed radiography; specificity ranged
from 0.84 to 1 .00 for screen film, and 0.88 to 1 .00 for
00 computed radiography (Table 2). Four of the eight readers
2 3 4 5 6 correctly identified pneumothoraces on screen-film much
more often than on computed images. These radiologists
Spatial Frequency (lp/mm)
(readers 2, 6, 7, and 8) were, on average, 27.5% more
Fig. 2.-Graph shows modulation transfer function for computed radiog- sensitive in interpreting screen-film chest images. None of the
raphy system when a 36 x 36 cm imaging plate is used. other four radiologists had higher sensitivity in reviewing
computed than screen-film images; the difference in sensitivity
with screen-film radiography, if computed radiography were truly between the two technologies was no greater than 5% for
equivalent to screen-film imaging for the detection of pneumothoraces any of these four readers. Only readers 5 and 6 had a greater
(see Appendix). The four models of the trinomial distribution we
tested were (1) 1 0% of readers perform better with each technique,
80% about the same; (2) 20% of readers perform better with each TABLE 2: Sensitivity, Specificity, and ROC Curve Areas for
technique, 60% about the same; (3) 33% of readers perform better Screen Film and Computed Radiography in Detecting
with each technique, 34% about the same; (4)40% of readers perform Pneumothoraces
better with each technique, 20% about the same.
Sensitivity Specificity ROC Curve Analysis
We also tested therelationship between detectability and the size
of pneumothorax for each technique. Two individuals traced the lung Reader SF CR
and pneumothorax outlines from each screen-film radiograph with a
SF CR SF CR Diff
Area Area
sensitized stylus on a sensitized tablet of an image analyzer (model 1 0.76 0.72 1.00 1.00 0.910 0.913 +0.3
MOT-3, Carl Zeiss, New York, NY). Electronic signals generated by 2 0.88 0.68 0.88 0.88 0.962 0.857 -10.9
the tracing process are relayed to a computer that calculates the 3 0.76 0.76 0.92 0.92 0.890 0.898 +0.9
two-dimensional areas of the pneumothorax and involved lung. These 4 0.84 0.80 0.96 1.00 0.929 0.915 -1.5
areas were expressed as the percentage of pneumothorax, that is, 5 0.88 0.84 1.00 0.88 0.950 0.930 -2.1
the ratio of pneumothorax area to total lung area ([pneumothorax 6 0.92 0.68 0.84 0.92 0.962 0.902 -6.2
area/lung area] x 100). When the percentage of pneumothorax was 7 0.80 0.64 1.00 1.00 0.968 0.824 -14.9
greater than 10%, we required agreement of the calculations within 8 0.92 0.76 0.96 0.96 0.931 0.884 -5.0
5% or recalculated the areas; for images with percentage of pneu- Note.-ROC = receiver operating characteristic, SF = screen film, CA =

mothorax less than 1 0%, measurements were recalculated if they computed radiography, % Diff = percent difference between SF and CR =

did not agree within 2%. When a sufficient level of agreement was [(CR - SF)/SF].
 478 FAJARDO ET AL. AJR:152, March 1989

- x Screen film
#{149}
CR
a)
E3
z4

0.800 0.820 0.840 0.860 0.880 0.900 0.920 0.940 0.960 0.980 1.000
Area Under ROC Curve

Fig. 3.-Graph shows areas under individual readers receiver operating Pneumothorax (%)
characteristic (ROC) curves for detecting pneumothoraces on screen-film
and computed radiography (CR) images. Four readers who best detected Fig. 4.-Graph shows size distribution of pneumothoraces among 25
pneumothoraces on screen-film images did most poorly with computed study cases.
images.

than 5% disparity in specificity between screen-film and corn- TABLE 3: Size of Pneumothorax for Cases Missed by 50% or
puted radiography images. More of the Observers on Screen-Film or Computed Chest
Areas under the ROC curves for individual readers detec- Radiographs
tion of pneumothoraces ranged from 0.890 to 0.968 for Observers Who Missed the
American Journal of Roentgenology 1989.152:475-480.

screen-film images and 0.824 to 0.930 for computed images. Pneumothorax (%)
Size of
The three readers with the greatest areas under their screen- Pneumothorax on Screen-
film ROC curves and one other radiologist detected pneu- (%) Film on Computed
mothoraces much less well on computed radiography images Radiographs
Radiographs
(readers 2, 6, 7, and 8) (Fig. 3). As shown in Table 2, the 25.2 0 62.5
areas under the computed radiography ROC curves for these 16.7 12.5 87.5
radiologists were, on average, 9.3% less than the areas under 1.4 25.0 62.5
their screen-film curves. The other four radiologists performed 3.4 25.0 50.0
7.3 25.0 50.0
similarly with both techniques, having an average difference
8.0 50.0 0
in the areas under their screen-film and computed radiography 2.7 50.0 50.0
ROC curves of only 1 .2%. A one-tailed paired t test of the 5.5 50.0 12.5
areas under individual readers screen-film and computed 1.5 62.5 75.0
7.3 100.0 87.5
radiography ROC curves shows a significant difference be-
tween the two techniques (t = 2.489, df = 7, p < .003).
Our ad hoc trinomial distribution analysis indicates that, by
all four models tested (categorizing readers with differences frequency of detection or diagnostic certainty for either tech-
in performance with each technology of 3% or less as being
nology.
equivalent), the probability is less than .03 that the readers
could have performed as poorly as they did with computed
radiography, if computed radiography were truly equivalent
Discussion
to screen-film imaging for detection of pneumothoraces This study extends previous psychophysical investigations
(model 1 : p = .003; model 2: p = .01 9; model 3: p = .025; of the suitability of computed imaging for chest radiography.
model 4: p = .015). If we apply a more stringent criterion for The results of the ROC analyses indicate that radiologists are
categorizing a reader as having performed equally with both better able to discriminate pneumothoraces on conventional
technologies-with a difference in performance of either 1% screen-film radiographs than on computed radiographic im-
or 2%-the result is even more significant. if we make
Only ages. The number of false-positive diagnoses of pneumotho-
the criterion a difference in performance between the two rax made with each technique was small: only 1 1 of the 400
technologies as large as 6% (five of eight readers with no possible responses for screen film and 1 0 for computed
appreciable difference) is the probability just greater than .05 radiography. This is important because of the low clinical
of obtaining a result as unfavorable to computed radiography prevalence of pneumothorax and because of the morbidity
by chance (and then only for trinomial model 2). associated with improper treatment. However, the radiolo-
The distribution of pneumothorax size among the 25 cases gists had significantly more false-negative diagnoses on the
is shown in Figure 4. Sixty-eight percent of the pneumotho- computed images. Because of the morbidity associated with
races involved 10% or less of the total area of the involved delayed treatment for missed pneumothoraces, these data
lung. In 10 cases (five for screen film and eight for computed suggest that computed radiography may be inadequate for
radiography), 50% or more of the observers failed to detect depicting pneumothoraces in clinical practice.
the pneumothorax (Table 3). There was no statistically signif- Previous reports have focused on the poorer spatial reso-
icant correlation between the size of pneumothorax and either lution of computed vs screen-film radiography to explain
 AJR:152, March 1989 DIGITAL VS CONVENTIONAL IMAGING OF PNEUMOTHORAX 479

differences in radiologists performances in detecting high- radiography for clinical chest diagnosis. Further psychophys-
frequency, low-contrast abnormalities such as pneumothora- ical researchis needed to determine what improvements are
ces and interstitial thickening [3-5]. These studies, which needed in physical factors of the computed radiography sys-
dealt with small numbers of these types of abnormalities, tem and/or radiologists interactions with computed images.
have provided conflicting results, variously indicating spatial
resolution requirements of 0.4-mm [3], 0.2-mm [5], and 0.1-
mm [4] pixel sizes. The biphasic nature of our results sug- Appendix
gests that influences other than spatial resolution may be As described in the analysis section, we tested the hypothesis that
important in explaining why radiologists detect pneumotho- the two methods did not differ in detectability of pneumothorax using
races less well on computed than on screen-film images. The the one-tailed paired t test. This test indicated that computed radiog-
radiologists who were among the most proficient in interpret- raphy was less effective (p < .003). However, this test assumes an
ing the screen-film images performed much more poorly with approximately normal distribution of differences in detectability and
is weighted in proportion to the size of differences in detectability
computed radiography images; the other four radiologists
among readers. Thus, in principle, the null hypothesis-that radiolo-
performances with and computed
screen-film radiography
gists detect pneumothoraces equally well with computed and screen-
differed little. Granted, it is possible to hypothesize that the
film radiography-could be inappropriately rejected using the t test
former group in some manner made better use than the latter because of several large differences in performance.
group of the greater spatial resolution that screen-film radiol- Indeed, our interpretation of the data suggests that a categorical
ogy offers. However, an alternative explanation suggested by approach, which does not weight differences among readers perform-
the dichotomy of performances is that subtle diagnostic clues ances according to magnitude, better reflects the results of the
are portrayed better on the screen film than on the computed experiment, where some readers did as well with either method and
images, clues that are better exploited
by the more proficient some interpreted computed radiographs less accurately. Using this
American Journal of Roentgenology 1989.152:475-480.

radiologists. Apparent differences between the two technol- categorical approach, we determined whether our result could have
ogies that might contribute to explaining our result include occurred by chance by calculating the total probability of the possible
outcomes that are as favorable to computed radiography as what we
computed radiographys lesser spatial resolution, greater im-
observed and the outcomes less favorable to computed radiography
age noise, and smaller image size, and radiologists relative
than what we observed. Our null hypothesis was that computed and
unfamiliarity with computed radiography. Indeed, several of screen-film technologies are equivalent for detecting pneumothora-
these factors may have been important in influencing our ces.
results. The effects of each of these, and perhaps other With the outcome we observed, this is easy to do. Even assuming
influences, bear further investigation. a difference of 1% or less as the criterion for doing about the same
As with any imaging study that uses selected cases and a with both methods, no reader did better with computed radiography.
small group of radiologist readers, questions will arise about Hence, we need to sum the probability of all the outcomes in which
the generalizability of our results. Our cases were not ran- no readerdetects pneumothoraces better with computed radiography
domly selected, nor can we be sure that the 25 pneumotho- and the probability of all the outcomes in which no fewer readers
than we observed do better with screen-film images. Outcomes for
races that composed the case material are representative of
which any readers do better with computed radiography, even if
all pneumothoraces encountered in clinical practice. Nonethe-
balancecf by more readers detecting pneumothoraces better with
less, these considerations are not germane to the purpose of screen-film images, are ones that are more favorable to computed
the study. Insofar as the selected cases portray pneumotho- radiography, because they would demonstrate the possibility that
races that, if discovered, would affect clinical care, the sample computed radiography could be used more effectively than screen-
is useful in determining whether computed radiography per- film images.
mits detection comparable with screen-film radiography of an Our null hypothesis (i.e., that the two technologies are equivalent)
important selection of pneumothoraces. The radiologist read- was that a reader taken at random had probability Pw of doing worse
ers were chosen primarily on the basis of their willingness to with computed radiography, probability Ps of doing about as well with
participate in the study. Although they were not randomly both technokgses, and probability p of doing better with computed
radiography, where Pb Pw. Such a distribution is represented by the
selected, because their number is a sizable fraction (47%) of
multinomial with three possible outcomes, the trinomial. In this ex-
our 1 7-member group, the readers are representative of our
perlment, none of eight readers performed better with computed
department. To the extent that radiologists in our department radiography and M of eight performed equally well with both methods
are like those in other groups, similar performances can be (the value of M depending on the level of difference in performance
expected in interpreting computed and screen-film chest im- accepted as equivalent).
The probability of getting results at least as
ages. unfavorable for computed radiography as those we obtained is the
In summary, our study indicates that some radiologists will sum of the probability of all the outcomes in whith no reader does
detect pneumothoraces less well on digital chest images than better with computed radiology and the probability of all the outcomes
on screen-film examinations. We chose to investigate the in which no fewer than M do better with screen film. It is given by the
following equation:
detectability of pneumothoraces because we think pneumo-
thorax is representative of other high-frequency, low-contrast
abnormalities, the adequate depiction of which is important
mO [m!(88-- m)!] (Pw)8m(Ps)m.
to chest diagnosis. Because an imaging system capable of
supplanting conventional chest radiography should fulfill the Thus, our trinomial test is ad hoc, developed specifically for the
need to visualize adequately all clinically important structures, results of this study. It does not represent a general test for the
there should be concern over fully implementing computed performance of any two radiographic systems.
 480 FAJARDO ET AL. AJR:152, March 1989

REFERENCES 7. Fraser RG, Breatnach E, Barnes GT. Digital radiography of the chest:
dinical experience with a prototype unit. RadioGraphics 1985;5:415-428
1. Templeton AW, Dwyer SJ Ill. Johnson JA, et al. An on-line digital image 8. Merritt CRB, Tutton RH, Bell KA, et al. Clinical application of digital
management system. Radiology 1984;152:321-325 radiography: computed radiographic imaging. RadioGraphics 1985;5:
2. Fajardo LF, Hillman B,J, Hunter TB, Claypool HR, Westerman BR, Mockbee 397-414
B. Excretory urography using computed radiography. Radiology 9. Seeley GW, Roehng H, Nudelman S. Frost MM, Capp MP. Psychophysical
1987;162:345-351 evaluation of video systems compared to film. AppI Photogr Eng
3. Lams PM, Cocklin ML. Spatial resolution requirements for digital chest 1978;4: 160-1 65
radiographs: an ROC study of observer performance in selected cases. 10. Sonoda M, Takano M, Miyahara J, Kato H. Computed radiography utilizing
Radiology 1986;158: 11-19 scanning laser stimulated luminescence. Radiology 1983;148:833-838
4. MacMahon H, Vybomy CJ, Metz CE, Dci K, Sabeti V. Solomon SL. Digital 11. Metz CE. Basic principles of ROC analysis. Semin NucI Med 1978;8:
radiography of subtle pulmonary abnormalities: an ROC study of the effect 283-298
of pixel size on observer performance. Radiology 1986;158:21-26 12. Swets JA. ROC analysis applied to the evaluation of medical imaging
5. Goodman LR, Foley DW, Wilson CR, Rimm AA, Lawson TL. Digital and techniques. Invest Radiol 1979;14:109-121
conventional chest images: observer performance with film digital radiog- 13. Dorfman DD, Alf E. Maximum likelihood estimation of parameters of signal
raphy system. Radiology 1986;158:27-33 detection and determination of confidence intervals: rating method data. J
6. Chakraborty DP, Breatnach ES, Yester MV, Soto B, Barnes GT, Fraser Math Psychol 1969;6:487-497
RG. Digital and conventional chest imaging: a modified ROC study of 14. Hanley JA, McNeil BJ. The meaning and use of area under a receiver
observer performance using simulated nodules. Radiology 1986;158: operating characteristic (ROC) curve. Radiology 1982;143:29-36
35-39
American Journal of Roentgenology 1989.152:475-480.