This talk will present the major elements to design, execute, and analyze reader studies. In a reader study, clinicians perform an objective task given medical images. The purpose of a reader study is to assess the clinicians’ performance doing the task given the images or to compare performance given images from a new technology to the performance given images from the reference technology. The clinician is part of the technology assessment. The clinicians are the readers and the medical images are the cases.  Of course, there is bias and variability from human evaluations. As such, analysis methods for reader performance (variance estimates, confidence intervals, hypothesis tests) need to account for the bias and variability from the readers while they also account for the variability from the cases. Such an analysis is referred to as a multi-reader, multi-case (MRMC) analysis. An MRMC analysis can summarize average reader performance that is expected to generalize to the population of readers and the population of cases.

• This talk has been given as training for medical imaging and image-based AI/ML device reviewers in the Center for Diagnostics and Radiological Health at the FDA.