Prediction Variability to Identify Reduced AI Performance in Cancer Diagnosis at MRI and CT
Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › fagfællebedømt
A priori identification of patients at risk of artificial intelligence (AI) failure in diagnosing cancer would contribute to the safer clinical integration of diagnostic algorithms.
Purpose
To evaluate AI prediction variability as an uncertainty quantification (UQ) metric for identifying cases at risk of AI failure in diagnosing cancer at MRI and CT across different cancer types, data sets, and algorithms.
Materials and Methods
Multicenter data sets and publicly available AI algorithms from three previous studies that evaluated detection of pancreatic cancer on contrast-enhanced CT images, detection of prostate cancer on MRI scans, and prediction of pulmonary nodule malignancy on low-dose CT images were analyzed retrospectively. Each task’s algorithm was extended to generate an uncertainty score based on ensemble prediction variability. AI accuracy percentage and partial area under the receiver operating characteristic curve (pAUC) were compared between certain and uncertain patient groups in a range of percentile thresholds (10%–90%) for the uncertainty score using permutation tests for statistical significance. The pulmonary nodule malignancy prediction algorithm was compared with 11 clinical readers for the certain group (CG) and uncertain group (UG).
Results
In total, 18 022 images were used for training and 838 images were used for testing. AI diagnostic accuracy was higher for the cases in the CG across all tasks (P < .001). At an 80% threshold of certain predictions, accuracy in the CG was 21%–29% higher than in the UG and 4%–6% higher than in the overall test data sets. The lesion-level pAUC in the CG was 0.25–0.39 higher than in the UG and 0.05–0.08 higher than in the overall test data sets (P < .001). For pulmonary nodule malignancy prediction, accuracy of AI was on par with clinicians for cases in the CG (AI results vs clinician results, 80% [95% CI: 76, 85] vs 78% [95% CI: 70, 87]; P = .07) but worse for cases in the UG (AI results vs clinician results, 50% [95% CI: 37, 64] vs 68% [95% CI: 60, 76]; P < .001).
Conclusion
An AI-prediction UQ metric consistently identified reduced performance of AI in cancer diagnosis.
| Originalsprog | Engelsk |
|---|---|
| Artikelnummer | e230275 |
| Tidsskrift | Radiology |
| Vol/bind | 308 |
| Udgave nummer | 3 |
| Antal sider | 11 |
| ISSN | 0033-8419 |
| DOI | |
| Status | Udgivet - 2023 |
Bibliografisk note
Funding Information:
Author contributions: Guarantors of integrity of entire study, N.A., H.H.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; approval of final version of submitted manuscript, all authors; agrees to ensure any questions related to the work are appropriately resolved, all authors; literature research, N.A., J.H., H.H.; clinical studies, Z.S., J.H.; experimental studies, N.A., J.B., C.J., J.H.; statistical analysis, N.A., J.H., H.H.; and manuscript editing, all authors Disclosures of conflicts of interest: N.A. No relevant relationships. J.S.B. Health Holland grant. K.V.V. Employee, Predible; stock options from Predible Health. C.J. Research grants from MeVis Medical Solutions; royalties from MeVis. Z.S. No relevant relationships. M.d.R. No relevant relationships. J.H. No relevant relationships. H.H. Partial grant support from Siemens Healthineers; patents related to processing and displaying dynamic contrast-enhanced MRI information.
Funding Information:
Supported by the European Union’s Horizon 2020 research and innovation program (grant number 101016851, project PANCAIM). Conflicts of interest are listed at the end of this article. See also the editorial by Babyn in this issue.
Publisher Copyright:
© RSNA, 2023.
ID: 386562944