Development and validation a radiomics nomogram for diagnosing occult brain metastases in patients with stage IV lung adenocarcinoma

Introduction

Lung cancer is the most common cause of cancer-related deaths worldwide, of which 80–85% are non-small cell lung cancer (NSCLC) (1,2). Brain metastases (BMs) remain an important cause of morbidity and mortality in patients with lung cancer, especially NSCLC (3).

Lung adenocarcinoma (LADC) is the most common histologic subtype of NSCLC and is the dominant epidemic factor of BM, and it usually leads to death (4). Therefore, the prevention of BM is of great significance for patients with LADC. However, approximately 10% of NSCLC patients already have BM at first diagnosis (5). Notably, the rate of BM in advanced-stage patients with NSCLC who also have bone metastases, liver metastases, or/and adrenal metastases is higher than that in patients at earlier stages (6). Moreover, LADCs with distant bone metastases and/or liver metastases who were diagnosed with stage IV disease are more likely to develop BM than those diagnosed at earlier stages (7). In this study, “occult” refers to cases where BMs are not found using computed tomography (CT) but are found when using magnetic resonance imaging (MRI). Treatment strategies are distinctly different depending on whether occult BM is present, and hence early detection of occult BM and delivery of tailored therapies, including radiotherapy (8), surgery (9), or/and tyrosine kinase inhibitors (TKIs) (10,11), can significantly improve the prognosis of stage IV LADC.

Occult BM usually measuring less than 1 cm may be ignored during scanning CT. Guidelines for NSCLC by the National Comprehensive Cancer Network (NCCN) recommend MRI to detect single or multiple intracranial BM (12). However, although cranial MRI is considered the gold standard for diagnosing occult BM, it is not suitable for patients with heart pacemakers, claustrophobia, metal implants, or low Karnofsky Performance Status (KPS) (13-15). Advances in computational image analysis, such as radiomics, involve the use of large amounts of quantitative imaging features derived from medical images to decode tumor pathology or heterogeneity (16-18). Recently, many studies aimed to construct CT radiomic models to assess the relationship between lung cancer and BM (19,20). Zhang et al. extracted radiomics features from contrast-enhanced brain CT to differentiate the pathological subtypes of primary lung cancer (21).

On the other hand, Chen et al. built a hybrid model incorporating clinical risk factors and unenhanced CT radiomics features to predict BM in patients with T1 LADCs (22). In contrast, a radiomics model was developed by Sun et al., which can accurately assess the BM-free survival of patients with locally advanced LADC following thoracic surgery (9). However, to the best of our knowledge, no previous radiomic studies have been conducted to address the diagnosis of occult BM in stage IV LADC, even though it has potential clinical applications that can improve prognosis. Therefore, a reasonable hypothesis is that CT radiomics can be utilized to predict occult BM in LADCs as a feasible alternative to MRI scanning, thus lessening the burden on patients and the health care system.

In the present study, we aimed to develop and validate a radiomics model using CT images acquired at first diagnosis to estimate the status of occult BM in patients with stage IV LADCs. We present the following article in accordance with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting checklist (available at https://dx.doi.org/10.21037/tcr-21-702).

Methods

Study design

The overall workflow of this study is illustrated in Figure 1. The workflow consists of five steps: (I) patients and images, (II) tumor segmentation, (III) feature extraction, (IV) radiomics signature and nomogram construction, and (V) model evaluation.

Figure 1 Workflow of study.

Patient sample

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This study was approved by the Institutional Review Board of Shandong University (KYLL-2014(LW)03), and the need for informed consent was waived because the study was an observational, retrospective study. In this study, patients who were first diagnosed with stage IV LADC between January 1, 2014, and July 31, 2020, and who had distant metastases (brain, lung, bone, liver, adrenal gland, or other organs) were included in the study. The inclusion criteria were as follows: (I) clinical diagnosis with stage IV lung cancer, (II) no BM observed in pretreatment baseline cranial CT images, and (III) had cranial MRI and baseline chest CT. The exclusion criteria were as follows: (I) no pretreatment baseline chest CT images, (II) insufficient image quality, (III) no cranial MRI scans, and (IV) a history of other malignancy. The occult BM status of enrolled patients was determined using CT and MRI, which were cross-reviewed by two experienced radiologists. Finally, enrolled patients were divided into two cohorts using a cutoff date of July 1, 2018, i.e., the training cohort to develop the prediction model and the validation cohort to evaluate the performance of the established model.

Clinical factors and laboratory indexes, including age, sex, smoking status, tumor diameter, CT-reported lymph node (LN), primary tumor location, pleural effusion, and carcinoma embryonic antigen (CEA), were recorded.

CT images acquisition

CT images were scanned and collected for all enrolled patients. All patients underwent standard chest unenhanced CT scanning using a GE scanner (Light Speed VCT, GE Healthcare Systems, Chicago, IL, USA) or a GE CT scanner (Discovery 750 HD, GE Healthcare Systems, Chicago, IL, USA). The scanning protocol was as follows: X-ray tube peak voltage 120 kV, tube current ranged from 230 to 327 mA, slice thickness 5.0 mm, in-plane resolution 0.7734×0.7734 mm/pixel, and helical scanning mode.

Tumor segmentation

Tumor segmentation was implemented using 3D Slicer software (version 4.10.2, www.slicer.org), an open-source platform for medical image processing, including segmentation, registration, visualization, etc. Primary gross tumor volumes (GTV) of LADCs were manually delineated by an independent radiologist with over five years of professional experience, and then another experienced oncologist crosschecked contoured GTVs. Finally, given that contoured GTVs may not be accurate, a 3×3 Gaussian filter with a variance of σ²=0.2 was used to thicken and smoothen the contours to facilitate all delineations more robustly and reduce the risk of interobserver variance.

Radiomic feature extraction

Radiomic features were extracted using the SlicerRadiomics extension module in 3D slicer (version 4.10.2, www.slicer.org), which is an embedded graphical user interface (GUI) of pyRadiomics packages used for specific feature calculations (23) for each contoured GTV. In all, 725 features were calculated, and they fell into the following six categories: (I) 14 shape features, (II) 18 first-order intensity histogram (IH)-based features, and 61 statistical matrix (SM)-based features divided into (III) 24 gray-level co-occurrence matrix (GLCM)-based features, (IV) 16 gray-level run-length matrix (GLRLM)-based features, (V) 16 gray-level size zone matrix (GLSZM)-based features, and (VI) 5 neighboring gray-tone difference matrixes (NGTDM). Moreover, 632 wavelet-based features (including IH and SM features) were extracted from eight wavelet decompositions.

Radiomics signature and nomogram construction

A two-step feature selection was implemented before building the radiomics signature. First, intra- and interobserver reliabilities were tested with 30 randomly selected patients using an intraclass correlation coefficient (ICC), where an ICC ≥0.75 was considered as indicating good reliability. The detailed workflow for feature selection and modeling can be found in Figure S1. Then, the least absolute shrinkage and selection operator (LASSO) (with a binary regression model, a five-fold cross-validation method, an “auc” loss measurement, and using normalized data) was performed to determine the most predictive features for diagnosing occult BM and to make subsequent models or signatures more robust and effective. Moreover, predictive clinical risk factors were selected from potential candidates using univariate analyses. Univariate analyses were performed using the training cohort.

After feature selection, the Rad-Score, also known as the radiomics signature or score, was constructed from a linear combination of features and corresponding weights derived from LASSO. Furthermore, a nomogram was established by incorporating the Rad-Score and predictive clinical risk factors to test whether the predictive performance would benefit from clinical information.

Validation of radiomics signature and nomogram

The Rad-Score and nomogram were developed based on the training cohort and validated in both the training and validation cohorts. In this study, discrimination and calibration were calculated to assess the developed models. First, the Rad-Score and nomogram discrimination were evaluated using an empirical receiver operating characteristic curve (ROC) and the area under the curve (AUC). Due to the limited cohort size, empirical ROC curves have the disadvantage of having an uneven appearance and introducing non-ignorable uncertainty, especially when the dataset is small. ROC curves may perform poorly for evaluation despite a superior AUC when the positive and negative data used for building a prediction model are imbalanced. In these cases, the precision-recall curve (PRC) plots the positive prediction value (PPV) against the true positive rate (TPR) across all thresholds and represents a more accurate method to assess established models and the area under PRC defined as average precision (AP). Hence, the α-binomial model-based ROC curve and PRC proposed by Brodersen et al. to plot smooth curves were used to address the above issues (24). Second, the nomogram calibration was assessed regarding its predictive accuracy and the agreement between the actual occult BM probability and that predicted by the nomogram; using a Hosmer-Lemeshow test, a P value ≥0.05 was considered as indicating good agreement.

Statistical analysis

Statistical analyses were performed in R version 3.3.1 (The R Foundation for Statistical Computing). Comparisons and univariate analyses were implemented in R with the stats package. Mann-Whitney U-tests or two-sample t-tests were employed where appropriate. The LASSO was performed in R with the glmnet package. Nomogram and calibration curves were performed in R with the rms package. The α-binomial model-based ROC curves and PRCs were plotted in MATLAB (version 2018a, The MathWorks Inc., Natick, MA, USA) using an open-source code derived from MathWorks Central File Exchange (25). All of the reported statistical significance levels were two-sided. The statistical significance level was set to P<0.05.

Results

Patients’ characteristics

In total, 193 consecutive patients were enrolled. The training cohort and the validation cohort consisted of 135 and 58 patients, respectively. Patients’ characteristics, including clinical factors and laboratory indexes, are summarized in Table 1. The demographic characteristics of the two cohorts were similar. No significant differences were observed for any factors with P values ranging from 0.094 to 0.836. Two examples of occult BM are depicted in Figure 2.

Figure 2 Two examples of occult brain metastases (BM). (A) and (B) are the same slice in computed tomography (CT) and magnetic resonance imaging (MRI) of Example 1, respectively. (C) and (D) are the same slice in CT and MRI of Example 2, respectively. Red arrows indicated that occult brain metastases.

Radiomics signature construction and validation results

After the first step of feature selection, 549 out of 725 extracted features had ICC ≥0.75, indicating high intra- and interobserver reliabilities for the multiple segmentation test, and the results of reproducible features listed in Table S1. Subsequently, eight features with nonzero coefficients were selected from the reliable features using a LASSO binary regression model while tuning with the parameter λ. The process of the tuning phase of LASSO and the feature dimension reduction is illustrated in Figure 3.

Figure 3 Feature selection using the least absolute shrinkage and selection operator (LASSO) with a binary classification model. (A) The LASSO coefficient profile graph is shown using coefficients against the log(λ) sequence. Vertical lines are shown corresponding to 1 standard error (1-se) and maximum criteria. (B) Tuning parameter λ-based 1-se and maximum criteria. The area under the curve (AUC) relating to λ is shown. The 1-se of maximum criteria (left vertical line) is shown using the right vertical line. In this study, λ with corresponding optimal AUC was chosen (λ=−2.781). As a result, eight radiomic features with nonzero coefficients were selected.

Then the Rad-Scores were established using the following formulas:

Rad−Score=−0.0728×Original−Shape−Flatness−0.0033×W−HLL−GLCM−Correlation+0.1020×W−HLH−GLSZM−SALGLE−0.1318×W−HHH−Firstorder−Skewness+0.0487×W−HHH−Firstorder−Mean+0.0065×W−HHH−NGTDM−Busyness+0.0568×W−HHL−GLCM−MCC−0.0295×W−LLL−Firstorder−Variance[1]

The Rad-Score values calculated from the training and validation cohort are shown in Figure 4. The empirical and α-binormal-based ROC curves and PRCs of the developed Rad-Scores are presented in Figure 5.

Figure 4 Bar plot of the Rad-Score value for each patient in the training cohort (A) and the validation cohort (B). Patients with Rad-Score values ≥0 were considered as being at high risk for metastases, while patients with values <0 were deemed as being at low risk for metastases. Patients with and without BM are marked with different colors.

Figure 5 The performance of the Rad-Score developed in this study. (A) The receiver operating characteristic (ROC) curves. (B) The precision-recall curve (PRC). Subscripts emp and αbin are empirical and α-binormal-based, respectively, ROC or PRC.

Nomogram construction and validation

In univariate analyses, only primary tumor location was an independent clinical risk factor in the nomogram. The results of univariate analyses can be found in Table S2. A nomogram was constructed by combining the independent clinical risk factor and the Rad-Score; it is depicted in Figure 6. The empirical and α-binormal-based ROC curves and PRCs of developed Rad-Scores are presented in Figure 7. The calibration curve of the nomogram is plotted in Figure 8. The Hosmer-Lemeshow test (P value was 0.427) showed that there was no significant difference between predicted metastases and the observed metastases in the validation cohort.

Figure 6 Nomogram developed by incorporating clinical risk factors and Rad-Score of the training cohort.

Figure 7 The performances of the nomogram developed in this study. (A) Receiver operating characteristic (ROC) curves. (B) Precision-recall curve (PRC). Subscripts emp and αbin are empirical and α-binormal-based, respectively, ROC curves or PRCs.

Figure 8 Calibration curve of nomogram plotted to assess the agreement between nomogram-predicted metastases and observed metastases of the validation cohort.

Comparisons of developed Rad-Score and nomogram

The developed Rad-Score and nomogram performances were calculated and compared in the training cohort and the validation cohort, as shown in Table 2. The nomogram yielded an optimal AUC of 0.911 [95% confidence interval (CI), 0.903–0.919] and an AP of 0.885 (95% CI: 0.876–0.894) in the training cohort, and an AUC of 0.873 (95% CI: 0.866–0.880) and an AP of 0.827 (95% CI: 0.820–0.834) in the validation cohort using an α-binomial-based method. Notably, all of the performances of the nomogram were significantly higher than those of the Rad-Score (all P value <0.001). Additionally, no significant differences were observed between the empirical and α-binormal-based methods (all P value >0.05), indicating that prediction results of occult BM are reasonable and robust.

Discussion

Patients who already have bone metastases and/or liver metastases when they are first diagnosed are defined as having stage IV LADC (26). Although they may not yet have BM, these patients are at higher risk for BM than those at earlier stages. Notably, some stage IV LADCs may have developed occult BM, which cannot be captured using CT imaging, with the result that occult BM continues to progress and lead to a poor prognosis. Patients harboring occult BM can be detected using an MR scanner; however, several contraindications limit its availability. With the emergence of quantitative imaging analysis in precision medicine, CT radiomics has been widely used for diagnosis. In this study, although Rad-Score displayed a significant correlation with occult BM, we assumed that it would achieve a better performance if combined with other clinical predictors. Indeed our results verified this hypothesis. The proposed CT radiomics-based nomogram for the diagnosis occult BM showed remarkable AUC and AP in both the training and the validation cohort compared to radiomics alone. The results demonstrated that CT is useful for detecting occult BM of stage IV LADC in the absence of MR imaging.

To make the model developed more predictive and robust, feature selection is a crucial step of a radiomics study (27). The Rad-Score consisted of eight features, and nearly all of the selected predictive features were wavelet, similar to the results of several published studies (28-30). A reasonable hypothesis is that the multifrequency decomposition of the original CT image captured underlies information about tumor heterogeneity between BM and BM-free LADCs, which clinicians cannot assess with unassisted vision. In addition, a bar plot showed that a Rad-Score value of zero is a potential diagnostic threshold and achieved satisfactory performance. Therefore, the established Rad-Score can efficiently stratify patients of unknown BM status into high- and low-risk subgroups concerning BM.

Previous studies have suggested that the radiomics signature combined with clinical risk factors can significantly improve the performance of various clinical tasks related to lung cancer (31-34). For instance, a combined model incorporating CT-based radiomics and clinical features outperforms CT-based radiomics alone for predicting distant metastases of NSCLC (35), estimating BM-free survival of curatively resected locally advanced NSCLC (9), and predicting survival after whole brain radiotherapy of NSCLC patients with BM (8). In this study, the results of univariate analyses of clinical risk factors showed that the primary tumor location had significant correlations with occult BM status and that the upper lobe, including the upper right and left lobe, were the most frequently occurring primary metastatic sites for BM, occurring at a rate of 47%. A recent study using multivariate logistic regressions demonstrated that most BM relating to stage IV NSCLCs were derived from the upper lobes (36). Another study reported that primary tumors in the upper lobe of LADCs were more prone to lymphovascular invasion with the highest incidence of 35.73% and mediastinal lymph node metastases (37). Our results are consistent with the studies mentioned above. Hence, the nomogram developed incorporated the primary tumor location and Rad-Score are achieved optimal performance.

On the other hand, to evaluate the developed models using a robust and unbiased approach, α-binomial model-based ROC curves and PRCs were employed and compared to empirical curves. Our results demonstrated that no significant differences were observed between α-binomial model-based and empirical ROC curves and PRCs. Performance measures based on PRCs are helpful to supplement and validate ROC curves. Furthermore, based on a distributional assumption about the underlying decisional values and an explicit estimation of the class-mixture parameter α-binomial model-based curves, the practical limitations raised by empirical approaches to estimate curves can be addressed effectively (24). Therefore, the nomogram is robust and accurate for predicting occult BM.

Our study has several limitations. First, contrast-enhanced CT images were unavailable for many enrolled patients due to the retrospective design of this study. We believe that the results will benefit greatly from contrast information. Second, survival analysis of patients after target treatment was not conducted in this study due to the limited cohort size and follow-up, including BM-free survival, progress-free survival, and overall survival. We will be conducting a future study, which will focus on predicting lung cancer outcomes, including BM occurrence.

Conclusions

We developed and validated a radiomics-based nomogram model that incorporates a radiomics signature derived from unenhanced CT imaging and clinical risk factors to diagnose occult BM in patients who received a diagnosis of stage IV LADC as their first diagnosis. The nomogram may serve as a powerful and robust tool for diagnosing occult BM.

Acknowledgments

Funding: This study was supported by the National Natural Science Foundation of China (grant nos. 82072094 and 82001902), the Natural Science Foundation of Shandong Province (grant nos. ZR2019LZL017 and ZR2020QH198), and the Taishan Scholars Project of Shandong Province (grant no. ts201712098).

Footnote

Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://dx.doi.org/10.21037/tcr-21-702

Data Sharing Statement: Available at https://dx.doi.org/10.21037/tcr-21-702

Peer Review File: Available at https://dx.doi.org/10.21037/tcr-21-702

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/tcr-21-702). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This study was approved by the Institutional Review Board (IRB) of Shandong University (ID: KYLL-2014(LW)03), and the need for informed consent was waived because the study was an observational, retrospective study.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Bray F, Ferlay J, Soerjomataram I, et al. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018;68:394-424. [Crossref] [PubMed]
2. Ferlay J, Colombet M, Soerjomataram I, et al. Estimating the global cancer incidence and mortality in 2018: GLOBOCAN sources and methods. Int J Cancer 2019;144:1941-53. [Crossref] [PubMed]
3. Villano JL, Durbin EB, Normandeau C, et al. Incidence of brain metastasis at initial presentation of lung cancer. Neuro Oncol 2015;17:122-8. [Crossref] [PubMed]
4. Shih DJH, Nayyar N, Bihun I, et al. Genomic characterization of human brain metastases identifies drivers of metastatic lung adenocarcinoma. Nat Genet 2020;52:371-7. [Crossref] [PubMed]
5. Huang Z, Hu C, Tong Y, et al. Construction of a nomogram to predict the prognosis of non-small-cell lung cancer with brain metastases. Medicine (Baltimore) 2020;99:e21339. [Crossref] [PubMed]
6. Goncalves PH, Peterson SL, Vigneau FD, et al. Risk of brain metastases in patients with nonmetastatic lung cancer: Analysis of the Metropolitan Detroit Surveillance, Epidemiology, and End Results (SEER) data. Cancer 2016;122:1921-7. [Crossref] [PubMed]
7. Huang X, Wang J, Lin W, et al. Kanglaite injection plus platinum-based chemotherapy for stage III/IV non-small cell lung cancer: A meta-analysis of 27 RCTs. Phytomedicine 2020;67:153154. [Crossref] [PubMed]
8. Zhang J, Jin J, Ai Y, et al. Computer Tomography Radiomics-Based Nomogram in the Survival Prediction for Brain Metastases From Non-Small Cell Lung Cancer Underwent Whole Brain Radiotherapy. Front Oncol 2021;10:610691. [Crossref] [PubMed]
9. Sun F, Chen Y, Chen X, et al. CT-based radiomics for predicting brain metastases as the first failure in patients with curatively resected locally advanced non-small cell lung cancer. Eur J Radiol 2021;134:109411. [Crossref] [PubMed]
10. Yang Z, Zhang Y, Li R, et al. Whole-brain radiotherapy with and without concurrent erlotinib in NSCLC with brain metastases: a multicenter, open-label, randomized, controlled phase III trial. Neuro Oncol 2021;23:967-78. [Crossref] [PubMed]
11. Xing L, Pan Y, Shi Y, et al. Biomarkers of Osimertinib Response in Patients with Refractory, EGFR-T790M-positive Non-Small Cell Lung Cancer and Central Nervous System Metastases: The APOLLO Study. Clin Cancer Res 2020;26:6168-75. [Crossref] [PubMed]
12. Ettinger DS, Wood DE, Aggarwal C, et al. NCCN Guidelines Insights: Non-Small Cell Lung Cancer, Version 1.2020. J Natl Compr Canc Netw 2019;17:1464-72. [Crossref] [PubMed]
13. Tsuchiya K. Editorial Comment: New Insights Into the MRI Diagnosis of Brain Metastasis From Lung Cancer. AJR Am J Roentgenol 2021; Epub ahead of print. [Crossref] [PubMed]
14. Smits M. Update on neuroimaging in brain tumours. Curr Opin Neurol 2021;34:497-504. [PubMed]
15. Schoenmaekers J, Hofman P, Bootsma G, et al. Screening for brain metastases in patients with stage III non-small-cell lung cancer, magnetic resonance imaging or computed tomography? A prospective study. Eur J Cancer 2019;115:88-96. [Crossref] [PubMed]
16. Qiu Q, Duan J, Yin Y. Radiomics in radiotherapy: Applications and future challenges. Precis Radiat Oncol 2020;4:29-33. [Crossref]
17. Zhou M, Scott J, Chaudhury B, et al. Radiomics in Brain Tumor: Image Assessment, Quantitative Feature Descriptors, and Machine-Learning Approaches. AJNR Am J Neuroradiol 2018;39:208-16. [Crossref] [PubMed]
18. Wang G, Wang B, Wang Z, et al. Radiomics signature of brain metastasis: prediction of EGFR mutation status. Eur Radiol 2021;31:4538-47. [Crossref] [PubMed]
19. Xu X, Huang L, Chen J, et al. Application of radiomics signature captured from pretreatment thoracic CT to predict brain metastases in stage III/IV ALK-positive non-small cell lung cancer patients. J Thorac Dis 2019;11:4516-28. [Crossref] [PubMed]
20. Ortiz-Ramón R, Larroza A, Ruiz-España S, et al. Classifying brain metastases by their primary site of origin using a radiomics approach based on texture analysis: a feasibility study. Eur Radiol 2018;28:4514-23. [Crossref] [PubMed]
21. Zhang J, Jin J, Ai Y, et al. Differentiating the pathological subtypes of primary lung cancer for patients with brain metastases based on radiomics features from brain CT images. Eur Radiol 2021;31:1022-8. [Crossref] [PubMed]
22. Chen A, Lu L, Pu X, et al. CT-Based Radiomics Model for Predicting Brain Metastasis in Category T1 Lung Adenocarcinoma. AJR Am J Roentgenol 2019;213:134-9. [Crossref] [PubMed]
23. van Griethuysen JJM, Fedorov A, Parmar C, et al. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res 2017;77:e104-7. [Crossref] [PubMed]
24. Brodersen KH, Cheng SO, Stephan KE, et al. The Binormal Assumption on Precision-Recall Curves. International Conference on Pattern Recognition; 2010. Available online: https://ieeexplore.ieee.org/document/5597760
25. Brodersen KH. Estimating a smooth precision-recall curve. Available online: https://wwwmathworkscom/matlabcentral/fileexchange/29250-estimating-a-smooth-precision-recall-curve
26. Wang L, Shi T, Feng L, et al. The Prognostic Value of Baseline Distant Metastasis in Icotinib-Treated Patients with EGFR-Mutated Stage IV Non-Small Cell Lung Cancer. Cancer Manag Res 2021;13:2613-22. [Crossref] [PubMed]
27. Ji Y, Qiu Q, Fu J, et al. Stage-Specific PET Radiomic Prediction Model for the Histological Subtype Classification of Non-Small-Cell Lung Cancer. Cancer Manag Res 2021;13:307-17. [Crossref] [PubMed]
28. Liang W, Yang P, Huang R, et al. A Combined Nomogram Model to Preoperatively Predict Histologic Grade in Pancreatic Neuroendocrine Tumors. Clin Cancer Res 2019;25:584-94. [Crossref] [PubMed]
29. Wu S, Zheng J, Li Y, et al. Development and Validation of an MRI-Based Radiomics Signature for the Preoperative Prediction of Lymph Node Metastasis in Bladder Cancer. EBioMedicine 2018;34:76-84. [Crossref] [PubMed]
30. Qiu Q, Duan J, Deng H, et al. Development and Validation of a Radiomics Nomogram Model for Predicting Postoperative Recurrence in Patients With Esophageal Squamous Cell Cancer Who Achieved pCR After Neoadjuvant Chemoradiotherapy Followed by Surgery. Front Oncol 2020;10:1398. [Crossref] [PubMed]
31. Wu L, Yang X, Cao W, et al. Multiple Level CT Radiomics Features Preoperatively Predict Lymph Node Metastasis in Esophageal Cancer: A Multicentre Retrospective Study. Front Oncol 2020;9:1548. [Crossref] [PubMed]
32. Zhang G, Cao Y, Zhang J, et al. Predicting EGFR mutation status in lung adenocarcinoma: development and validation of a computed tomography-based radiomics signature. Am J Cancer Res 2021;11:546-60. [PubMed]
33. Cui EN, Yu T, Shang SJ, et al. Radiomics model for distinguishing tuberculosis and lung cancer on computed tomography scans. World J Clin Cases 2020;8:5203-12. [Crossref] [PubMed]
34. Liu S, Liu S, Zhang C, et al. Exploratory Study of a CT Radiomics Model for the Classification of Small Cell Lung Cancer and Non-small-Cell Lung Cancer. Front Oncol 2020;10:1268. [Crossref] [PubMed]
35. Tao J, Lv R, Liang C, et al. Development and Validation of a CT-Based Signature for the Prediction of Distant Metastasis Before Treatment of Non-Small Cell Lung Cancer. Acad Radiol 2021; Epub ahead of print. [Crossref] [PubMed]
36. Shan Q, Li Z, Lin J, et al. Tumor Primary Location May Affect Metastasis Pattern for Patients with Stage IV NSCLC: A Population-Based Study. J Oncol 2020;2020:4784701. [Crossref] [PubMed]
37. Grbić K, Mehić B. Characteristics of lymphovascular metastatic spread in lung adenocarcinoma according to the primary cancer location. Med Glas (Zenica) 2020;17:66-72. [PubMed]

(English Language Editors: B. Meiser and J. Chapnick)