Computational resources for radiomics

Introduction

Radiomics, the use of quantitative image features to probe the tumor phenotype, can significantly improve our ability to stratify patients for true personalized cancer care. Radiomics is the logical next step for a research field that has spent years developing tools for medical image analysis, including computer-aided diagnosis for detecting breast cancer on mammograms and lung nodules on radiographs and computed tomography (CT) studies (1-5). We can combine the body of knowledge from these years of analyzing medical images with the increasing access to large amounts of imaging data and new, sophisticated analytic tools for quantitative analysis of biomarkers with the goal of refining clinical decision making and improving patient outcomes.

The advanced radiomics tools now available can help answer important clinical questions and provide necessary information for patient-specific personalized treatments (i.e., precision medicine). For example, conventional prognostic factors, such as tumor volume or patient age, are simply insufficient to help the clinician stratify patient risk or predict outcome.

The purpose of this review article is to introduce the reader to the armory of software available for segmenting imaging data, extracting image features, and carrying out modeling/statistical analysis for such radiomics projects.

Radiomics workflow

Radiomics project workflows comprise the following stages:

• Identify a question and patient cohort;
• Segment the regions of interest (ROI) in the patient images;
• Extract the image features;
• Statistical analysis/modeling;

This article briefly reviews the main issues involved in identifying a patient cohort and focuses on the software resources available for stages 2–4.

Identifying a patient cohort

Even in this era of big data, good patient datasets are surprisingly difficult to build. Although not the focus of this review, identifying a patient cohort is a critical part of any radiomics project. Below we have outlined strategies for building good datasets.

Patient cohort homogeneity

Some heterogeneity in the patient cohort may be necessary to achieve sufficient patient numbers, but too much heterogeneity not only can dilute the potential impact of the findings but also can introduce too much variability into the dataset. An example of diluting the impact could be including patients whose overall staging varies widely because staging may already be a prognostic factor. An example of too much variability in the dataset could be including patients whose treatments vary significantly (i.e., widely divergent regimens comprising different surgery, chemotherapy, and radiotherapy approaches).

Region of interest size

Many radiomics image features do not make sense when tumors are too small. There are no consistent guidelines on the smallest ROI that can be assessed, although some authors have suggested 5 cm³ as a suitable cutoff. The cutoff value depends on the imaging modality [e.g., CT and positron emission tomography (PET) have different voxel sizes and may require different cutoffs] and may also vary depending on the site or tumor under investigation. Smaller ROIs can either give meaningless radiomics feature values because there are not enough pixels for a true evaluation or the smaller the ROI, the more related the results may be to tumor volume. This issue is discussed in more detail by Fave et al. later in this journal.

Imaging data homogeneity

Some datasets have significant heterogeneity in imaging parameters, such as pixel size (6). These variations could significantly affect the values of the calculated image features. The use of different reconstruction algorithms can change the values of the calculated features. Furthermore, data from different scanners can increase the uncertainties in the calculated features (7). Thus it is important to minimize the variability in the source of the images, although compromises are often necessary to ensure sufficient patient numbers. Important information about the images (such as pixel size and tube voltage) is listed in the digital imaging and communications in medicine (DICOM) header and can easily be extracted and viewed in any DICOM image viewer or in simple in-house software, e.g., MATLAB-based software.

Sample size

Small sample sizes increase both the type-I (incorrectly detecting a difference) and type-II (not detecting an actual difference) error rates. Chalkidou et al. suggested that linear models (e.g., multiple regression) require a minimum of 10–15 observations per variable (8). Radiomics studies have been published with as few as 15 patients, but there is much risk of over fitting the data, and researchers should generally aim for much larger datasets.

Sometimes it is possible for authors to supplement their own datasets with images from The Cancer Imaging Archive (9), an open-access database that facilitates sharing image data. At the time of writing, The Cancer Imaging Archive had 63 image sets, covering a range of sites (lung, prostate, thyroid, etc.). Access to a few of the image sets is limited; the median number of subjects for the datasets that do not have restricted access is 46 subjects (range, 1–1,010 subjects). The archive includes test-retest data, such as the RIDER dataset (10) used by several groups to test the reproducibility of different image features (11-14), as well as several datasets that have been collected specifically for radiomics studies (12,15). These image sets are an excellent source of data for testing algorithms or for validating models.

Segmentation

After collecting a dataset, the next step in the radiomics workflow is the segmentation of the ROI. Important considerations in the choice of software and technique include uncertainties in the contours and efficiency of workflow. Manually segmented ROIs can have high inter-user variability, especially for some modalities (16,17), which may affect the radiomics image features. Many successful radiomics studies use manually-delineated contours (18), but inter-user variability should be minimized. Inter-user variability can be reduced by the use of semi- or fully-automated segmentation tools (19), although the user is cautioned that these tools can fail, and we strongly recommend that the results always be visually checked. Importantly, these automated tools can significantly affect the time it takes to segment the ROI—a key consideration when data from hundreds of patients will be used. Another way to reduce the impact of inter-user variability is to use algorithms such as STAPLE [Simultaneous truth and performance level estimation (20)] to create consensus contours from segmentations generated by experts, or from different auto-segmentation algorithms.

Users also must decide whether the entire tumor (or other structure) or only portions of the ROI will be segmented. Some researchers have segmented the axial slice where the tumor is largest (21). Segmenting a single slice or fixed-size ROI significantly improves efficiency when manual segmentation is used. However, the extracted ROI may not represent the entire tumor. The effect of segmenting a single slice or fixed-size ROI on the extracted radiomics image features varies widely, depending on the image feature, but can be significant (22). Other approaches that can improve workflow and reduce inter-user variability are using a fixed-size ROI [e.g., Bang et al. used a spherical ROI (23)] and including the maximal circle within manually segmented liver lesions (24). Researchers may also have to decide which phase of a four-dimensional CT to segment (22).

In this section we describe several software programs used for target segmentation in radiomics studies. The main points to consider when choosing software for delineation are as follows:

• Import options. Can the software import and work with your modality?
• Export options. What formats can the software use to export your delineated structures? Transferring segmentation results from the segmentation software to the feature extraction software can be challenging;
• Ease of use. How user-friendly is the software? This includes considerations such as time to open patient data files;
• Delineation tools. What manual or semiautomatic tools does the software provide for delineation?
• Auto-contouring tools. What auto-contouring tools does the software provide? Do these tools work for your target (e.g., lung tumor vs. liver tumor)?

Treatment planning systems

Many radiomics research projects use ROIs delineated by radiation oncologists for treatment planning systems. Because accurate delineation of ROIs is crucial in radiation therapy, the tools for manual delineation used in radiation therapy treatment planning systems often are more sophisticated than software focused on radiology applications. Radiation oncologists are very experienced in using these systems.

Before using these treatment-planning contours for radiomics research, however, it is important to consider whether this is the appropriate choice. For example, because the original contours are used to determine which tissue/tumor volumes will be treated with radiation, the physician may have included tissue with a reasonable likelihood of representing tumor tissue, and the contoured volume may be larger than the actual tumor. Including healthy tissue in the radiomics ROI could significantly affect the calculated image features. One approach to address this potential problem is to use the original gross tumor volume and manually modify contours to include only regions that have a high likelihood of representing tumor tissue. Thresholding can also be used to exclude healthy tissue, such as lung or bone (25). However, the choice of thresholds can affect the value of image features and reproducibility, so some experimentation may be needed (26).

Another issue is uncertainty in the delineated ROI, i.e., variation in the region contoured by different physicians. Some radiomics researchers address this issue by carrying out multi-user delineation studies and by ranking image features according to their sensitivity to uncertainties in delineation (12).

3DSlicer

3DSlicer (www.slicer.org) is a free, open-source software package for image analysis (27) that includes tools for image registration and segmentation and works with many medical imaging modalities, including CT, MRI, and PET. The 3DSlicer webpage has a series of tutorials and datasets for learning how to use the software. Many extensions are available, including one for tumor segmentation on PET images. 3DSlicer has been used in various radiomics projects, including the semiautomatic delineation of lung tumors on CT and PET (13,19). Some researchers use tools already available in 3DSlicer, whereas others add their own tools to the software platform. Parmar et al., for example, implemented a semiautomatic region-growing segmentation algorithm in the 3DSlicer platform and showed that this approach was much more reproducible than manually drawn regions (19).

MIM software

MIM (www.mimsoftware.com) is a commercial software package with many useful tools for radiomics applications, such as manual contouring, auto-contouring, and image registration. The software works with CT, MR, and PET images. On phantom data, the MIM algorithm for semiautomatic contouring of tumors on PET images (PET Edge) was shown to be the most accurate and consistent technique for delineating lung cancer lesions (28), and the MIM algorithm has been used in radiomics studies for lung and esophageal cancers (29,30).

Other software

Because we do not have the space to list all software programs, we have focused on the most commonly used software for segmentation. in radiomics research; however, many other software programs may be useful for radiomics studies, depending on the specific study. Examples include the itk-SNAP (www.itksnap.org), Definiens Lung Tumor Analysis (LuTA) tool (www.definiens.com) (14,31) and the Velocity ROI delineation tool (www.velocitymedical.com) (32). Also, as described below, most image feature extraction software programs include some manual or semiautomatic segmentation capabilities.

Image feature extraction

Several commercial and open-source software programs are available for extracting image features. Important points to consider when choosing feature calculation software are as follows:

• Transparency. Is it clear how the image features are being calculated? Transparency is important if you want to mimic features used in the literature. It is also important to know how the images are being processed—for example, several authors have shown the importance of knowing how many bit levels are used for the calculations (33,34);
• Features classes. There are many possible image features and image feature classes, including shape-based features, histogram-based features, and wavelet-based features. The supporting data of Aerts et al. (12) and the references in Zhang et al. (35) provide in-depth descriptions of the various features found in the literature. Most feature extraction software includes many of these image features, but the selection is limitedin some software programs. Also, users should consider whether they want the capability to add their own image features;
• Import features. Can the software import your image format and ROIs?
• Segmentation tools. In some cases, it may be possible to use segmentation tools built into the image feature extraction software. If this is the case, it may also be necessary to have the ability to export the ROIs (or contours) for comparison with other software;
• Image data summary availability. Some software programs will list important imaging parameters (e.g., pixel size) for each image. This feature can be useful when filtering data (given the effect of some parameters on the calculated image features);
• Batch processing. Sometimes it can be useful to calculate features for many patients as a batch.

These software programs typically cannot be used interchangeably, as a feature with a specific name is probably unlikely to produce the same results when calculated using different software, not only because the names are not always standardized but also because of differences in the implementation of the feature calculation (e.g., bit depth) as well as in the details of how the ROI edges are interpreted. Even when using the same software, researchers should ensure that the algorithm settings are identical.

TexRAD

Much of the early radiomics research used TexRAD (www.texrad.com), a commercial image feature software program developed by Brighton and Sussex Medical School and the University of Sussex (Brighton, UK). TexRAD is widely used (mostly with CT images, but also PET, MR and mammography) (11,21,36-52). TexRAD feature analysis uses Laplacian of Gaussian filtering, which allows calculation of various features corresponding to different scales and intensity variation. The creators of TexRAD designed it to fit into the clinical workflow, including Picture archiving and communication system (PACS) connectivity and several segmentation tools (semi-automated, automated manual). TexRAD also includes a data-miner to facilitate visualization and exploration of the data (including statistical analysis).

MaZda

MaZda (http://eletel.eu/mazda) (53,54) is a two-dimensional and three-dimensional image texture analysis software program that is widely used for radiomics and other image analysis tasks. MaZda was first developed in the 1990s for texture analysis of mammograms and has been extended for use with three-dimensional images; tools for ROI definition, normalization, statistical analysis of features, and classification have been added. The MaZda website provides good documentation and tutorials. MaZda software is widely used (the original article has been cited 148 times) and is a well-tested tool. Over the past several decades, applications of this software include detection of osteoporotic changes in bones (55), assessment of cellular necrosis in optical microscopic images (56), diagnosis of acute ischemic stroke in CT images (57), as well as the evaluation of the quality of cold meats (58). Radiomics applications of this software include the use of pretreatment PET texture to predict treatment response of locally advanced rectal cancer (23).

MaZda includes many tools that may be useful, including simple contouring tools and some tools for feature reduction and analysis (supervised/unsupervised clustering, etc.). As with other feature extraction software, the ROI can be delineated in another software program and imported into MaZda for analysis. Then, after calculating the image features in MaZda, they can be exported to a statistical analysis software program, such as R or SPSS.

Chang-Gung Image Texture Analysis (CGITA)

CGITA (http://code.google.com/p/cgita) (59), an open-source texture analysis software program based on MATLAB, was developed at the Chang-Gung Memorial Hospital (Taoyuan City, Taiwan) for the analysis of molecular images. Although focused on molecular images, CGITA can import any DICOM image and has been tested with CT and MR images. Several groups have used CGITA for feature extraction in PET studies (60,61).

ROIs either can be delineated in other software programs and imported into CGITA using several formats (including DICOM-radiotherapy) or can be delineated using CGITA’s semiautomatic segmentation function. A different software program must be used for statistical analysis.

Executable and source versions are available free of charge for academic research. The executable version is useful for researchers who do not have a MATLAB license and are not interested in implementing their own features. The source code is useful for researchers who wish to implement new features.

IBEX

IBEX (Stand-aloneIBEX: http://bit.ly/IBEX_MDAnderson; Source-code version: http://bit.ly/IBEXSrc_MDAnderson), an open-source platform for radiomics image feature extraction, is based on MATLAB and C/C++ programming languages and includes tools for importing various image and contour formats, some contouring tools, and image feature extraction tools (35). Developed at The University of Texas MD Anderson Cancer Center (Houston, TX, USA), IBEX has been used in several MD Anderson radiomics studies, including CT and PET studies (7,25,29,30,62). As with other software, such as MaZda, IBEX allows the user to vary the preprocessing and feature algorithm parameters (e.g., bit depth). This flexibility allows the user to optimize the settings for each modality and site. These settings can have a dramatic effect on the image feature values, and failure to adjust these settings can result in feature calculations that are, at best, not optimized and, at worst, meaningless (9). Importantly, IBEX also displays the processed patient images, which can be very useful for determining whether the processing makes sense (e.g., to ensure that the images have not been smoothed excessively). Also, IBEX allows users to export the actual co-occurrence matrix or intensity histogram—a useful capability when investigating different preprocessing approaches. Other features of IBEX include the ability to export entire feature sets, so users at other institutions can easily apply the same features, and the ability to anonymize patient data.

Computational environment for radiotherapy research (CERR)

CERR (www.cerr.info) is an open-source MATLAB-based software platform for importing, displaying, and analyzing radiation therapy treatment plans (63). CERR includes functions to import various imaging modalities, image fusion, and contouring. CERR is widely used in radiotherapy research (cited almost 300 times), and work is underway to add a radiomics toolbox to the software platform (64,65). Other researchers have used the CERR platform for importing and managing their medical images and have added their own in-house image feature toolbox (18).

In-house software

Many groups have written in-house software to extract features using MATLAB (14,66). In many cases, this in-house software builds on other software programs, such as CERR (18,67). Of particular note is the Insight Segmentation and Registration Toolkit, (ITK, itk.org). ITK is an open-source set of software tools for image analysis, including image pre-processing, segmentation, registration and texture calculation, which can be incorporated into researchers’ own software.

Several groups are currently building in-house software to create radiomics platforms that can be used at other institutions.

Modeling/statistical analysis

Radiomics image features are used for many different analyses. Perhaps the most common is to incorporate radiomics image features into models to improve patient risk stratification (overall survival, freedom from metastasis, etc.). In this case, the question is whether image features add value to clinical data. Other questions include whether image features are linked to tumor histology (40), tumor grade (38), or gene signatures (12,68-70).

Model development is an important component of the radiomics process and has many potential pitfalls. In spite of the increasing number of publications with positive results, Chalkidou et al. reviewed 15 radiomics studies published between 2000 and 2013 and estimated that the average type-I error probability was 76% (range, 34–99%), and most studies’ results did not reach statistical significance (8). Inexperienced users cannot simply copy a published approach; modeling and statistical analysis must be carefully considered, preferably with the involvement of a colleague experienced in this type of analysis. Chalkidou et al. provide the following best practices to develop statistically and clinically significant models while reducing false discoveries: (I) assess feature reproducibility; (II) perform cross-correlation analysis; (III) include clinically important variables (volume should be included); (IV) ensure datasets have adequate observation rates (more than 10–15 per feature); (V) include an external validation cohort (using the same feature calculations and cutoff values). Finally, care should be taken when using image features for which there is no physical (or biological) interpretation.

Below is a partial list of software packages available for modeling/statistical analysis.

• R (www.r-project.org/). R is a free software environment for statistical computing and includes many statistical techniques that are used in radiomics research, such as linear and nonlinear modeling, classification, and clustering. R has been used extensively by the radiomics research community (29,36). Parmar et al. investigated the many machine-learning algorithms available in R to determine which methods are optimal for radiomics applications (67);
• SPSS (www.ibm.com/software/analytics/spss). SPSS is a widely used commercial software program for statistical analysis and has been used in many radiomics publications (4,21,23,24);
• Stata (www.stata.com). Stata is another commercial statistics package that has been used for radiomics research (14);
• MedCalc (www.medcalc.org). MedCalc has been used by Desseroit et al. (13) for their radiomics research and by others for many image analysis-related projects (71,72);
• Bioconductor (www.bioconductor.org). Bioconductor is an open-source software for computational biology and bioinformatics (73) and has been used by radiomics researchers such as Coroller et al. (18);
• Weka (www.cs.waikato.ac.nz/ml/weka/). Weka is a free software program for data visualization, analysis, and predictive modeling that was developed at the University of Waikato (Hamilton, New Zealand) in 1993. Weka has been used for several radiomics projects (31,74) and other closely related projects using image features (75). For example, Hawkins et al. used Weka classifiers and CT-based features to predict overall survival duration for patients with lung cancer (74).

Several of the radiomics feature extraction software programs described earlier (e.g., MaZda and TexRAD) also include some statistical analysis/modeling features.

Summary

Many software programs are available for use in radiomics research. This review describes the most popular choices for segmentation, feature extraction, and statistical modeling or analysis and includes examples from the literature. Researchers should carefully weigh the benefits and drawbacks of each software program before selecting one for their analysis. To increase feature reproducibility between studies, furthermore, it is important for researchers to detail any specific segmentation, image processing, or feature parameters used during their study.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Translational Cancer Research for the series “Radiomics in Radiation Oncology”. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/tcr.2016.06.17). The series “Radiomics in Radiation Oncology” was commissioned by the editorial office without any funding or sponsorship. Court LE served as the unpaid Guest Editor of the series. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Doi K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph 2007;31:198-211. [Crossref] [PubMed]
2. Rogers W, Ryack B, Moeller G. Computer-aided medical diagnosis: literature review. Int J Biomed Comput 1979;10:267-89. [Crossref] [PubMed]
3. van Ginneken B, ter Haar Romeny BM, Viergever MA. Computer-aided diagnosis in chest radiography: a survey. IEEE Trans Med Imaging 2001;20:1228-41. [Crossref] [PubMed]
4. Doi K. Current status and future potential of computer-aided diagnosis in medical imaging. Br J Radiol 2005;78:S3-S19. [Crossref] [PubMed]
5. Suzuki K. A review of computer-aided diagnosis in thoracic and colonic imaging. Quant Imaging Med Surg 2012;2:163-76. [PubMed]
6. Kumar V, Gu Y, Basu S, et al. Radiomics: the process and the challenges. Magn Reson Imaging 2012;30:1234-48. [Crossref] [PubMed]
7. Mackin D, Fave X, Zhang L, et al. Measuring Computed Tomography Scanner Variability of Radiomics Features. Invest Radiol 2015;50:757-65. [Crossref] [PubMed]
8. Chalkidou A, O'Doherty MJ, Marsden PK. False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review. PLoS One 2015;10:e0124165 [Crossref] [PubMed]
9. Kirby J, Tarbox L, Freymann J, et al. TU-AB-BRA-03: The Cancer Imaging Archive: Supporting Radiomic and Imaging Genomic Research with Open-Access Data Sets. Med Phys 2015;42:3587. [Crossref]
10. Zhao B, James LP, Moskowitz CS, et al. Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. Radiology 2009;252:263-72. [Crossref] [PubMed]
11. Ganeshan B, Strukowska O, Skogen K, et al. Heterogeneity of focal breast lesions and surrounding tissue assessed by mammographic texture analysis: preliminary evidence of an association with tumor invasion and estrogen receptor status. Front Oncol 2011;1:33. [Crossref] [PubMed]
12. Aerts HJ, Velazquez ER, Leijenaar RT, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun 2014;5:4006. [PubMed]
13. Desseroit MC, Visvikis D, Tixier F, et al. Development of a nomogram combining clinical staging with (18)F-FDG PET/CT image features in non-small-cell lung cancer stage I-III. Eur J Nucl Med Mol Imaging 2016;43:1477-85. [Crossref] [PubMed]
14. Grove O, Berglund AE, Schabath MB, et al. Quantitative computed tomographic descriptors associate tumor shape complexity and intratumor heterogeneity with prognosis in lung adenocarcinoma. PLoS One 2015;10:e0118261 [Crossref] [PubMed]
15. Gevaert O, Xu J, Hoang CD, et al. Non-small cell lung cancer: identifying prognostic imaging biomarkers by leveraging public gene expression microarray data--methods and preliminary results. Radiology 2012;264:387-96. [Crossref] [PubMed]
16. White EA, Brock KK, Jaffray DA, et al. Inter-observer variability of prostate delineation on cone beam computerised tomography images. Clin Oncol (R Coll Radiol) 2009;21:32-8. [Crossref] [PubMed]
17. Persson GF, Nygaard DE, Hollensen C, et al. Interobserver delineation variation in lung tumour stereotactic body radiotherapy. Br J Radiol 2012;85:e654-60. [Crossref] [PubMed]
18. Coroller TP, Grossmann P, Hou Y, et al. CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma. Radiother Oncol 2015;114:345-50. [Crossref] [PubMed]
19. Parmar C, Rios Velazquez E, Leijenaar R, et al. Robust Radiomics feature quantification using semiautomatic volumetric segmentation. PLoS One 2014;9:e102107 [Crossref] [PubMed]
20. Warfield SK, Zou KH, Wells WM. Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 2004;23:903-21. [Crossref] [PubMed]
21. Ganeshan B, Panayiotou E, Burnand K, et al. Tumour heterogeneity in non-small cell lung carcinoma assessed by CT texture analysis: a potential marker of survival. Eur Radiol 2012;22:796-802. [Crossref] [PubMed]
22. Fave X, Cook M, Frederick A, et al. Preliminary investigation into sources of uncertainty in quantitative imaging features. Comput Med Imaging Graph 2015;44:54-61. [Crossref] [PubMed]
23. Bang JI, Ha S, Kang SB, et al. Prediction of neoadjuvant radiation chemotherapy response and survival using pretreatment [(18)F]FDG PET/CT scans in locally advanced rectal cancer. Eur J Nucl Med Mol Imaging 2016;43:422-31. [Crossref] [PubMed]
24. Echegaray S, Gevaert O, Shah R, et al. Core samples for radiomics features that are insensitive to tumor segmentation: method and pilot study using CT images of hepatocellular carcinoma. J Med Imaging (Bellingham) 2015;2:041011 [Crossref] [PubMed]
25. Fried DV, Tucker SL, Zhou S, et al. Prognostic value and reproducibility of pretreatment CT texture features in stage III non-small cell lung cancer. Int J Radiat Oncol Biol Phys 2014;90:834-42. [Crossref] [PubMed]
26. Hunter LA, Krafft S, Stingo F, et al. High quality machine-robust image features: identification in nonsmall cell lung cancer computed tomography images. Med Phys 2013;40:121916 [Crossref] [PubMed]
27. Fedorov A, Beichel R, Kalpathy-Cramer J, et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn Reson Imaging 2012;30:1323-41. [Crossref] [PubMed]
28. Werner-Wasik M, Nelson AD, Choi W, et al. What is the best way to contour lung tumors on PET scans? Multiobserver validation of a gradient-based method using a NSCLC digital PET phantom. Int J Radiat Oncol Biol Phys 2012;82:1164-71. [Crossref] [PubMed]
29. Fried DV, Mawlawi O, Zhang L, et al. Stage III Non-Small Cell Lung Cancer: Prognostic Value of FDG PET Quantitative Imaging Features Combined with Clinical Prognostic Factors. Radiology 2016;278:214-22. [Crossref] [PubMed]
30. van Rossum PS, Fried DV, Zhang L, et al. The Incremental Value of Subjective and Quantitative Assessment of 18F-FDG PET for the Prediction of Pathologic Complete Response to Preoperative Chemoradiotherapy in Esophageal Cancer. J Nucl Med 2016;57:691-700. [Crossref] [PubMed]
31. Basu S, Hall LO, Goldgof D, et al. Developing a Classifier Model for Lung Tumors in CT-Scan Images. IEEE SMC 2011:1306-12.
32. Cordova JS, Schreibmann E, Hadjipanayis CG, et al. Quantitative tumor segmentation for evaluation of extent of glioblastoma resection to facilitate multisite clinical trials. Transl Oncol 2014;7:40-7. [Crossref] [PubMed]
33. Leijenaar RT, Nalbantov G, Carvalho S, et al. The effect of SUV discretization in quantitative FDG-PET Radiomics: the need for standardized methodology in tumor texture analysis. Sci Rep 2015;5:11075. [Crossref] [PubMed]
34. Orlhac F, Soussan M, Maisonobe JA, et al. Tumor texture analysis in 18F-FDG PET: relationships between texture parameters, histogram indices, standardized uptake values, metabolic volumes, and total lesion glycolysis. J Nucl Med 2014;55:414-22. [Crossref] [PubMed]
35. Zhang L, Fried DV, Fave XJ, et al. IBEX: an open infrastructure software platform to facilitate collaborative work in radiomics. Med Phys 2015;42:1341-53. [Crossref] [PubMed]
36. Yip C, Davnall F, Kozarski R, et al. Assessment of changes in tumor heterogeneity following neoadjuvant chemotherapy in primary esophageal cancer. Dis Esophagus 2015;28:172-9. [Crossref] [PubMed]
37. Davnall F, Yip CS, Ljungqvist G, et al. Assessment of tumor heterogeneity: an emerging imaging tool for clinical practice? Insights Imaging 2012;3:573-89. [Crossref] [PubMed]
38. Ganeshan B, Abaleke S, Young RC, et al. Texture analysis of non-small cell lung cancer on unenhanced computed tomography: initial evidence for a relationship with tumour glucose metabolism and stage. Cancer Imaging 2010;10:137-43. [Crossref] [PubMed]
39. Ganeshan B, Burnand K, Young R, et al. Dynamic contrast-enhanced texture analysis of the liver: initial assessment in colorectal cancer. Invest Radiol 2011;46:160-8. [Crossref] [PubMed]
40. Ganeshan B, Goh V, Mandeville HC, et al. Non-small cell lung cancer: histopathologic correlates for texture parameters at CT. Radiology 2013;266:326-36. [Crossref] [PubMed]
41. Ganeshan B, Miles KA, Young RC, et al. Hepatic enhancement in colorectal cancer: texture analysis correlates with hepatic hemodynamics and patient survival. Acad Radiol 2007;14:1520-30. [Crossref] [PubMed]
42. Ganeshan B, Miles KA, Young RC, et al. In search of biologic correlates for liver texture on portal-phase CT. Acad Radiol 2007;14:1058-68. [Crossref] [PubMed]
43. Ganeshan B, Miles KA, Young RC, et al. Hepatic entropy and uniformity: additional parameters that can potentially increase the effectiveness of contrast enhancement during abdominal CT. Clin Radiol 2007;62:761-8. [Crossref] [PubMed]
44. Ganeshan B, Miles KA, Young RC, et al. Three-dimensional selective-scale texture analysis of computed tomography pulmonary angiograms. Invest Radiol 2008;43:382-94. [Crossref] [PubMed]
45. Ganeshan B, Miles KA, Young RC, et al. Texture analysis in non-contrast enhanced CT: impact of malignancy on texture in apparently disease-free areas of the liver. Eur J Radiol 2009;70:101-10. [Crossref] [PubMed]
46. Ganeshan B, Miles KA, Young RC, et al. Three-dimensional textural analysis of brain images reveals distributed grey-matter abnormalities in schizophrenia. Eur Radiol 2010;20:941-8. [Crossref] [PubMed]
47. Ganeshan B, Skogen K, Pressney I, et al. Tumour heterogeneity in oesophageal cancer assessed by CT texture analysis: preliminary evidence of an association with tumour metabolism, stage, and survival. Clin Radiol 2012;67:157-64. [Crossref] [PubMed]
48. Goh V, Ganeshan B, Nathan P, et al. Assessment of response to tyrosine kinase inhibitors in metastatic renal cell cancer: CT texture as a predictive biomarker. Radiology 2011;261:165-71. [Crossref] [PubMed]
49. Miles KA, Ganeshan B, Griffiths MR, et al. Colorectal cancer: texture analysis of portal phase hepatic CT images as a potential marker of survival. Radiology 2009;250:444-52. [Crossref] [PubMed]
50. Win T, Miles KA, Janes SM, et al. Tumor heterogeneity and permeability as measured on the CT component of PET/CT predict survival in patients with non-small cell lung cancer. Clin Cancer Res 2013;19:3591-9. [Crossref] [PubMed]
51. Parikh J, Selmi M, Charles-Edwards G, et al. Changes in primary breast cancer heterogeneity may augment midtreatment MR imaging assessment of response to neoadjuvant chemotherapy. Radiology 2014;272:100-12. [Crossref] [PubMed]
52. Raman SP, Schroeder JL, Huang P, et al. Preliminary data using computed tomography texture analysis for the classification of hypervascular liver lesions: generation of a predictive model on the basis of quantitative spatial frequency measurements--a work in progress. J Comput Assist Tomogr 2015;39:383-95. [PubMed]
53. Szczypiński PM, Strzelecki M, Materka A, et al. MaZda--a software package for image texture analysis. Comput Methods Programs Biomed 2009;94:66-76. [Crossref] [PubMed]
54. Strzelecki M, Szczypinski P, Materka A, et al. A software tool for automatic classification and segmentation of 2D/3D medical images. Nuclear Inst. and Methods in Physics Research 2013;702:137-40. [Crossref]
55. Blouin S, Moreau MF, Baslé MF, et al. Relations between radiograph texture analysis and microcomputed tomography in two rat models of bone metastases. Cells Tissues Organs 2006;182:182-92. [Crossref] [PubMed]
56. Santos A, Ramiro C, Desco M, et al. Automatic detection of cellular necrosis in epithelial cell cultures. Proc. SPIE 4322, Medical Imaging 2001: Image Processing, 1836. 2001.
57. Oliveira MS, Fernandes PT, Avelar WM, et al. Texture analysis of computed tomography images of acute ischemic stroke patients. Braz J Med Biol Res 2009;42:1076-9. [Crossref] [PubMed]
58. Ashton TJ, Michie I, Johnston IA. A novel tensile test method to assess texture and gaping in salmon fillets. J Food Sci 2010;75:S182-90. [Crossref] [PubMed]
59. Fang YH, Lin CY, Shih MJ, et al. Development and evaluation of an open-source software package "CGITA" for quantifying tumor heterogeneity with molecular images. Biomed Res Int 2014;2014:248505.
60. Cheng NM, Fang YH, Chang JT, et al. Textural features of pretreatment 18F-FDG PET/CT images: prognostic significance in patients with advanced T-stage oropharyngeal squamous cell carcinoma. J Nucl Med 2013;54:1703-9. [Crossref] [PubMed]
61. Yip S, McCall K, Aristophanous M, et al. Comparison of texture features derived from static and respiratory-gated PET images in non-small cell lung cancer. PLoS One 2014;9:e115510 [Crossref] [PubMed]
62. Fave X, Cook M, Frederick A, et al. Investigating the impact of potential sources of uncertainty on quantitative imaging features. Comput Med Imaging Graph 2015;44:54-61. [Crossref] [PubMed]
63. Deasy JO, Blanco AI, Clark VH. CERR: a computational environment for radiotherapy research. Med Phys 2003;30:979-85. [Crossref] [PubMed]
64. Apte A, Veeraraghavan H, Oh J, et al. SU-E-J-253: The Radiomics Toolbox in the Computational Environment for Radiological Research (CERR). Med Phys 2015;42:3324. [Crossref]
65. Apte A, Wang Y, Deasy J. SU-E-I-66: Radiomics and Image Registration Updates for the Computational Environment for Radiotherapy Research (CERR). Med Phys 2014;41:145. [Crossref]
66. Chaddad A, Zinn PO, Colen RR. Radiomics texture feature extraction for characterizing GBM phenotypes using GLCM. New York:IEEE, 2015:84-7.
67. Parmar C, Grossmann P, Bussink J, et al. Machine Learning methods for Quantitative Radiomic Biomarkers. Sci Rep 2015;5:13087. [Crossref] [PubMed]
68. Lambin P, Rios-Velazquez E, Leijenaar R, et al. Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 2012;48:441-6. [Crossref] [PubMed]
69. Segal E, Sirlin CB, Ooi C, et al. Decoding global gene expression programs in liver cancer by noninvasive imaging. Nat Biotechnol 2007;25:675-80. [Crossref] [PubMed]
70. Pickering CR, Shah K, Ahmed S, et al. CT imaging correlates of genomic expression for oral cavity squamous cell carcinoma. AJNR Am J Neuroradiol 2013;34:1818-22. [Crossref] [PubMed]
71. Lee JH, Park CM, Park SJ, et al. Value of Computerized 3D Shape Analysis in Differentiating Encapsulated from Invasive Thymomas. PLoS One 2015;10:e0126175 [Crossref] [PubMed]
72. Hatt M, Majdoub M, Vallières M, et al. 18F-FDG PET uptake characterization through texture analysis: investigating the complementary nature of heterogeneity and functional tumor volume in a multi-cancer site patient cohort. J Nucl Med 2015;56:38-44. [Crossref] [PubMed]
73. Gentleman RC, Carey VJ, Bates DM, et al. Bioconductor: open software development for computational biology and bioinformatics. Genome Biol 2004;5:R80. [Crossref] [PubMed]
74. Hawkins SH, Korecki JN, Balagurunathan Y, et al. Predicting Outcomes of Nonsmall Cell Lung Cancer Using CT Image Features. IEEE Access 2014;2:1418-26.
75. Wang J, Kato F, Oyama-Manabe N, et al. Identifying Triple-Negative Breast Cancer Using Background Parenchymal Enhancement Heterogeneity on Dynamic Contrast-Enhanced MRI: A Pilot Radiomics Study. PLoS One 2015;10:e0143308 [Crossref] [PubMed]