StAR: a simple tool for the statistical comparison of ROC curves.
paper
Cited
Public
- Authors
- Vergara, Ismael A; Norambuena, TomΓ‘s; Ferrada, Evandro; Slater, Alex W; Melo, Francisco
- Year
- 2008
- Journal
- BMC bioinformatics
- PMID
- 18534022
- DOI
- 10.1186/1471-2105-9-265
- PMCID
- PMC2435548
BACKGROUND: As in many different areas of science and technology, most important problems in bioinformatics rely on the proper development and assessment of binary classifiers. A generalized assessment of the performance of binary classifiers is typically carried out through the analysis of their receiver operating characteristic (ROC) curves. The area under the ROC curve (AUC) constitutes a popular indicator of the performance of a binary classifier. However, the assessment of the statistical significance of the difference between any two classifiers based on this measure is not a straightforward task, since not many freely available tools exist. Most existing software is either not free, difficult to use or not easy to automate when a comparative assessment of the performance of many binary classifiers is intended. This constitutes the typical scenario for the optimization of parameters when developing new classifiers and also for their performance validation through the comparison to previous art. RESULTS: In this work we describe and release new software to assess the statistical significance of the observed difference between the AUCs of any two classifiers for a common task estimated from paired data or unpaired balanced data. The software is able to perform a pairwise comparison of many classifiers in a single run, without requiring any expert or advanced knowledge to use it. The software relies on a non-parametric test for the difference of the AUCs that accounts for the correlation of the ROC curves. The results are displayed graphically and can be easily customized by the user. A human-readable report is generated and the complete data resulting from the analysis are also available for download, which can be used for further analysis with other software. The software is released as a web server that can be used in any client platform and also as a standalone application for the Linux operating system. CONCLUSION: A new software for the statistical comparison of ROC curves is released here as a web server and also as standalone software for the LINUX operating system.
No figures extracted from this document.
1β20 of 38
Next β
No entities extracted from this document yet.
No uploaded files.
Not in any collection.
In this knowledge base
| Title | Year | PMID |
|---|---|---|
| On the association of common and rare genetic variation influencing body mass index: a combined SNP and CNV analysis. | 2014 | 24884913 |
External
| Title | Authors | Journal | Year | Link |
|---|---|---|---|---|
| ELW-CNN: An extremely lightweight convolutional neural network for enhancing interoperability in colon and lung cancer identification using explainable AI. | Opee SA et al. | β | 2025 | β |
| Identifying melanoma among benign simulators - Is there a role for deep learning convolutional neural networks? (MelSim Study). | Vollmer AS et al. | β | 2025 | β |
| The art of diagnosing rare skin tumors: Can DL-CNNs enhance dermatologists' diagnostic accuracy? | Vollmer AS et al. | β | 2025 | β |
| Discriminating Parkinson's disease patients from healthy controls using nasal respiratory airflow. | Andelman-Gur M et al. | β | 2024 | β |
| Utility of diffusion-weighted imaging in differentiating benign and malignant breast lesions. | Johnson A et al. | β | 2024 | β |
| Assessment of Diagnostic Performance of Dermatologists Cooperating With a Convolutional Neural Network in a Prospective Clinical Study: Human With Machine. | Winkler JK et al. | β | 2023 | β |
| Direct Gene Expression Profile Prediction for Uveal Melanoma from Digital Cytopathology Images via Deep Learning and Salient Image Region Identification. | Liu TYA et al. | β | 2023 | β |
| How Effective is the Systemic Inflammatory Immune Index in the Etiopathogenesis of Isolated Coronary Artery Ectasia?Reply. | Toprak K et al. | β | 2023 | β |
| Muscle atonia index during multiple sleep latency test: A possible marker to differentiate narcolepsy from other hypersomnias. | Romigi A et al. | β | 2023 | β |
| Predictive value of abbreviated olfactory tests in prodromal Parkinson disease. | Vaswani PA et al. | β | 2023 | β |
| Accurate and generalizable quantitative scoring of liver steatosis from ultrasound images <i>via</i> scalable deep learning. | Li B et al. | β | 2022 | β |
| A New Family of Similarity Measures for Scoring Confidence of Protein Interactions Using Gene Ontology. | Paul M et al. | β | 2022 | β |
| An olfactory self-test effectively screens for COVID-19. | Snitz K et al. | β | 2022 | β |
| Integrated structure-based protein interface prediction. | Walder M et al. | β | 2022 | β |
| PEGALUS: predictivity of elderly age, arterial gas analysis, and lung ultrasound. A new prognostic score for COVID-19 patients in the emergency department-an observational prospective study. | Borio G et al. | β | 2022 | β |
| Combining information on nocturnal rapid eye movement sleep latency and atonia to facilitate diagnosis of pediatric narcolepsy type 1. | Silvani A et al. | β | 2021 | β |
| Machine learning for filtering out false positive grey matter atrophies in single subject voxel based morphometry: A simulation based study. | KΓΌlsgaard HC et al. | β | 2021 | β |
| Which is the optimally defined vestibular cross-section to diagnose unilateral Meniere's disease with delayed post-gadolinium 3D fluid-attenuated inversion recovery MRI? | Connor S et al. | β | 2021 | β |
| Deep hiearchical multi-label classification applied to chest X-ray abnormality taxonomies. | Chen H et al. | β | 2020 | β |
| Diagnostic utility of the Addenbrooke's Cognitive Examination - III (ACE-III), Mini-ACE, Mini-Mental State Examination, Montreal Cognitive Assessment, and Hasegawa Dementia Scale-Revised for detecting mild cognitive impairment and dementia. | Senda M et al. | β | 2020 | β |
| MHCII3D-Robust Structure Based Prediction of MHC II Binding Peptides. | Laimer J et al. | β | 2020 | β |
| Past and present of computer-assisted dermoscopic diagnosis: performance of a conventional image analyser versus a convolutional neural network in a prospective data set of 1,981 skin lesions. | Sies K et al. | β | 2020 | β |
| REFUGEΒ Challenge: A unified framework for evaluating automatedΒ methods for glaucomaΒ assessment from fundus photographs. | Orlando JI et al. | β | 2020 | β |
| A baseline study for detection of Parkinson's disease with 3D-transcranial sonography and uni-lateral reconstruction. | Plate A et al. | β | 2019 | β |
| A Weighted Genetic Risk Score Based on Four APOE-Independent Alzheimer's Disease Risk Loci May Supplement APOE E4 for Better Disease Prediction. | Zhang C et al. | β | 2019 | β |
| External carotid artery plaques are associated with intracranial stenosis in patients with advanced coronary artery disease. | Valaikiene J et al. | β | 2019 | β |
| Histopathological risk scoring system as a tool for predicting lymph nodal metastasis in penile squamous cell carcinoma. | Sali AP et al. | β | 2019 | β |
| Interoperator Reproducibility of Carotid Elastography for Identification of Vulnerable Atherosclerotic Plaques. | Liu Z et al. | β | 2019 | β |
| Prognostic power of biomarkers for short-term mortality in the elderly patients seen in Emergency Departments due to infections. | JuliΓ‘n-JimΓ©nez A et al. | β | 2019 | β |
| Use of the Bispectral Index to Predict Recovery of Consciousness in Patients with Spontaneous Intracerebral Hemorrhage After Surgical Hematoma Evacuation: A Prospective Cohort Study. | Yang YL et al. | β | 2019 | β |
| Validation of Addenbrooke's cognitive examination III for detecting mild cognitive impairment and dementia in Japan. | Takenoshita S et al. | β | 2019 | β |
| Comparative Analysis of Markers of Mass Effect after Ischemic Stroke. | Ostwaldt AC et al. | β | 2018 | β |
| Machine learning algorithms based on signals from a single wearable inertial sensor can detect surface- and age-related differences in walking. | Hu B et al. | β | 2018 | β |
| Response to cardiac resynchronization therapy is determined by intrinsic electrical substrate rather than by its modification. | Strik M et al. | β | 2018 | β |
| Characterisation of preproendothelin-1 derived peptides identifies Endothelin-Like Domain Peptide as a modulator of Endothelin-1. | Yuzugulen J et al. | β | 2017 | β |
| Diffusion-weighted imaging of breast lesions: Region-of-interest placement and different ADC parameters influence apparent diffusion coefficient values. | Bickel H et al. | β | 2017 | β |
| Optimization of SPECT Measurement of Myocardial Blood Flow with Corrections for Attenuation, Motion, and Blood Binding Compared with PET. | Wells RG et al. | β | 2017 | β |
| Predicting primary progressive aphasias with support vector machine approaches in structural MRI data. | Bisenius S et al. | β | 2017 | β |
| Rey's Auditory Verbal Learning Test scores can be predicted from whole brain MRI in Alzheimer's disease. | Moradi E et al. | β | 2017 | β |
| A chronic neural interface to the macaque dorsal column nuclei. | Richardson AG et al. | β | 2016 | β |
| A Potential VEP Biomarker for Mild Cognitive Impairment: Evidence from Selective Visual Deficit of Higher-Level Dorsal Pathway. | Yamasaki T et al. | β | 2016 | β |
| Comparison of frequency doubling and flicker defined form perimetry in early glaucoma. | Horn FK et al. | β | 2016 | β |
| COMPUTING THERAPY FOR PRECISION MEDICINE: COLLABORATIVE FILTERING INTEGRATES AND PREDICTS MULTI-ENTITY INTERACTIONS. | Regenbogen S et al. | β | 2016 | β |
| Does Epoetin Beta Still Have a Place in Peginterferon Alpha-2a Plus Ribavirin Treatment Strategies for Chronic Hepatitis C? | Veillon P et al. | β | 2016 | β |
| Efficient and automated large-scale detection of structural relationships in proteins with a flexible aligner. | GutiΓ©rrez FI et al. | β | 2016 | β |
| A multigene array for measurable residual disease detection in AML patients undergoing SCT. | Goswami M et al. | β | 2015 | β |
| Differential methylation of the promoter and first exon of the RASSF1A gene in hepatocarcinogenesis. | Jain S et al. | β | 2015 | β |
| Flicker-defined form perimetry in glaucoma patients. | Horn FK et al. | β | 2015 | β |
| Genome-wide prediction of prokaryotic two-component system networks using a sequence-based meta-predictor. | Kara A et al. | β | 2015 | β |
| MiR-205 and MiR-375 microRNA assays to distinguish squamous cell carcinoma from adenocarcinoma in lung cancer biopsies. | Patnaik S et al. | β | 2015 | β |
| NABS: non-local automatic brain hemisphere segmentation. | Romero JE et al. | β | 2015 | β |
| Prediction of Spontaneous Protein Deamidation from Sequence-Derived Secondary Structure and Intrinsic Disorder. | Lorenzo JR et al. | β | 2015 | β |
| Serum HE4 detects recurrent endometrial cancer in patients undergoing routine clinical surveillance. | Brennan DJ et al. | β | 2015 | β |
| Urinary neutrophil gelatinase-associated lipocalin time course during cardiac surgery. | Bignami E et al. | β | 2015 | β |
| A new clinical multivariable model that predicts postoperative acute kidney injury: impact of endogenous ouabain. | Simonini M et al. | β | 2014 | β |
| In silico comparison of low- and high-risk human papillomavirus proteins. | Alemi M et al. | β | 2014 | β |
| On the association of common and rare genetic variation influencing body mass index: a combined SNP and CNV analysis. | Peterson RE et al. | β | 2014 | β |
| PhosphoSVM: prediction of phosphorylation sites by integrating various protein sequence attributes with a support vector machine. | Dou Y et al. | β | 2014 | β |
| Predicting acute acetaminophen hepatotoxicity with acetaminophen-aminotransferase multiplication product and the Psi parameter. | Chomchai S et al. | β | 2014 | β |
| Prediction of long-term segmental and global functional recovery of hibernating myocardium after revascularisation based on low dose dobutamine and late gadolinium enhancement cardiovascular magnetic resonance. | Glaveckaite S et al. | β | 2014 | β |
| Proteome screening of pleural effusions identifies galectin 1 as a diagnostic biomarker and highlights several prognostic biomarkers for malignant mesothelioma. | Mundt F et al. | β | 2014 | β |
| Serum peptide pattern that differentially diagnoses hepatitis B virus-related hepatocellular carcinoma from liver cirrhosis. | Wang N et al. | β | 2014 | β |
| Systematic detection of internal symmetry in proteins using CE-Symm. | Myers-Turnbull D et al. | β | 2014 | β |
| Two new mathematical models for prediction of early mortality risk in coronary artery bypass graft surgery. | Ghavidel AA et al. | β | 2014 | β |
| An MLP classifier for prediction of HBV-induced liver cirrhosis using routinely available clinical parameters. | Cao Y et al. | β | 2013 | β |
| Cortical hand bone porosity and its association with distal radius fracture in middle aged and elderly women. | Dhainaut A et al. | β | 2013 | β |
| Effectiveness of FISK, an invasiveness screening tool for non-native freshwater fishes, to perform risk identification assessments in the Iberian Peninsula. | Almeida D et al. | β | 2013 | β |
| Leukemia prediction using sparse logistic regression. | Manninen T et al. | β | 2013 | β |
| Preoperative endogenous ouabain predicts acute kidney injury in cardiac surgery patients. | Bignami E et al. | β | 2013 | β |
| Prognostic value of plasma lactate levels among patients with acute pulmonary embolism: the thrombo-embolism lactate outcome study. | Vanni S et al. | β | 2013 | β |
| The role of transient elastography in the detection of liver disease in patients with chronic pancreatitis. | Frossard JL et al. | β | 2013 | β |
| The utility of patients' self-perceptions of violence risk: consider asking the person who may know best. | Skeem JL et al. | β | 2013 | β |
| Two classifiers based on serum peptide pattern for prediction of HBV-induced liver cirrhosis using MALDI-TOF MS. | Cao Y et al. | β | 2013 | β |
| Baseline tumor oxygen saturation correlates with a pathologic complete response in breast cancer patients undergoing neoadjuvant chemotherapy. | Ueda S et al. | β | 2012 | β |
| Biomine: predicting links between biological entities using network models of heterogeneous databases. | Eronen L et al. | β | 2012 | β |
| Can child care workers contribute to the early detection of autism spectrum disorders? A comparison between screening instruments with child care workers versus parents as informants. | Dereu M et al. | β | 2012 | β |
| Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them). | Berrar D et al. | β | 2012 | β |
| Cystatin C as a predictor of all-cause mortality and myocardial infarction in patients with non-ST-elevation acute coronary syndrome. | Ristiniemi N et al. | β | 2012 | β |
| Dissecting sources of quantitative gene expression pattern divergence between Drosophila species. | Wunderlich Z et al. | β | 2012 | β |
| Extending signaling pathways with protein-interaction networks. Application to apoptosis. | Planas-Iglesias J et al. | β | 2012 | β |
| HIV-1 phenotypic reverse transcriptase inhibitor drug resistance test interpretation is not dependent on the subtype of the virus backbone. | Bronze M et al. | β | 2012 | β |
| L1pred: a sequence-based prediction tool for catalytic residues in enzymes with the L1-logreg classifier. | Dou Y et al. | β | 2012 | β |
| MicroRNA expression profiles of whole blood in lung adenocarcinoma. | Patnaik SK et al. | β | 2012 | β |
| Noninvasive work of breathing improves prediction of post-extubation outcome. | Banner MJ et al. | β | 2012 | β |
| Optimizing the risk estimation after a transient ischaemic attack - the ABCDEβ score. | Engelter ST et al. | β | 2012 | β |
| Prediction of incident type 2 diabetes mellitus based on a twenty-year follow-up of the Ventimiglia heart study. | Noto D et al. | β | 2012 | β |
| Risk stratification of patients suspected of coronary artery disease: comparison of five different models. | Jensen JM et al. | β | 2012 | β |
| Spectral characteristics of the PhNR in the full-field flash electroretinogram of normals and glaucoma patients. | Kremers J et al. | β | 2012 | β |
| SVMTriP: a method to predict antigenic epitopes using support vector machine to integrate tri-peptide similarity and propensity. | Yao B et al. | β | 2012 | β |
| Troponin T measurements by high-sensitivity vs conventional assays for risk stratification in acute dyspnea. | van Wijk S et al. | β | 2012 | β |
| Validation of the revised Addenbrooke's Cognitive Examination (ACE-R) for detecting mild cognitive impairment and dementia in a Japanese population. | Yoshida H et al. | β | 2012 | β |
| Variable decelerations: do size and shape matter? | Hamilton E et al. | β | 2012 | β |
| Asymptomatic carotid plaque rupture with unexpected thrombosis over a non-canonical vulnerable lesion. | Mauriello A et al. | β | 2011 | β |
| Baseline circulating tumor cell counts significantly enhance a prognostic score for patients participating in phase I oncology trials. | Olmos D et al. | β | 2011 | β |
| Clinical prediction of pulmonary embolism in respiratory emergencies. | Tsimogianni AM et al. | β | 2011 | β |
| Diabetes antibody standardization program: first proficiency evaluation of assays for autoantibodies to zinc transporter 8. | Lampasona V et al. | β | 2011 | β |
| External validation and comparison of three scores to predict renal replacement therapy after cardiac surgery: a multicenter cohort. | Vives M et al. | β | 2011 | β |
| Frequency doubling technique perimetry and spectral domain optical coherence tomography in patients with early glaucoma. | Horn FK et al. | β | 2011 | β |
| Improving the prediction of protein binding sites by combining heterogeneous data and Voronoi diagrams. | Segura J et al. | β | 2011 | β |
| One-class classification as a novel method of ligand-based virtual screening: the case of glycogen synthase kinase 3Ξ² inhibitors. | Karpov PV et al. | β | 2011 | β |
| Pentraxin-3 as a marker of disease activity in Takayasu arteritis. | Dagna L et al. | β | 2011 | β |
| Phalangeal densitometry compared with dual energy X-ray absorptiometry for assessment of bone mineral density in elderly women. | Dhainaut A et al. | β | 2011 | β |
| Receiver operating characteristic (ROC) analysis of day 5 morphology grading and metabolomic Viability Score on predicting implantation outcome. | Seli E et al. | β | 2011 | β |
| Structure-based virtual screening of glycogen synthase kinase 3Ξ² inhibitors: analysis of scoring functions applied to large true actives and decoy sets. | Osolodkin DI et al. | β | 2011 | β |
| Value of scar imaging and inotropic reserve combination for the prediction of segmental and global left ventricular functional recovery after revascularisation. | Glaveckaite S et al. | β | 2011 | β |
| X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. | Avni U et al. | β | 2011 | β |
| Assessment of depression in Parkinson's disease: the contribution of somatic symptoms to the clinimetric performance of the Hamilton and Montgomery-Asberg rating scales. | Reijnders JS et al. | β | 2010 | β |
| Fast and accurate protein substructure searching with simulated annealing and GPUs. | Stivala AD et al. | β | 2010 | β |
| Glaucoma risk index: automated glaucoma detection from color fundus images. | Bock R et al. | β | 2010 | β |
| Predictive value of abdominal obesity vs. body mass index for determining risk of intensive care unit mortality. | Paolini JB et al. | β | 2010 | β |
| The ability of hand digital X-ray radiogrammetry to identify middle-aged and elderly women with reduced bone density, as assessed by femoral neck dual-energy X-ray absorptiometry. | Dhainaut A et al. | β | 2010 | β |
| Accurate discrimination of conserved coding and non-coding regions through multiple indicators of evolutionary dynamics. | RΓ¨ M et al. | β | 2009 | β |
| Automated MRI-based classification of primary progressive aphasia variants. | Wilson SM et al. | β | 2009 | β |
| Clusterin in stool: a new biomarker for colon cancer screening? | Pucci S et al. | β | 2009 | β |
| Effective knowledge-based potentials. | Ferrada E et al. | β | 2009 | β |
| Identification of novel DNA repair proteins via primary sequence, secondary structure, and homology. | Brown JB et al. | β | 2009 | β |
| Single-unit stability using chronically implanted multielectrode arrays. | Dickey AS et al. | β | 2009 | β |
| Splitting statistical potentials into meaningful scoring functions: testing the prediction of near-native structures from decoy conformations. | Aloy P et al. | β | 2009 | β |
| Tailoring vocabularies for NLP in sub-domains: a method to detect unused word sense. | Figueroa RL et al. | β | 2009 | β |
| Towards the prediction of essential genes by integration of network topology, cellular localization and biological process information. | Acencio ML et al. | β | 2009 | β |
| Improving position-specific predictions of protein functional sites using phylogenetic motifs. | Dukka BK et al. | β | 2008 | β |