Reliability of assessment of protein structure prediction methods

Marc A. Marti-Renom, M. S. Madhusudhan, Andras Fiser, Burkhard Rost, Andrej Sali

Research output: Contribution to journalArticle

76 Citations (Scopus)

Abstract

The reliability of ranking of protein structure modeling methods is assessed. The assessment is based on the parametric Student's t test and the nonparametric Wilcox signed rank test of statistical significance of the difference between paired samples. The approach is applied to the ranking of the comparative modeling methods tested at the fourth meeting on Critical Assessment of Techniques for Protein Structure Prediction (CASP). It is shown that the 14 CASP4 test sequences may not be sufficient to reliably distinguish between the top eight methods, given the model quality differences and their standard deviations. We suggest that CASP needs to be supplemented by an assessment of protein structure prediction methods that is automated, continuous in time, based on several criteria applied to a large number of models, and with quantitative statistical reliability assigned to each characterization.

Original languageEnglish (US)
Pages (from-to)435-440
Number of pages6
JournalStructure
Volume10
Issue number3
DOIs
StatePublished - 2002
Externally publishedYes

Fingerprint

Proteins
Nonparametric Statistics
Students

Keywords

  • CASP
  • Comparative protein structure modeling
  • EVA
  • Homology modeling
  • LiveBench
  • Model assessment
  • Statistical significance

ASJC Scopus subject areas

  • Molecular Biology
  • Structural Biology

Cite this

Marti-Renom, M. A., Madhusudhan, M. S., Fiser, A., Rost, B., & Sali, A. (2002). Reliability of assessment of protein structure prediction methods. Structure, 10(3), 435-440. https://doi.org/10.1016/S0969-2126(02)00731-1

Reliability of assessment of protein structure prediction methods. / Marti-Renom, Marc A.; Madhusudhan, M. S.; Fiser, Andras; Rost, Burkhard; Sali, Andrej.

In: Structure, Vol. 10, No. 3, 2002, p. 435-440.

Research output: Contribution to journalArticle

Marti-Renom, MA, Madhusudhan, MS, Fiser, A, Rost, B & Sali, A 2002, 'Reliability of assessment of protein structure prediction methods', Structure, vol. 10, no. 3, pp. 435-440. https://doi.org/10.1016/S0969-2126(02)00731-1
Marti-Renom, Marc A. ; Madhusudhan, M. S. ; Fiser, Andras ; Rost, Burkhard ; Sali, Andrej. / Reliability of assessment of protein structure prediction methods. In: Structure. 2002 ; Vol. 10, No. 3. pp. 435-440.
@article{70d1dd2c0b164e58b6c6d88010e1ce62,
title = "Reliability of assessment of protein structure prediction methods",
abstract = "The reliability of ranking of protein structure modeling methods is assessed. The assessment is based on the parametric Student's t test and the nonparametric Wilcox signed rank test of statistical significance of the difference between paired samples. The approach is applied to the ranking of the comparative modeling methods tested at the fourth meeting on Critical Assessment of Techniques for Protein Structure Prediction (CASP). It is shown that the 14 CASP4 test sequences may not be sufficient to reliably distinguish between the top eight methods, given the model quality differences and their standard deviations. We suggest that CASP needs to be supplemented by an assessment of protein structure prediction methods that is automated, continuous in time, based on several criteria applied to a large number of models, and with quantitative statistical reliability assigned to each characterization.",
keywords = "CASP, Comparative protein structure modeling, EVA, Homology modeling, LiveBench, Model assessment, Statistical significance",
author = "Marti-Renom, {Marc A.} and Madhusudhan, {M. S.} and Andras Fiser and Burkhard Rost and Andrej Sali",
year = "2002",
doi = "10.1016/S0969-2126(02)00731-1",
language = "English (US)",
volume = "10",
pages = "435--440",
journal = "Structure with Folding & design",
issn = "0969-2126",
publisher = "Cell Press",
number = "3",

}

TY - JOUR

T1 - Reliability of assessment of protein structure prediction methods

AU - Marti-Renom, Marc A.

AU - Madhusudhan, M. S.

AU - Fiser, Andras

AU - Rost, Burkhard

AU - Sali, Andrej

PY - 2002

Y1 - 2002

N2 - The reliability of ranking of protein structure modeling methods is assessed. The assessment is based on the parametric Student's t test and the nonparametric Wilcox signed rank test of statistical significance of the difference between paired samples. The approach is applied to the ranking of the comparative modeling methods tested at the fourth meeting on Critical Assessment of Techniques for Protein Structure Prediction (CASP). It is shown that the 14 CASP4 test sequences may not be sufficient to reliably distinguish between the top eight methods, given the model quality differences and their standard deviations. We suggest that CASP needs to be supplemented by an assessment of protein structure prediction methods that is automated, continuous in time, based on several criteria applied to a large number of models, and with quantitative statistical reliability assigned to each characterization.

AB - The reliability of ranking of protein structure modeling methods is assessed. The assessment is based on the parametric Student's t test and the nonparametric Wilcox signed rank test of statistical significance of the difference between paired samples. The approach is applied to the ranking of the comparative modeling methods tested at the fourth meeting on Critical Assessment of Techniques for Protein Structure Prediction (CASP). It is shown that the 14 CASP4 test sequences may not be sufficient to reliably distinguish between the top eight methods, given the model quality differences and their standard deviations. We suggest that CASP needs to be supplemented by an assessment of protein structure prediction methods that is automated, continuous in time, based on several criteria applied to a large number of models, and with quantitative statistical reliability assigned to each characterization.

KW - CASP

KW - Comparative protein structure modeling

KW - EVA

KW - Homology modeling

KW - LiveBench

KW - Model assessment

KW - Statistical significance

UR - http://www.scopus.com/inward/record.url?scp=0036122662&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0036122662&partnerID=8YFLogxK

U2 - 10.1016/S0969-2126(02)00731-1

DO - 10.1016/S0969-2126(02)00731-1

M3 - Article

C2 - 12005441

AN - SCOPUS:0036122662

VL - 10

SP - 435

EP - 440

JO - Structure with Folding & design

JF - Structure with Folding & design

SN - 0969-2126

IS - 3

ER -