Results Recalculated in 2019
Retrospective view

Introduction

Methods for obtaining information about protein structure from the amino acid sequence have apparently been advancing rapidly. But just what can these methods currently deliver?

A first large scale experiment aimed at beginning to answer these questions was conducted in 1994, and culminated in a meeting at Asilomar, California at the end of that year. Some 135 predictions were made by 35 different groups. The results are published in a special issue of Proteins: Structure, Function and Genetics, volume 23, No 3, November 1995.

A second meeting on the Critical Assessment of Techniques for Protein Structure Prediction (December 1996) was a culmination of a 9 month long, community wide experiment. Before the meeting, 42 structural targets provided by crystallographers and NMR spectroscopists were made available to the prediction community. Prior to the public release of structures, more than 900 predictions by approximately 70 research groups world wide were collected. The results are published in a special issue of Proteins: Structure, Function and Genetics, Suppl.1, 1997.

We now announce the third experiment. As before, the goal is to obtain an in-depth and objective assessment of our current abilities and inabilities in this area. To this end, participants will predict as much as possible about a set of soon to be known structures in advance of the meeting. Sessions will be devoted to presentation of the results and comparison with experiment, and to the description of the methods used.

Experiment

The broad goals of the experiment are to address the following questions about the current state of the art in protein structure prediction:
  1. Are the models produced similar to the corresponding experimental structure?
  2. Are the models correctly aligned with the experimental structures?
  3. Have similar structures that a model can be based on been identified?
  4. Are the details of the models correct?
  5. Has there been progress between CASP2 and CASP3?
  6. What methods are most effective?
  7. Where can future effort must be productively be focused?
As in CASP2, all types of methods for predicting protein structure will be considered. However, docking predictions will not be included in CASP3, since a separate experiment is being planned to evaluate this.

Collection of Prediction Targets

As before, for the experiment to succeed, it is essential that we obtain the help of the experimental community. Therefore we invite Protein crystallographers and NMR spectroscopists to provide details of structures they expect to have made public before 1st October 1998. They are also asked to notify the organizers shortly before the structure is solved.

Participation

Participation in the experiment is open to all. Predictors may form teams, and each team must have a designated group leader. Except in special circumstances, and then by agreement with the organizers, a predictor may only be a member of a single team. Each team will be issued a unique ID number, which will serve to identify their predictions. Those interested in receiving mailings concerning progress of the experiment may also register as 'observers' . Prediction targets will be made available through this web site. All targets will be assigned an expiry date, and predictions must be received and accepted before that expiration. Predictions may be submitted through a web interface, or by email. In order to be accepted, submissions must contain all the required information, and conform to the specified format. All accepted predictions will be given an accession number, and this number will be considered to provide proof of prediction submission. One team may submit up to five models of a target, although most evaluation and assessment will focus on only one of these.

Assessment of Predictions

As in previous CASPs, independent assessors will evaluate the predictions. There will be three assessors, representing expertise in the comparative modeling, fold recognition and ab initio prediction areas. Predictions will be assigned to assessors on the basis of the level of detail and accuracy of the models, as well as the methods used. Assessors will be provided with the results of numerical evaluation of the predictions, and will judge the results primarily on that basis. Numerical evaluation criteria have been revised from those used in CASP2, and are now being finalized, in consultation with the prediction community. The assessors will be asked to focus particularly on the effectiveness of different methods.

Release of Results

All predictions and prediction evaluations will be made available through this web site, shortly before the meeting.

Principal Changes from CASP2

  1. No formal pre-division of targets in the categories of comparative modeling, threading and ab initio. These distinctions were blurred in CASP2. For example, alignments produced by threading should be evaluated in the same way as those from comparative modeling, and so called 'ab initio' methods are increasingly relying on the structure knowledge base in a direct manner, as in the 'mini-threading' methods.

    Predictions will be assessed on the basis of the quality of the models produced, as much as on the individual methods used. However, there will still be three assessors, one with responsibility of assessing methods that produce relatively accurate and detailed models (nominally comparative modeling), one for assessing aspects of models based on recognition of a fold or parts of a fold, by threading or advanced sequence comparison techniques (previously threading), and one for assessing aspects of models based on ab initio methods. As before, approximately one day of the meeting will be devoted to each of these areas.

  2. More emphasis on identifying which methods are most effective. Several mechanisms will be used for this:

    • Predictors will be asked to provide long abstracts describing their methods, and these will be available on the web in advance of the meeting.

    • Predictors will be encouraged to bring methods oriented posters to the meeting.

    • Assessors will be asked to take originality of methods into account in choosing meeting speakers.

    • Assessors will have more time to do their work, and thus more time to extract messages about methods from the results.

  3. Revisions to numerical evaluation criteria. The numerical criteria implemented for CASP2 are being revised. The most extensive changes are to those used in the threading category, and these have been assembled in consultation with a group of veteran CASP threaders.

  4. Simplification and unification of prediction file formats. In order to facilitate 'category free' predictions, a common core format will be used for all predictions.

  5. More prediction targets. A limitation on the significance of the CASP2 results, particularly in threading, was the relatively small number of targets available. Every effort will be made increase the stock for CASP3. Predictors are urged to help in this process, and as always, we rely on the co-operation of our experimental colleagues.

Meeting

A meeting will be held 13-17 December, 1998 at Asilomar, California, USA to evaluate the results of the prediction experiment. The meeting will be limited to about 170 participants and precedence will be given to active predictors. It is anticipated that some financial assistance will be available for the more successful predictors.

There will be lectures by the assessors and the more successful predictors and a moderated discussion on the state of art in each category. Emphasis will be on what went right and what went wrong, and on what progress has been made since the last experiment. In particular, in what areas further effort is likely to pay off in terms of improved predictions. There will also be poster sessions and informal sessions using computer workstations. Broadly one day will be devoted to comparative modeling, fold recognition, and ab initio methods, depending on the levels of response.

The proceedings of the meeting will be refereed and published in Proteins.

Timetable

April '98 - August '98
Distribution of targets to predictors as they become available, withdrawal of structures as they are solved. Collection of predictions.
December 13 - 17 '98
Meeting, Asilomar, California

Organizing Committee

John Moult            CARB, University of Maryland, USA
Tim Hubbard           Sanger Centre, Hinxton, UK
Jan Pedersen          Acadia Pharmaceuticals, Denmark
Krzysztof Fidelis     Lawrence Livermore National Laboratory, USA
One organizer will co-ordinate affairs in the each of the three types of prediction: Jan Pedersen for comparative modeling, John Moult for fold recognition, and Tim Hubbard for ab initio prediction. As far as practical, the co-ordinators will consult with veteran CASP participants in all aspects of the experiment. Collection and distribution of targets, registration of predictors, and collection and analysis of predictions will be handled through the Prediction Center at Livermore, run by Krzysztof Fidelis, Adam Zemla and Ceslovas Venclovas.
Protein Structure Prediction Center
Sponsored by the US National Institute of General Medical Sciences (NIH/NIGMS)
Please address any questions or queries to:
© 2007-2019, University of California, Davis