Introduction
Methods for obtaining information about protein structure from the
amino acid sequence have apparently been advancing rapidly. But just
what can these methods currently deliver?
A first large scale
experiment aimed at beginning to answer these questions was conducted
in 1994, and culminated in a meeting at Asilomar, California at the
end of that year. Some 135 predictions were made by 35 different
groups. The results are published in a special issue of Proteins:
Structure, Function and Genetics, volume 23, No 3, November 1995.
A second meeting on the Critical Assessment of
Techniques for Protein Structure Prediction
(December 1996) was a culmination of a 9 month long, community wide
experiment.
Before the meeting, 42 structural targets provided by crystallographers and NMR
spectroscopists were made available to the prediction community.
Prior to the
public release of structures, more than 900 predictions by approximately
70 research groups world wide were collected.
The results are published in a special issue of Proteins:
Structure, Function and Genetics, Suppl.1, 1997.
We now announce the third experiment. As before, the goal is to
obtain an in-depth and objective assessment of our current abilities
and inabilities in this area. To this end, participants will predict
as much as possible about a set of soon to be known structures in
advance of the meeting. Sessions will be devoted to presentation of
the results and comparison with experiment, and to the description of
the methods used.
Experiment
The broad goals of the experiment are to address the following
questions about the current state of the art in protein structure
prediction:
- Are the models produced similar to the corresponding
experimental structure?
- Are the models correctly aligned with the experimental
structures?
- Have similar structures that a model can be based on been
identified?
- Are the details of the models correct?
- Has there been progress between CASP2 and CASP3?
- What methods are most effective?
- Where can future effort must be productively be focused?
As in CASP2, all types of methods for predicting protein
structure will be considered. However, docking predictions will
not be included in CASP3, since a separate experiment is being
planned to evaluate this.
Collection of Prediction Targets
As before, for the experiment to succeed, it is essential that we
obtain the help of the experimental community. Therefore we invite
Protein crystallographers and NMR spectroscopists to
provide details of structures they expect to have made public before 1st
October 1998.
They are also asked to notify the organizers shortly before the structure
is solved.
Participation
Participation in the experiment is open to all. Predictors may form teams, and each team must have a designated
group leader. Except in special circumstances, and then by
agreement with the organizers, a predictor may only be a member
of a single team. Each team will be issued a unique ID number,
which will serve to identify their predictions. Those interested
in receiving mailings concerning progress of the experiment may
also register as 'observers' .
Prediction targets will be made available through
this web site.
All targets will be assigned an expiry date, and predictions must
be received and accepted before that expiration. Predictions may
be submitted through a web interface,
or by email. In order to be accepted, submissions must contain
all the required information, and conform to the specified
format. All accepted predictions will be given an accession
number, and this number will be considered to provide proof of
prediction submission. One team may submit up to five models of a target,
although most evaluation and assessment will focus on only one of these.
Assessment of Predictions
As in previous CASPs, independent assessors will evaluate the
predictions. There will be three assessors, representing
expertise in the comparative modeling, fold recognition and ab
initio prediction areas. Predictions will be assigned to
assessors on the basis of the level of detail and accuracy of the
models, as well as the methods used. Assessors will be provided
with the results of numerical evaluation of the predictions, and
will judge the results primarily on that basis. Numerical
evaluation criteria have been revised from those used in CASP2,
and are now being finalized, in consultation with the prediction
community. The assessors will be asked to focus particularly on
the effectiveness of different methods.
Release of Results
All predictions and prediction evaluations will be made available
through this web site, shortly before the meeting.
Principal Changes from CASP2
- No formal pre-division of targets in the categories of
comparative modeling, threading and ab initio. These distinctions
were blurred in CASP2. For example, alignments produced by
threading should be evaluated in the same way as those from
comparative modeling, and so called 'ab initio' methods are
increasingly relying on the structure knowledge base in a direct
manner, as in the 'mini-threading' methods.
Predictions will be assessed on the basis of the quality of the
models produced, as much as on the individual methods used.
However, there will still be three assessors, one with
responsibility of assessing methods that produce relatively
accurate and detailed models (nominally comparative modeling),
one for assessing aspects of models based on recognition of a
fold or parts of a fold, by threading or advanced sequence
comparison techniques (previously threading), and one for
assessing aspects of models based on ab initio methods. As
before, approximately one day of the meeting will be devoted to
each of these areas.
- More emphasis on identifying which methods are most effective.
Several mechanisms will be used for this:
- Predictors will be asked to provide long abstracts describing
their methods, and these will be available on the web in advance
of the meeting.
- Predictors will be encouraged to bring methods oriented
posters to the meeting.
- Assessors will be asked to take originality of methods into
account in choosing meeting speakers.
- Assessors will have more time to do their work, and thus more
time to extract messages about methods from the results.
- Revisions to numerical evaluation criteria. The numerical
criteria implemented for CASP2 are being revised. The most
extensive changes are to those used in the threading category,
and these have been assembled in consultation with a group of
veteran CASP threaders.
- Simplification and unification of prediction file formats. In
order to facilitate 'category free' predictions, a common core
format will be used for all predictions.
- More prediction targets. A limitation on the significance of
the CASP2 results, particularly in threading, was the relatively
small number of targets available. Every effort will be made
increase the stock for CASP3. Predictors are urged to help in
this process, and as always, we rely on the co-operation of our
experimental colleagues.
Meeting
A meeting will be held 13-17 December, 1998 at Asilomar, California,
USA to evaluate the results of the prediction experiment. The meeting
will be limited to about 170 participants and precedence will be given
to active predictors. It is anticipated that some financial assistance
will be available for the more successful predictors.
There will be lectures by the assessors and the more successful
predictors and a moderated discussion on the state of art in each
category. Emphasis will be on what went right and what went wrong, and
on what progress has been made since the last experiment. In
particular, in what areas further effort is likely to pay off in terms
of improved predictions. There will also be poster sessions and
informal sessions using computer workstations. Broadly one day will be
devoted to comparative modeling, fold recognition, and ab initio methods,
depending on the levels of response.
The proceedings of the meeting will be refereed and published in Proteins.
Timetable
- April '98 - August '98
- Distribution of targets to predictors as they become available,
withdrawal of structures as they are solved. Collection of
predictions.
- December 13 - 17 '98
- Meeting, Asilomar, California
Organizing Committee
John Moult CARB, University of Maryland, USA
Tim Hubbard Sanger Centre, Hinxton, UK
Jan Pedersen Acadia Pharmaceuticals, Denmark
Krzysztof Fidelis Lawrence Livermore National Laboratory, USA
One organizer will co-ordinate affairs in the each of the three
types of prediction: Jan Pedersen for comparative modeling, John
Moult for fold recognition, and Tim Hubbard for ab initio
prediction. As far as practical, the co-ordinators will consult
with veteran CASP participants in all aspects of the experiment.
Collection and distribution of targets, registration of
predictors, and collection and analysis of predictions will be
handled through the Prediction Center at Livermore, run by
Krzysztof Fidelis, Adam Zemla and Ceslovas Venclovas.
