Preliminary results

[mcguffin]

Hi,

I've calculated some preliminary results for CASP8, based on 46 targets with released structures:

http://www.reading.ac.uk/bioinf/CASP8/

So far it looks as though the Zhang-Server will remain the best server in the TS category, by a significant margin. The only new server that is close in performance is the Phyre_de_novo server:

Cumulative results: http://www.reading.ac.uk/bioinf/CASP8/CASP8_results_TS1_46_targets.html
Significance tests: http://www.reading.ac.uk/bioinf/CASP8/R_wilcox_CASP8_results_46_targets_highlighted.html

Yang Zhang has also published results on his website:

http://zhang.bioinformatics.ku.edu/casp8/index.html

Liam

[mcguffin]

Results for 54 targets are now available.

Check here for updates:
http://www.reading.ac.uk/bioinf/CASP8/index.html

[kevin_karplus]

I like Zhang's assessment by TM_score and HB_score (perhaps because it puts my server second, right behind Zhang's). It seems that Zhang fixed the problem in CASP7 of bad models built on good CA traces, if he is now doing best at the Hbonds.

[Guest]

Whichever score we use, we need to assess the significance of the differences between methods based on common subsets. Slight differences in cumulative scores may be insignificant.

Based on the GDT-TS score, Zhang's server significantly out performs all others. However, there is no significant difference between the next 10 methods (p >= 0.01), so the rankings based on cumulative score may be due to chance.

For example, there are only two servers that significantly outperform the SAM-T08-server, which are the Zhang-Server and pro-sp3-TASSER. Follow the link below:

http://www.reading.ac.uk/bioinf/CASP8/R_wilcox_CASP8_results_93_targets_highlighted.html

[mcguffin]

BTW the last post was me. I didn't realise you could post without logging in...

Liam McGuffin

[RAPTOR]

I agree that there is no big difference between #2 and #10.

It also depends on how you calculate and interpret the "significance". Given two servers A and B, if A is better than B by 0.001 for each target, do you
think A is significantly better than B?

Whichever score we use, we need to assess the significance of the differences between methods based on common subsets. Slight differences in cumulative scores may be insignificant.

Based on the GDT-TS score, Zhang's server significantly out performs all others. However, there is no significant difference between the next 10 methods (p >= 0.01), so the rankings based on cumulative score may be due to chance.

For example, there are only two servers that significantly outperform the SAM-T08-server, which are the Zhang-Server and pro-sp3-TASSER. Follow the link below:

http://www.reading.ac.uk/bioinf/CASP8/R_wilcox_CASP8_results_93_targets_highlighted.html

[guest]

I like Zhang's assessment by TM_score and HB_score (perhaps because it puts my server second, right behind Zhang's). It seems that Zhang fixed the problem in CASP7 of bad models built on good CA traces, if he is now doing best at the Hbonds.

I think HB_score should be used only on prediction of a "new category", say "H-bond prediction" , just like side-chain modeling, not on
traditional 3D-structure prediction which most focused on.

[Guest]

I like Zhang's assessment by TM_score and HB_score (perhaps because it puts my server second, right behind Zhang's). It seems that Zhang fixed the problem in CASP7 of bad models built on good CA traces, if he is now doing best at the Hbonds.

I think HB_score should be used only on prediction of a "new category", say "H-bond prediction" , just like side-chain modeling, not on
traditional 3D-structure prediction which most focused on.

The CASP7 assessors ranked groups by HB and GDT. It's time for CASPs to set up a somewhat consistent criterion.

[jlee]

In the CASP7, to assess the quality of C_alpha trace, GDT-HA was used for both TBM and HA-TBM targets.
I would like to see the CASP8 assessors to use a even higher-accuracy measure such as GDT-TL (0.25, 0.5 1.0 2.0)
especially for HA-TBM tagets as done in the CAST6. Nowadays, protein model quality is improving steadily
especially for TBM targets, and CASP should ask/encourage predictors to devise more accurate modeling
globally (for FM targets) as well as locally (for TBM targets). I feel like 8A is too large a distance to be
meaningful even for FM targets (however, 8A gives us a complacent feeling of good protein modeling)

On the other hand, for the calculation of GDT scores, only positions of C_alpha atoms matter.
Since there are many more non-C_alpha atoms in protein models (CASP8 did not accept C_alpha only models),
CASP8 assessors should consider to include additional measures other than the HB score used in the CASP7.
Candidate measures include Chi_1 and Chi_12 for all/TBM targets.
One should also consider to use the HB score for all targets not restricted to TBM.

Just a thought....

[test]

Well, the definition of TBM and FM are subjective instead of objective. How to implement what you suggested without introducing too much artificial bias?
In addition, GDT-TL may be too strict and is likely to bury some subtle difference.

In the CASP7, to assess the quality of C_alpha trace, GDT-HA was used for both TBM and HA-TBM targets.
I would like to see the CASP8 assessors to use a even higher-accuracy measure such as GDT-TL (0.25, 0.5 1.0 2.0)
especially for HA-TBM tagets as done in the CAST6. Nowadays, protein model quality is improving steadily
especially for TBM targets, and CASP should ask/encourage predictors to devise more accurate modeling
globally (for FM targets) as well as locally (for TBM targets). I feel like 8A is too large a distance to be
meaningful even for FM targets (however, 8A gives us a complacent feeling of good protein modeling)

On the other hand, for the calculation of GDT scores, only positions of C_alpha atoms matter.
Since there are many more non-C_alpha atoms in protein models (CASP8 did not accept C_alpha only models),
CASP8 assessors should consider to include additional measures other than the HB score used in the CASP7.
Candidate measures include Chi_1 and Chi_12 for all/TBM targets.
One should also consider to use the HB score for all targets not restricted to TBM.

Just a thought....