Preliminary results

Preliminary results

Postby mcguffin on Mon Aug 04, 2008 6:30 am

Hi,

I've calculated some preliminary results for CASP8, based on 46 targets with released structures:

http://www.reading.ac.uk/bioinf/CASP8/

So far it looks as though the Zhang-Server will remain the best server in the TS category, by a significant margin. The only new server that is close in performance is the Phyre_de_novo server:

Cumulative results: http://www.reading.ac.uk/bioinf/CASP8/CASP8_results_TS1_46_targets.html
Significance tests: http://www.reading.ac.uk/bioinf/CASP8/R_wilcox_CASP8_results_46_targets_highlighted.html

Yang Zhang has also published results on his website:

http://zhang.bioinformatics.ku.edu/casp8/index.html

Liam
mcguffin
 
Posts: 12
Joined: Mon Aug 04, 2008 6:12 am

Re: Preliminary results

Postby mcguffin on Thu Aug 07, 2008 6:26 am

Results for 54 targets are now available.

Check here for updates:
http://www.reading.ac.uk/bioinf/CASP8/index.html
mcguffin
 
Posts: 12
Joined: Mon Aug 04, 2008 6:12 am

Re: Preliminary results

Postby kevin_karplus on Thu Aug 28, 2008 11:36 am

I like Zhang's assessment by TM_score and HB_score (perhaps because it puts my server second, right behind Zhang's). It seems that Zhang fixed the problem in CASP7 of bad models built on good CA traces, if he is now doing best at the Hbonds.
kevin_karplus
 
Posts: 7
Joined: Tue Jul 22, 2008 10:05 am

Re: Preliminary results

Postby Guest on Wed Sep 03, 2008 2:46 am

Whichever score we use, we need to assess the significance of the differences between methods based on common subsets. Slight differences in cumulative scores may be insignificant.

Based on the GDT-TS score, Zhang's server significantly out performs all others. However, there is no significant difference between the next 10 methods (p >= 0.01), so the rankings based on cumulative score may be due to chance.

For example, there are only two servers that significantly outperform the SAM-T08-server, which are the Zhang-Server and pro-sp3-TASSER. Follow the link below:

http://www.reading.ac.uk/bioinf/CASP8/R_wilcox_CASP8_results_93_targets_highlighted.html
Guest
 

Re: Preliminary results

Postby mcguffin on Wed Sep 03, 2008 2:52 am

BTW the last post was me. I didn't realise you could post without logging in...

Liam McGuffin
mcguffin
 
Posts: 12
Joined: Mon Aug 04, 2008 6:12 am

Re: Preliminary results

Postby RAPTOR on Thu Sep 04, 2008 1:19 pm

I agree that there is no big difference between #2 and #10.

It also depends on how you calculate and interpret the "significance". Given two servers A and B, if A is better than B by 0.001 for each target, do you
think A is significantly better than B?


Guest wrote:Whichever score we use, we need to assess the significance of the differences between methods based on common subsets. Slight differences in cumulative scores may be insignificant.

Based on the GDT-TS score, Zhang's server significantly out performs all others. However, there is no significant difference between the next 10 methods (p >= 0.01), so the rankings based on cumulative score may be due to chance.

For example, there are only two servers that significantly outperform the SAM-T08-server, which are the Zhang-Server and pro-sp3-TASSER. Follow the link below:

http://www.reading.ac.uk/bioinf/CASP8/R_wilcox_CASP8_results_93_targets_highlighted.html
RAPTOR
 

Re: Preliminary results

Postby guest on Fri Sep 05, 2008 2:44 am

kevin_karplus wrote:I like Zhang's assessment by TM_score and HB_score (perhaps because it puts my server second, right behind Zhang's). It seems that Zhang fixed the problem in CASP7 of bad models built on good CA traces, if he is now doing best at the Hbonds.



I think HB_score should be used only on prediction of a "new category", say "H-bond prediction" , just like side-chain modeling, not on
traditional 3D-structure prediction which most focused on.
guest
 

Re: Preliminary results

Postby Guest on Fri Sep 05, 2008 9:52 am

guest wrote:
kevin_karplus wrote:I like Zhang's assessment by TM_score and HB_score (perhaps because it puts my server second, right behind Zhang's). It seems that Zhang fixed the problem in CASP7 of bad models built on good CA traces, if he is now doing best at the Hbonds.



I think HB_score should be used only on prediction of a "new category", say "H-bond prediction" , just like side-chain modeling, not on
traditional 3D-structure prediction which most focused on.


The CASP7 assessors ranked groups by HB and GDT. It's time for CASPs to set up a somewhat consistent criterion.
Guest
 

Re: Preliminary results

Postby jlee on Sat Sep 06, 2008 5:25 am

In the CASP7, to assess the quality of C_alpha trace, GDT-HA was used for both TBM and HA-TBM targets.
I would like to see the CASP8 assessors to use a even higher-accuracy measure such as GDT-TL (0.25, 0.5 1.0 2.0)
especially for HA-TBM tagets as done in the CAST6. Nowadays, protein model quality is improving steadily
especially for TBM targets, and CASP should ask/encourage predictors to devise more accurate modeling
globally (for FM targets) as well as locally (for TBM targets). I feel like 8A is too large a distance to be
meaningful even for FM targets (however, 8A gives us a complacent feeling of good protein modeling)

On the other hand, for the calculation of GDT scores, only positions of C_alpha atoms matter.
Since there are many more non-C_alpha atoms in protein models (CASP8 did not accept C_alpha only models),
CASP8 assessors should consider to include additional measures other than the HB score used in the CASP7.
Candidate measures include Chi_1 and Chi_12 for all/TBM targets.
One should also consider to use the HB score for all targets not restricted to TBM.

Just a thought....
jlee
 

Re: Preliminary results

Postby test on Sat Sep 06, 2008 9:23 am

Well, the definition of TBM and FM are subjective instead of objective. How to implement what you suggested without introducing too much artificial bias?
In addition, GDT-TL may be too strict and is likely to bury some subtle difference.

jlee wrote:In the CASP7, to assess the quality of C_alpha trace, GDT-HA was used for both TBM and HA-TBM targets.
I would like to see the CASP8 assessors to use a even higher-accuracy measure such as GDT-TL (0.25, 0.5 1.0 2.0)
especially for HA-TBM tagets as done in the CAST6. Nowadays, protein model quality is improving steadily
especially for TBM targets, and CASP should ask/encourage predictors to devise more accurate modeling
globally (for FM targets) as well as locally (for TBM targets). I feel like 8A is too large a distance to be
meaningful even for FM targets (however, 8A gives us a complacent feeling of good protein modeling)

On the other hand, for the calculation of GDT scores, only positions of C_alpha atoms matter.
Since there are many more non-C_alpha atoms in protein models (CASP8 did not accept C_alpha only models),
CASP8 assessors should consider to include additional measures other than the HB score used in the CASP7.
Candidate measures include Chi_1 and Chi_12 for all/TBM targets.
One should also consider to use the HB score for all targets not restricted to TBM.

Just a thought....
test
 

Next

Return to CASP Discussion

Who is online

Users browsing this forum: No registered users and 1 guest

cron