Thoughts when looking at the results

Thoughts when looking at the results

Postby arneelof on Tue Nov 30, 2010 2:36 am

Thoughts when looking at the results

When looking at the first set of "official" results from the predictioncenter I noticed one thing

Several "consensus/MQAP" based methods are actually performing better than the best server and actually only one (I think) non-consensus based method did.

This really puts strong arguments for that we as a community have a bigger impact than as individuals.

Unfortunately this is a difficult argument to sell to granting agencies.

yours

Arne
arneelof
 
Posts: 10
Joined: Mon Mar 29, 2010 10:40 pm

Re: Thoughts when looking at the results

Postby RAPTOR on Tue Nov 30, 2010 2:02 pm

To evaluate the real value added by the "consensus/MQAP" based methods or human groups, I think it is better to compare them with a naive clustering method instead of the best individual servers.
RAPTOR
 
Posts: 6
Joined: Tue Sep 09, 2008 6:47 pm

Re: Thoughts when looking at the results

Postby RAPTOR on Tue Nov 30, 2010 2:06 pm

To evaluate the real value added by the "consensus/MQAP" based methods or human groups, I think it is better to compare them with a simple clustering method instead of the best individual servers.
For example, we can have a simple clustering method by using 3D-Jury to rank all the server models.
A "consensus/MQAP" based method or human group adds real value to modeling if and only if it performs (statistically) significantly better than such a simple clustering method.
RAPTOR
 
Posts: 6
Joined: Tue Sep 09, 2008 6:47 pm

Re: Thoughts when looking at the results

Postby arneelof on Wed Dec 01, 2010 12:31 am

Pcons is a naive clustering method (basically identical to 3d-jury). It is clear that Pcomb (that is the method used in Elofsson) performs better in any measure that we tried. In earlier CASPs earlier versions of Pcomb did not perform better than the best individual server.

So perhaps we had some progress. On the other hand I think there are other MQAPs that perform at least as well as Pcomb
arneelof
 
Posts: 10
Joined: Mon Mar 29, 2010 10:40 pm

Re: Thoughts when looking at the results

Postby RAPTOR on Wed Dec 01, 2010 6:53 am

arneelof wrote:Pcons is a naive clustering method (basically identical to 3d-jury). It is clear that Pcomb (that is the method used in Elofsson) performs better in any measure that we tried. In earlier CASPs earlier versions of Pcomb did not perform better than the best individual server.

So perhaps we had some progress. On the other hand I think there are other MQAPs that perform at least as well as Pcomb



The performance of a consensus method depends on the performance of individual servers and also their correlation. If the individual servers are highly correlated or the best individual server is significantly better than the others, it is possible that a simple clustering method may not perform better than the best individual server. However, when there are a few pretty good and independent individual servers, a simple clustering method may perform better than the best individual server. Therefore, I think 3D-Jury instead of the best individual server may be a better control for benchmarking the "consensus/MQAP" methods.
RAPTOR
 
Posts: 6
Joined: Tue Sep 09, 2008 6:47 pm


Return to CASP Discussion

Who is online

Users browsing this forum: No registered users and 5 guests

cron