Page 1 of 1

Thoughts when looking at the results

PostPosted: Tue Nov 30, 2010 2:36 am
by arneelof
Thoughts when looking at the results

When looking at the first set of "official" results from the predictioncenter I noticed one thing

Several "consensus/MQAP" based methods are actually performing better than the best server and actually only one (I think) non-consensus based method did.

This really puts strong arguments for that we as a community have a bigger impact than as individuals.

Unfortunately this is a difficult argument to sell to granting agencies.

yours

Arne

Re: Thoughts when looking at the results

PostPosted: Tue Nov 30, 2010 2:02 pm
by RAPTOR
To evaluate the real value added by the "consensus/MQAP" based methods or human groups, I think it is better to compare them with a naive clustering method instead of the best individual servers.

Re: Thoughts when looking at the results

PostPosted: Tue Nov 30, 2010 2:06 pm
by RAPTOR
To evaluate the real value added by the "consensus/MQAP" based methods or human groups, I think it is better to compare them with a simple clustering method instead of the best individual servers.
For example, we can have a simple clustering method by using 3D-Jury to rank all the server models.
A "consensus/MQAP" based method or human group adds real value to modeling if and only if it performs (statistically) significantly better than such a simple clustering method.

Re: Thoughts when looking at the results

PostPosted: Wed Dec 01, 2010 12:31 am
by arneelof
Pcons is a naive clustering method (basically identical to 3d-jury). It is clear that Pcomb (that is the method used in Elofsson) performs better in any measure that we tried. In earlier CASPs earlier versions of Pcomb did not perform better than the best individual server.

So perhaps we had some progress. On the other hand I think there are other MQAPs that perform at least as well as Pcomb

Re: Thoughts when looking at the results

PostPosted: Wed Dec 01, 2010 6:53 am
by RAPTOR
arneelof wrote:Pcons is a naive clustering method (basically identical to 3d-jury). It is clear that Pcomb (that is the method used in Elofsson) performs better in any measure that we tried. In earlier CASPs earlier versions of Pcomb did not perform better than the best individual server.

So perhaps we had some progress. On the other hand I think there are other MQAPs that perform at least as well as Pcomb



The performance of a consensus method depends on the performance of individual servers and also their correlation. If the individual servers are highly correlated or the best individual server is significantly better than the others, it is possible that a simple clustering method may not perform better than the best individual server. However, when there are a few pretty good and independent individual servers, a simple clustering method may perform better than the best individual server. Therefore, I think 3D-Jury instead of the best individual server may be a better control for benchmarking the "consensus/MQAP" methods.