ITRG evaluation: CSC400, Fall 2004


This table gives the results of various methods of ranking the
proposals. I have listed the results of 3 ways of evaluating the
individual votes: Total first place votes, total votes of any sort,
and a weighted sum where 1st place gets 3 points, 2nd place 2 points,
3rd 1 point.
I have also given the rank of the top few in each category.
Note that the two methods of combining multiple votes agree fairly
well in picking the top few (though the exact ordering differs).
The first-place votes seem to have a sparsity problem.

The panel evaluation differs significantly from the voting results.
In some cases, proposals that were ranked highly by the individual voting
schemes were rated non-competitive by the panel. In others, proposals ranked
fairly low by the individual voting, were ranked competitively by the panel.
Clearly, discussion plays a significant role. This seems to be a general
phenomenon. Aware of this difference, NSF has decided that the results
involving panel discussion are a better method of evaluation than combining
individual reviews, and consequently make most awards through some sort of
panel process (although individual reviews are sometimes solicited from
non-panel members and considered byt the panel).


Proposal     1st Place   Total     Weighted   Panel
             votes       votes     sum        eval
             (rank)      (rank)    (rank) 

-------------------------------------------------------------------------------

Balazar      0           2         3          C    (RFID in Cars)

Balu         1 (2-6)     3 (6-7)   6 (5-7)    C    (Security Motes)

Choudhury    0           4 (3-5)   6 (5-7)    NC   (Social Robots)

Faliszewski  1 (2-6)     4 (3-5)   8 (2-3)    HC   (Data Immunity)

Liu          0           0         0          C    (NL Translation)

Lu           0           1         1          C    (Home Security Network)

Messing      0           0         0          C    (Integrated Tracking)

Mittal       1 (2-6)     2         5          HC   (Image-based Search)

Post         1 (2-6)     4 (3-5)   8 (2-3)    HC   (Distributed Human Work)

Privman      0           5 (2)     7 (4)      NC   (Laser Satellites)

Shriraman    0           2         3          C    (Memory in Processor)

Spear        9 (1)       12 (1)    32 (1)     HC   (Reconfigurable Trains)

Tan          0           0         0          NC   (AI for OS)

VanDurme     1 (2-6)     3 (6-7)   6 (5-7)    C    (Data Mining for Security)