Re: Ordering tasks by subjective difficulty

From: Richard Ulrich (Rich.Ulrich_at_comcast.net)
Date: 02/23/05 

Date: Wed, 23 Feb 2005 11:10:39 -0500

On 22 Feb 2005 18:30:12 -0800, petermichaux@yahoo.com wrote:

[snip, about rank ordering of tasks.]
>
> In general how do you take data from such a survey with many tasks and
> many people and order the difficulty of the tasks? Imagine that a few
> tasks are of similar difficulty and people order them differently. It
> gets confusing.
>
> Any help or direction to help much appreciated!

The best idea is probably to never collect information
of this kind as Ranks - since that permanently LOSES
all subjective information about the size of perceived
differences. Give the raters an "interval" rating scale.
Then you can make statements about the actual perceived
levels of difficulty, too.

The basic analytic tool *could* be an unbalanced blocks
design, ANOVA controlling for Subject. If you have to
start with ranks, then perform it with Ranks.

What judgement are you trying to make, anyway?

Is it enough to conclude that a few tasks "are of similar
difficulty" or do you need a winner? There is a
literature on Voting schemes which makes it clear that
ranking does not have to result in transitive decisions.
That is, you can end up with A>B>C>A . 

-- 
Rich Ulrich, wpilib@pitt.edu
http://www.pitt.edu/~wpilib/index.html