Re: test re test reliability

On 13 Apr 2005 03:19:07 -0700, popscan@xxxxxxxxxxx (matt) wrote:

> as i understand things, intra class correlation is being advocated as
> a more accurate expression of reliability over time as it also

I think it is fair to say that when you are *investigating*
reliability, you ought to be very clear in your own mind
about the effects of changes of mean, changes of variance,
and low correlation-as-standard r. You do that most elegantly,
in my opinion, by looking at those separately. You also,
in your investigating and developing, want to look at each
rater separately.

The IC compresses these concerns into one number. That
*can* be good for certain decisions. The IC for multiple
raters is good for planning for multiple raters. I don't think
of a good use for an IC across time... But, undeniably, some
reviewers and editors like to see an IC and possibly consider
any r as 'crude.'


> addresses changes in magnitude that sytematically affect a population,
> whereas the standard r will not recognise this.
>
> Is there any situation where the r statistic is more appropriate to
> use that the intra class correlation?
>

I think I addressed that, above. There might be some
relevant comments in my stats-FAQ, or you could
google groups, in sci.stat.* .

Here are other good comments on reliability -
http://ourworld.compuserve.com/homepages/jsuebersax/agree.htm


--
Rich Ulrich, wpilib@xxxxxxxx
http://www.pitt.edu/~wpilib/index.html
.

Relevant Pages

  • Re: kappa & ICC questions
    ... Can the data from more than one pairs of raters in an inter-rater ... reliability study ... There exists a creature called a multi-rater kappa. ... rated by each rater in the test-retest reliability study ranges from 9 ...
    (sci.stat.edu)
  • Re: interrater reliability
    ... >> I am wondering how to evaluate interrater reliability ... My study purpose is to evaluate the interrater ... >poor if the Subjects are nearly identical ... How often are the raters ...
    (sci.stat.math)
  • Re: intraclass correlation
    ... coefficients discussed in Shrout and Fleiss "Intraclass correlations: ... uses in assessing rater reliability," Psychological Bulletin, 1979, 86, ... coefficients for the ratings of n raters and also for the average ... variability and results in a much higher value of icc. ...
    (sci.stat.consult)
  • test re test reliability
    ... intra class correlation is being advocated as ... a more accurate expression of reliability over time as it also ... addresses changes in magnitude that sytematically affect a population, ...
    (sci.stat.edu)
  • Inter-rater reliability question
    ... If I have three columns of data representing three different raters, ... conduct an "inter-rater" reliability? ...
    (microsoft.public.excel.worksheet.functions)