Re: Quadratic weighted Kappa and the Intraclass Correlation Coefficient

On Nov 5, 8:52 am, John Uebersax <jsueber...@xxxxxxxxx> wrote:
Hello Greg,

First, there are two weighting systems for weighted kappa with ordinal
ratings -- Fleiss-Cohen weights and Cicchetti-Allison weights.  Only
the former produce the equivalence of weighted kappa and an ICC.

There are basically 3 kinds of ICC:

Case 1 Raters for each subject are selected at random
Case 2 The same raters rate each case. These are a random sample.
Case 3 The same raters rate each case. These are the only raters.

For each, there is an ICC formula for (a) the reliability of a single
rating and (b) the reliability of a mean of k ratings.  For more
information on the various kinds of ICC see:

http://luna.cas.usf.edu/~mbrannic/files/pmet/shrout1.htm

and/or the paper:

McGraw KO, Wong SP. Forming inferences about some intraclass
correlations. Psychological Methods 1996;1:30-46.

Offhand I would expect that weighted kappa corresponds to Case 3a.
But I might be wrong and naturally you'd want to verify this with
actual data.

If you'd like to send me an example dataset, I could compare results
of weighted kappa with SAS (which has an option for Fleiss-Cohen
weights) and various programs for estimating the ICC.

John Uebersax PhD

Thanks,

I'd like to take you up on that offer.
What's the best format?
Tab delimited ASCII? Excel? SPSS? (Afraid I don't have SAS)

Jenny
.

Relevant Pages

  • Re: Quadratic weighted Kappa and the Intraclass Correlation Coefficient
    ... there are two weighting systems for weighted kappa with ordinal ... There are basically 3 kinds of ICC: ... Case 2 The same raters rate each case. ... rating and the reliability of a mean of k ratings. ...
    (sci.stat.edu)
  • Re: Quadratic weighted Kappa and the Intraclass Correlation Coefficient
    ... A comma-delimited ASCII file or else an Excel worksheet would be fine. ... the former produce the equivalence of weighted kappa and an ICC. ...
    (sci.stat.edu)
  • Re: kappa & ICC questions
    ... Poor, Fair, Good, Very Good (corresponding 1,2,3,4 response choices ... I believe that any time one can use weighted kappa with ordinal ... ratings one can also use the ICC (and, since the ICC is more flexible, ... in theory one could scale the rating categories ...
    (sci.stat.edu)
  • ICC for 2x2 Table
    ... Let two raters rate N subjects on a binary measure, ... Is there a computational formula for the intraclass correlation (ICC) ... raters rate each case) perhaps better would be to summarize data as ... , where n_i denotes the number of positive ratings ...
    (sci.stat.math)