Re: comparing Kappa Statistics in case of dependence
- From: Richard Ulrich <Rich.Ulrich@xxxxxxxxxxx>
- Date: Tue, 15 Jan 2008 17:45:26 -0500
On Tue, 15 Jan 2008 02:36:00 -0800 (PST), Bart Hamers
<bart.hamers@xxxxxxxxx> wrote:
Hello,
In order to compare different raters, we often use the nonparametric
Kappa statistic as a measure of agreement.
However, we would like to compare different kappa statistics. Moreover
do we need a test to compare the kappa's in case of dependence.
Is there a test to compare Kappa(X,Y) and Kappa(Y,Z) taking into
account the dependence introduced by the mutual rater Y?
First, I hope that you are using 2x2 kappa and not
anything else, since r x k kappa is a fairly lousy statistic
for the larger tables, with a bad dependence on marginal
counts. Or, weighted kappa is practically equivalent to
using a correlation, so you might test for correlated
correlations.
Here is the simple situation for 3 raters, 0/1 ratings.
There are 8 possible outcomes of the 2x2x2 table:
Take the 1st column as the shared rater.
YXZ
000 - most cases not scored 111 (you hope)
001 outcome A: X agrees, Z disagrees with Y
010 outcome B: X disagrees, Z agrees with Y
011 - both disagree with Y
100 - both disagree with Y
101 outcome B
110 outcome A
111 - most cases not scored 000
If the raters are equivalent, the sum of A (Sa) will be
the same as the sum of B (Sb). This is the information
that you have on DIFFERENCES between the kappas,
and for the differences between the raters,
ignoring the level-differences in how many 0s /1s
were given.
A simple test on Sz vs Sb is the sign test, like using McNemar's
test on the off-diagonal scores for testing "change."
One reason for looking at this is that it may show
you that you have very little power for comparing
your ratings. A conclusion of "not different" is not very
powerful until the count of disagreements is large.
I haven't looked at the citation that Ray provided.
"Weighted least squares" is obviously more complicated.
--
Rich Ulrich
http://www.pitt.edu/~wpilib/index.html
.
- References:
- comparing Kappa Statistics in case of dependence
- From: Bart Hamers
- comparing Kappa Statistics in case of dependence
- Prev by Date: Re: Math of Expected Values
- Next by Date: Trying to contact group moderator
- Previous by thread: Re: comparing Kappa Statistics in case of dependence
- Next by thread: probability question
- Index(es):
Relevant Pages
|
