Re: interrater reliability
From: Kun Nie (kun_nie_at_yahoo.com)
Date: 02/03/05
- Next message: Richard Ulrich: "Re: Help with simple statistical test"
- Previous message: Hot Robot Monkey Sex: "Re: Help with simple statistical test"
- In reply to: Richard Ulrich: "Re: interrater reliability"
- Next in thread: Richard Ulrich: "Re: interrater reliability"
- Reply: Richard Ulrich: "Re: interrater reliability"
- Messages sorted by: [ date ] [ thread ]
Date: Thu, 3 Feb 2005 22:00:02 +0000 (UTC)
Thank you for your information! I also think we should have more than
one subject to make inference in a population. Could you help us to
find a statistical approach for this? We have both ordinal data and
continuous data. I appreciate your assistance!!
On Thu, 03 Feb 2005 10:40:04 -0500, Richard Ulrich wrote:
>On Thu, 3 Feb 2005 15:00:25 +0000 (UTC), kun_nie@yahoo.com (Kun Nie)
>wrote:
>
>> Dear colleagues,
>> I am wondering how to evaluate interrater reliability
>> coefficient. My study purpose is to evaluate the interrater
>> reliability for a new questionaire. In a start-up meeting for this
>> study, a few clinicians were trained to evaluate the responses of a
>> subject by watching the same vedio. My data look like:
>>
>> evaluater item_1 ... item_k total_score
>> 1 X ... X X
>> 2 X ... X X
>> .
>> .
>> .
>> n X ... X X
>>
>> where evaluaters are those clinicians, item_1 to item_k are
different
>> questions in the questionaire, X stands for the response, and
>> total-score is the sum of item_1 to item_k. We have only one suject
in
>
>Only one subject?
>Okay, that ruins everything.
>
>"Conventional reliability" estimates how well a test works,
>using a sample from some specific population
>
>Basically, the conventional statistics are a re-statement
>of the ANOVA that tells how well the raters can
>distinguish between Subjects. So, 'reliability' will look
>poor (and, be poor) if the Subjects are nearly identical
>on what is being measured.
>
>A sample of 1? There's no test, so there's no possible
>computation.
>
>
>> this study and this subject was evaluated by n evaluaters. item_1
to
>> item_k are ordinal data (e.g.,0=not at all, 1=sometimes, 2=always,
>> etc.
>> Our purposes are:
>> 1. evaluate the interrater reliability for each item;
>> 2. evaluate the interrater reliability for total_score.
>> Can anybody give me some suggestions on how to get the
interrater
>> reliability coefficient? Thank you very much in advance!
>
>You can describe some versions of consistency.
>How much of the scale is used? - that depends, of course,
>on what the Subject looked like. How often are the raters
>the same? - that is what you can describe, but it will be
>won't have any outside reference unless the readers bring
>their own knowledge of what range of scores might be
>expected, for some sample.
>
>You will not use Cronbach's alpha, or anything called an ICC.
>
>--
>Rich Ulrich, wpilib@pitt.edu
><a
href="http://www.pitt.edu/~wpilib/index.html">http://www.pitt.edu/~wpilib/index.html>
Relevant Pages
|
