Discussion: View Thread

Bart Craig asked:

> As someone involved
> with the development of 360 feedback instruments, I would be very
> interested to hear others' thoughts regarding how to extract meaningful
> information from such instruments in the context of less-than-perfect rater
> agreement.

If being used for developmental purposes, and time/money permitted,
you could consider a form of Delphi approach. Feed the ratings back
to the original raters, asking for their comments on why the rating
discrepancies might exist. It might throw light on the criteria being
used by the raters, and on the possibility of differential treatment
of the raters causing the differences.

--
Jeff Kennedy Phone: 64-3-325 2811
Economics & Marketing Department Fax: 64-3-325 3847
P.O. Box 84 E-mail: KENNEDJ@LINCOLN.AC.NZ
Lincoln University Canterbury
NEW ZEALAND

This is a good idea but.....

If subordinate feedback is to be honest and valid, don't most people
seek anonymity? In which case won't it be difficult to return their
rating?

(We used to often think that it would be funny to get a reply to an
anonymous letter!)

PHIL




Jeff Kennedy wrote:
>
> Bart Craig asked:
>
> > As someone involved
> > with the development of 360 feedback instruments, I would be very
> > interested to hear others' thoughts regarding how to extract meaningful
> > information from such instruments in the context of less-than-perfect rater
> > agreement.
>
> If being used for developmental purposes, and time/money permitted,
> you could consider a form of Delphi approach. Feed the ratings back
> to the original raters, asking for their comments on why the rating
> discrepancies might exist. It might throw light on the criteria being
> used by the raters, and on the possibility of differential treatment
> of the raters causing the differences.
>
> --
> Jeff Kennedy Phone: 64-3-325 2811
> Economics & Marketing Department Fax: 64-3-325 3847
> P.O. Box 84 E-mail: KENNEDJ@LINCOLN.AC.NZ
> Lincoln University Canterbury
> NEW ZEALAND

>------------------------------
>
>Date: Thu, 23 Oct 1997 06:58:12 +1000
>From: robnphil <robnphil@OZEMAIL.COM.AU>
>
>If subordinate feedback is to be honest and valid, don't most people
>seek anonymity? In which case won't it be difficult to return their
>rating?
[snip]

A very good point! In some (but not most) cases here at the Center, the
ratings are confidential, rather than anonymous, so we could potentially
follow up with raters individually, but it would be impractical to try
to involve the various raters in a dialogue with each other (which is
what is really needed), because the program participant usually
distributes the instruments, and the raters mail them to us; thus they
are frequently at scattered locations and don't know each other.
Additionally, we'd be talking about following up with a LOT of raters;
hundreds or thousands of instruments are scored per year (I'm really not
sure exactly how many), sometimes with as many as 10-15 raters per focal
manager.

>Bart

Jeff Kennedy wrote:
>
> Bart Craig asked:
>
> > As someone involved
> > with the development of 360 feedback instruments, I would be very
> > interested to hear others' thoughts regarding how to extract meaningful
> > information from such instruments in the context of less-than-perfect rater
> > agreement.
>
> If being used for developmental purposes, and time/money permitted,
> you could consider a form of Delphi approach. Feed the ratings back
> to the original raters, asking for their comments on why the rating
> discrepancies might exist. It might throw light on the criteria being
> used by the raters, and on the possibility of differential treatment
> of the raters causing the differences.
>
> --
> Jeff Kennedy Phone: 64-3-325 2811
> Economics & Marketing Department Fax: 64-3-325 3847
> P.O. Box 84 E-mail: KENNEDJ@LINCOLN.AC.NZ
> Lincoln University Canterbury
> NEW ZEALAND
Sounds like you have had experience with the main problem of many
feedback receivers when the responses are all over the place. This
tends to discredit the experience and value of 360 as a valid process.

Rater discrepancies can be more easily understood if the 360 feedback
instrument being used asks additional questions of the rater during the
completion of the survey. For instance "How sure are you of this
rating"? A similar point scale is provided for response. IN addition,
the opportunity to provide 'comments' after each item is answered, as
well as a general 'Comments Summary' at the end of the survey is
desirable. Refinements such as these help the feedback receiver to 'make
sense of the data'. Insrumens are available that are more sophisticated
than earlier iterations of the model. Please contact me if you are
interested in more information.
Nancy H. Haynes
New Horizons Consullting Group
e-mail: nhhaynes@uscom.com
fax: 1609 303-0073

>----------
>From: Nancy H. Haynes[SMTP:nhhaynes@USCOM.COM]
>Sent: Saturday, November 08, 1997 7:52 PM
>
>Rater discrepancies can be more easily understood if the 360 feedback
>instrument being used asks additional questions of the rater during the
>completion of the survey. For instance "How sure are you of this
>rating"?

A very interesting idea! I take it that you have actually used
instruments that employ such techniques... What do you do with the
certainty information? Just feed it back to the manager, along with
everything else? Or do you delete responses with low certainty ratings?
Or do you instruct the manager in how to interpret the ratings in the
context of their respective certainties? Have you evaluated how doing it
this way affects the feedback? Have you validated the certainty ratings
against any external criteria? For instance, does there tend to be lower
inter-rater agreement when mean certainty is lower?

Thanks!
Bart