>Date: Fri, 17 Oct 1997 01:57:40 -0400
>From: Rick Stamm <
TeamDoc@AOL.COM>
>Subject: Evaluating management training
>One of the greatest tools to come along in this area is 360 feedback (or
>muti-rater assessment).
[snip]
I agree that 360 feedback can be an enormously valuable tool. One of the
interesting issues associated with its use, though, is how to handle
disagreement among raters, either within or across rater groups. When,
say, subordinates report different perceptions than peers, it's a fairly
straightforward matter to advise the focal manager to "take a look" at
how/why this might be the case. But when raters within a group don't
agree, it seems less clear what the manager's "take away" message should
be. It's a common practice to report mean ratings from each group (e.g.,
"superiors, on average, rated you a 4 on this dimension"), but without
regard for within-group agreement, those means can end up not reflecting
the perceptions of even a single actual rater.
Obviously, the disagreement itself can be turned into feedback
("consider how you might be treating different individuals differently,
and how that affects their perceptions"). But that approach reduces the
various dimensions of the instrument down to one: differential
treatment. The psychological climate literature suggests calculating an
index of within-group agreement, such as rwg, and only reporting central
tendencies if that agreement index is sufficiently high. But that
strategy also threatens to discard potentially useful information. As
someone involved with the development of 360 feedback instruments, I
would be very interested to hear others' thoughts regarding how to
extract meaningful information from such instruments in the context of
less-than-perfect rater agreement.
Bart Craig
Center for Creative Leadership
Greensboro, NC