Discussion: View Thread

Sandy, you didn't ask me (and I'll be interested in reading what
Glenn does also). But, I do about what you are thinking about. I
have ten factors (I'm not happy with all of them) that I ask each
group member to rate the other members on from 1 - 5. I require
specific comments on anything below a 2. There is a potential of 50
points. I add all of the rating for an individual and divide that by
the number of raters. I apply whatever percentage of 50 I get to the
group effort. I do this for both a group paper and the group effort
on a business simulation.

While it is true that some groups give each other all 5s, I don't see
this as a terrible thing. If the group has that kind of rapport and
worked well together, why not? In my experience, most groups do not
give each other all 5s. What I don't like about my method is that it
is possible that I give a 100 on a paper and no one gets 100 after
the adjustment for peer evaluation. Another problem I have run into
is that members may get " down" on a member who has a legitimate
problem, won't work with the member, then give that member low
ratings. I try to alleviate that by telling groups that, if they
have a problem with a group member, they should confront that group
member and discuss the problem. If that doesn't solve the problem,
they should come as a group to see me. On one occasion, after I
thought the group had worked out their problem the "self-appointed
leader" of the group still gave the member a bad rating and trashed
me on the class evaluation. Ah well, if it were easy, everyone would
be doing it I suppose.

Ralph W. Parrish
rparrish@lamar.colostate.edu

Ralph W. Parrish wrote:
>
> Sandy, you didn't ask me (and I'll be interested in reading what
> Glenn does also). But, I do about what you are thinking about. I
> have ten factors (I'm not happy with all of them) that I ask each
> group member to rate the other members on from 1 - 5. I require
> specific comments on anything below a 2. There is a potential of 50
> points. I add all of the rating for an individual and divide that by
> the number of raters. I apply whatever percentage of 50 I get to the
> group effort. I do this for both a group paper and the group effort
> on a business simulation.
>
> Ralph W. Parrish
> rparrish@lamar.colostate.edu

Ralph, how about sharing your instrument items with the rest of us. I
do something similar to what Sandy was suggesting, with the proviso that
whatever their rating, it be accompanied by a brief behavioral
justification, but it is a single rating per team member. I would like
to see the items you use, if you don't mind sharing.

Greg
--
Gregory K. Stephens, Ph.D.
Associate Professor of Management
Department of Management
Box 298530
Texas Christian University
Fort Worth, TX 76129

Telephone: (817) 921-7548
Fax: (817) 921-7227
Homepage: http://zeta.is.tcu.edu/~stephens

Our experience and the research seems to support using peer review as an
important component in performance assessment. There is an increasing trend
towards the use of peer review and one can glean from the reports of
schools and corporations who are using it that there are certain
qualifications, guidelines or cautions needed to make it work effectively
and not contribute to problems.

Some of this research is available in our annotated bibliography which can
be accessed online at http://www.islandnet.com/~rcarr/Docs.html

Here are a few ideas gained from our understanding of our own experience
and the research:

1. Criteria to be used by peers must be explicit, clear, and objective. In
other words, if a peer is to rate a colleague, the peer must be certain
that he or she knows exactly what is meant by an item in the rating. (Not
attending to this guideline often leaves people confused and can result in
unreliable ratings, including giving the maximum possible on each rating
regardless of actual performance.)

2. One way to insure the above is to increase the role that peers play in
developing the criteria upon which they will rate one another. This also
increases the relevance of the criteria and the value that the feedback
will have for the peers;

3. For peer review to contribute to a positive course experience,
particularly if the course or corporate goals stress teamwork and
cooperation (rather than create a competitive hostility and lack of trust,
for example), peers should have the opportunity to rate each criteria as to
its relevance and meaning for their work as they see it. In addition as
much openness as possible is desireable: those people who are involved in
being reviewed should have access to all data and where possible, anonymity
should be minimized.

4. Every effort should be made to open the peer review system to continuous
improvement by the peers involved. In other words rather than creating one
system for all time, encourage students to improve the peer review system,
rather than changing it from the top.

Hope these ideas are helpful to you. If you are planning on designing a
peer portfolio system, we would be happy to assist. You can also see the
results of a study we completed: Carr, R. (1977). The effects of specific
guidelines on the accuracy of student self-evaluation. Canadian Journal of
Education, 2, 4, 65-78.

Best wishes.

Rey Carr

***********************************************************************
Peer Resources is Canada's Leading Authority
on Peer Program Development and Mentoring for Schools and Business

Rey A. Carr, Ph.D. Tel: (250) 595-3503
1052 Davie Street Fax: (250) 595-3504
Victoria, British Columbia V8S 4E3 e-mail: rcarr@islandnet.com
Peer Resources: http://www.islandnet.com/~rcarr/peer.html
Directory of Healing Arts: http://www.islandnet.com/helpful/Index.html

Sandy and Ralph

Here is how I do my peer evaluations. I have five points that they have to
rate on a scale of 0-10 with usually 5 people per group. This gives 125
data points. The students then rank themselves and each other on these five
points with the potential to receive 50 points from each peer and
themselves. I then add the individual scores for each person to obtain a
score out of 250 ( for 5, it is 5 * 50). I then take the highest score in
the group and divide each group members score by that high score. This
allows me to calculate a percentage for each member of the group with the
[person with the highest score obviously receiving 100%. Each member's
percentage is then multiplied by the group grade to get an individual grade
for each member of the group. This is the grade that goes into the
computation of each student's final grade.

Glenn

At 02:49 PM 2/26/97 -0700, you wrote:
>Sandy, you didn't ask me (and I'll be interested in reading what
>Glenn does also). But, I do about what you are thinking about. I
>have ten factors (I'm not happy with all of them) that I ask each
>group member to rate the other members on from 1 - 5. I require
>specific comments on anything below a 2. There is a potential of 50
>points. I add all of the rating for an individual and divide that by
>the number of raters. I apply whatever percentage of 50 I get to the
>group effort. I do this for both a group paper and the group effort
>on a business simulation.
>
>While it is true that some groups give each other all 5s, I don't see
>this as a terrible thing. If the group has that kind of rapport and
>worked well together, why not? In my experience, most groups do not
>give each other all 5s. What I don't like about my method is that it
>is possible that I give a 100 on a paper and no one gets 100 after
>the adjustment for peer evaluation. Another problem I have run into
>is that members may get " down" on a member who has a legitimate
>problem, won't work with the member, then give that member low
>ratings. I try to alleviate that by telling groups that, if they
>have a problem with a group member, they should confront that group
>member and discuss the problem. If that doesn't solve the problem,
>they should come as a group to see me. On one occasion, after I
>thought the group had worked out their problem the "self-appointed
>leader" of the group still gave the member a bad rating and trashed
>me on the class evaluation. Ah well, if it were easy, everyone would
>be doing it I suppose.
>
>Ralph W. Parrish
>rparrish@lamar.colostate.edu
>

W. Glenn Rowe
Faculty of Business Administration
Memorial University of Newfoundland
St. John's, NF, Canada, A1B 3X5
709 737 4363
709 737 7999 (Fax)

ASAC 97 is being held in St. John's, Newfoundland, Canada from May 31 to
June 3, 1997. St. John's is the oldest city in North America and is only 8
miles west of Cape Spear, the most easterly point of North America. Come
join us at ASAC 97 and help us celebrate the 500th anniversary of the
arrival of John Cabot to Newfoundland in 1497.

Many of the teaching techniques outlined by Glenn Rowe (a man with
many years in the military) are similar to those I use in my
undergrad OB and HR classes, and are what I consider to be "best
practice."

This semester, students are presenting a team seminar - I will
be assessing two thirds of the work, while the students themselves
will have the opportunity to grade their team mates on their
performance in the preparation and presentation of the seminar.

What I would like to know is how Glenn designs "peer evaluations which
impact on the individual grades of the students". I had considered
simply averaging the team mark (a mark out of 5), but some of my
colleagues have argued that the team will often give all members 5/5,
which is not a true reflection of the contribution of individual team
members.

Any thoughts?

Sandy Millar
Manukau Institute of Technology

Ralph Parrish shared the following regarding student peer evaluations:

Gregory Stephens asked that I share the factors on my peer evaluation
form. Here they are:
DEPENDABILITY (Could this person be counted on to be at meetings
and/or handle assigned tasks?)
QUALITY OF WORK (Consider such things as accuracy and neatness.)
ATTITUDE (Did the member have a helpful, positive attitude?)
IDEAS (How well did the member do at contributing ideas?)
COOPERATION (Keep in mind that the member may have been cooperative
whether or not their ideas were used.)
OUTPUT (Amount of work done in comparison with other members.)
CASE KNOWLEDGE (General knowledge of the case OR the business
simulation.)
INTEGRITY(Consider the person's demonstrated honesty.) Note:This is one I
don't like but haven't come up with anything to replace it yet.
I WOULD LIKE TO WORK WITH THIS PERSON AGAIN


I feel these are too subjective. How can you measure "attitude"? Ideas? Cooperation? If you're going to evaluate someone, let's
practice what we preach in our management classes: do it on specific behaviors. For example, one of the seminars I teach is a
certification/train-the-trainer. In order to be certified to facilitate/train the materials they must show competency in a majority of
various behaviors. Some of the behaviors I look for in order to certify them are:

Delivery Techniques
 1. Demonstrates understanding.
 2. Rephrases or restates the concepts when necessary.
 3. Tailors the course emphasis and application to meet needs of the group.
 4. Summarizes and clarifies course content throughout the session.
 5. Provides effective transitions from one concept to another.
 6. Shares application examples of the concepts.
 7. Ties the course to other courses and issues.

Delivery Techniques
 8. Models course principles while facilitating.
 9. Listens empathically.
 10. Pulls data/examples from the group.
 11. Defers to the group.
 12. Uses media effectively (flipcharts, overheads, videos).
 13. Keeps airtime under 40%.
 14. Calls on people by name.
 15. Maintains eye contact.
 16. Gives verbal and nonverbal reinforcement.
 17. Asks open-ended questions.
 18. Counts to 10.
 19. Uses appropriate humor.
 20. Gives clear and concise instructions.

As can be seen from the list above, they either do it or they don't....it's very specific. Plus, participants know the behaviors before
they begin. Maybe you and the students could pull together your own evaluation at the beginning of the semester...look for specific
behaviors that relate to the topics at hand. For example, if you're teaching an OB class, and everything is done in teams, some of the
behaviors on which students evaluate peers might be:
- Comes to class prepared (all readings and prework complete)
- Maintains self-esteem of others during team meetings
- Listens empathically to team members
- Works toward win/win solutions with team members
- Seeks consensus
- Uses the problem solving model
- etc.


George Ferguson
georgeferguson@hmri.com

In partial response to your question (below) Denise:

I find that having students do a dry run midway through their project,
where they give each other feedback publicly within the group on all of the
criteria, is very effective. It helps each student to get real value
(instead of just a grade) from the feedback, while there is still time to
adjust. It also provides a context to bound the feedback in realism. This
feedback is **not** handed in to become part of the final grade.

The final feedback can be handled either confidentially or not. I don't
have an easy answer for the dilemma you present. I frankly think it is the
interim feedback which is most useful, and the final feedback is there
mostly to make the interim feedback really figural.

>..... I doubt students would say anything negative about their classmates
>if the evaluations were not confidential.
>
>However, given that the peer evaluations are a part of a student's grade, I
>also feel students should have the right to know how their final grade is
>determined. I do let students know their average score, based on how they
>were rated by the group, but I don't let them see the comments, or who gave
>them what rating.

>Denise Bane
>Assistant Professor of Management
>Baruch College, The City University of New York
>Denise-Bane@worldnet.att.net

Don Austin, Ph.D.
_____________________________________
Department of Organizational Behavior
Case Western Reserve University
Cleveland, OH 44106 (216) 932-8421

Regarding the discussion on peer evaluations in the classroom - I agree
with the comments by both Putzel and Coddington, and use peer evaluations
very effectively just as they seem to do. One technique that is slightly
different, but that I have found to be very useful, is to have the
students design their own peer evaluation in about the 3rd week of class.
Each group determines privately in an in-class meeting what the criteria
for
evalution should be. The only stipulation that I give them is that the
rating scale must be consistent across all groups -- I prescribe a 1 - 5
scale ranging from dismal performance to excellent performance. They can
not use fractions, and each team member must rate each peer and they must
rate themselves. I use the self-rating as a reality check, and
investigate cases where self-eval varies substantially from the average of
the peer's ratings.

The exercise of developing the peer rating criteria has turned into a
fabulous learning experience for the teams. It provides a forum for them
to really get into a discussion on what matters to each of them in terms
of a team mate, and to explore the differences in work preferences on the
front end. some teams get into serious debates over the specificity of the
criteria, over the specific behaviors etc. This really opens their eyes
early on to the challenge of managing diverse points of view in a team,
and allows them to engage in constructive debate over ideas before anyone
has strong reasons to get defensive about their own prior behavior. Some
times, teams really blow it and either fail to take the exercise
seriously, or they fail to really gain a consensus on the items. But this
also provides a great learning experience as many of them write in their
final papers.

One other benefit of this method is that it seems to provide a sense of
ownership of or investment in the team at an early stage. I do not
intervene in the process or outcome at all. I have had teams come up with
evals that are 2 items long, and others that come up with 27 items long,
and I indicate that either is okay with me -- that this is their criteria
for performance, their team expectations, that they own it, not me. But
that they must commit to and live with the final outcome.

After this exercise, they seem more prepared to do a mid-term
developmental feedback session with each other, an
think that their final peer evals have greater validity. Finally, this
exercise, more than any other, seem to alert them the the complexity of
the entire assessment process. . I am very pleased with the outcomes, and
thought you might like to try it.

One other thing -- I use the peer evaluations to weight their final
project grade, so it does have fairly serious reprecussions to their
overall course grade.


Susanne G. Scott
Assistant Professor of Management and Organization
University of Colorado at Colorado Springs
1420 Austin Bluffs Parkway
Colorado Springs, CO 80933-7150
719-262-3579
FAX: 719-590-1543
email: sscott@mail.uccs.edu

>Date: Fri, 12 Dec 1997 15:53:06 -0500
>From: Dr Harry J Bury <hbury@BW.EDU>
>Subject: Re: Peer Evaluation in the Classroom
>
>I really appreciate the thoughtful ideas about the pluses and minuses
>about peer evaluation in the class room. I am particularly struck with
>the comment that we expect the students to do what we as a faculty fail to
>do, namely give each other feedback on our performance. What I have been
>doing in courses which has helped the peer evaluation dynamic is to
>average the individual scores of everyone on the team and the average is
>the score for everyone, so if there are four people on a team and in their
>exercises, exams and so forth, two people get a 95 and two people get an
>85 everybody gets a 90 for the exam or exercise. Then what happens is the
>"better students" teach the others so that their grade improves.

I think this is a great idea, Harry!

Don Austin, Ph.D.
_____________________________________
Department of Organizational Behavior
Case Western Reserve University
Cleveland, OH 44106 (216) 932-8421
_____________________________________
Creating Appreciative Dialogue.

Researching how small groups
create valued organization.