Discussion: View Thread

On 9 Oct 97 at 12:35, Ray Rasmussen wrote:


> It's interesting to me to see that the debate on tests for screening
> usually follows the line of how high the correlations are between test
> results and subsequent performance, often as measured by a variable like
> supervisory ratings.
>
> I've not seen a discussion of the less measurable impacts and wonder
> whether any of you folks have.

Interesting line of thinking...and thought provoking. There may be
some other corollaries here to consider.

If we acknowledge that workplaces are in a state of flux or change,
and the staff requirements change as they go, then selection (IF it
is effective) can only consider "fit" for the present. Clearly one
cannot measure the future needs in an objective way, although one can
speculate. An extension then (if we rule out development and
training) is that if/when job requirements change, that we turf out
our experienced staff, and re-select new ones. In a more knowledge
based economy this is clearly a problem and a waste of resources.

One aspect of selection is the contention that those selected and
promoted are similar to those that are selecting and promoting, in a
number of ways, including race, approach, etc. It seems sensible to
realize that when one selects for a "job fit", one is going to be
perpetuating the current state of the organization, for better or
worse, fixing it in time. My view is that organizations NEED some bad
fits, new blood, and the old blood is not likely to "select" it using
job fit measures.

If we look at selection and job fit, what we see is the setting of
barriers, particularly to management levels, to the extent that the
measures seek to select managers who fit the white, male,
middle/upper class values, ethics and style. In my view the more we
talk about job fit, the more likely we are to make things like the
glass ceiling thicker and thicker.



Robert Bacal, Inst.For Cooperative Communication, rbacal@escape.ca
Visit our Resource Centre for articles on mgmt.,training,communication, and defusing hostility
at http://www.escape.ca/~rbacal (204) 888-9290
*Site Last Updated On Sept.11, 1997*

Warren, I agree with most of your points, so have snipped them for
space reasons.

On 9 Oct 97 at 16:04, Warren Bobrow wrote:

> Just because your student is ignorant of industrial psychology research
> methods is no reason to question the field. If your student was in a field
> where people research employment tests (such as industrial psychology) s/he
> would have no problem identifying a room full of research that documents the
> validity (regression coefficients) of a large variety of pre-employment
> tests. As well as research that demonstrates the return on investment of
> such tests.

There is a point to be made that borders on the
academician/practitioner issue. While there may be research (and no
doubt there is), the question is whether practitioners are ABLE to
understand the meaning, assess the research designs, and choose the
instruments that are most appropriate, and then USE it appropriately.
This isn't a knock on industrial psychology, but is a comment on what
I see as a real problem: the use of psychological tests by
unqualified persons.

There are many psychological tests, that, I believe, CANNOT be
administered by anyone other than a licensed professional (I recall
the MMPI as one). There is a reason for this, of course, and that is
that in psychology, as I am sure you well know, no test is to be used
as a sole basis for assessment or diagnosis--the clinical aspect of
testing IS important...as it is in diagnosis.

We have many people selling tests, and many people using "packages",
and I seriously doubt that they are all adequately trained in their
use...companies are franchising and licensing people from all strange
walks of life to be quasi-psychological assessors. Many couldn't
critique a research design if their lives depended on it.

I have a set of horror stories connected with inappropriate use of
the MBTI for example, where people "certified" by the parent company,
paid for their certification, and had no idea what they were doing.
Again, this isn't a comment on the instrument per se, but the push to
market, and the notion that anyone with two days of training can
assess human beings properly, and communicate the results.

So, Ray's comment, while not about testing per se IS about the
application of testing by non-academically trained folks or badly
trained folks. Testing is simply seductive in nature, for
psychological reasons I am sure you are aware of. For that reason it
is critically important that tests be used properly.




Robert Bacal, Inst.For Cooperative Communication, rbacal@escape.ca
Visit our Resource Centre for articles on mgmt.,training,communication, and defusing hostility
at http://www.escape.ca/~rbacal (204) 888-9290
*Site Last Updated On Sept.11, 1997*

At 12:35 PM 10/9/97 -0700, you wrote:
<snip snip>
>
>I don't want to diminish these tests completely. I think that they could be
>put to good use .. for example, the ones that I've looked at can cause some
>interesting soul searching ... most are based on models of human
>interaction or models of the human psyche that could enhance
>self-awareness. Suppose that instead of using these tests to screen, we
>allowed employees to take them and to use them as food for thought ... to
>try to ask themselves whether there is information contained in them that
>could help them become better problem solvers, better interactors, better
>leaders, etc. Of course, such a practice, would have to be disconnected
>from job screening, promotion screening, performance appraisal. It would
>truly have to be a tool for self awareness.
>

This is very much how we have used psych. tests with employees. As part of
our contract with the employee and the employer, we agree to review the
results of the assessment with the candidate/employee *prior* to any
discussions/presentation to the employer. We emphasise that the purpose of
the assessment is to detail strengths and the "kind of mistakes" that the
person make in difficult/stressful situations. The candidate has an
opportunity to review this with the understanding that the report will
reflect what they believe about themselves.

We discourage the use of these tests as the sole or singularly important
aspect of selection.


______________________________________

Ivan Zendel, Ph.D., C.Psych.
Paradigm Solutions - Creating Change
#420, 910-7th Ave. SW
Calgary, Alberta, Canada, T2P 3N8
Phone: (403)264-4442 Fax:(403)264-4447
______________________________________

http://www.cadvision.com/parasol/

It's interesting to me to see that the debate on tests for screening
usually follows the line of how high the correlations are between test
results and subsequent performance, often as measured by a variable like
supervisory ratings.

I've not seen a discussion of the less measurable impacts and wonder
whether any of you folks have.

For example, I think that the interviewing/selection process is the initial
stage of a psychological contract that implicitly, if not explicitly,
defines the expectations of the employee. In addition, I think that a
device such as a pencil-and-paper test can have psychological consequences
that need to be considered.

Here's the way I see it:

The employee is in a trap, especially in a job scarce market s/he has no
choice of refusal.

S/he will be tempted to answer the test in the right way ... thus, there's
a motivation to lie. Even when given a test and supplied with an answer
key, my students tell me that they simply can't keep themselves from
answering the way the test demands [of course they may be wrong about
what's demanded]

The test-makers, of course, will screen for social desirability in the
response patters ... a complication within the test's reliability and
validity ... that is often not considered when promoting the usefulness of
the test.

The employee MAY feel demeaned. This organization is telling the testtaker
that it can know about who s/he is by his/her responses to their test.
It's like being hooked up to a lie detector.

The employee MAY buy in and yield to the power of tests -- thus, becomeing
more other directed or externally located with respect to feelings of
self-control/mastery. S/he MAY actually believe that the test-maker and
administrator can know more about him/her than s/he knows. That the rich,
inner experience that constitutes being human can be broken down into a set
of numbers and used to place people in jobs, careers, etc.

Along these lines, I notice that my students buy into these devices to an
alarming extent ... they believe that they can be defined by these tests. I
try to disabuse them of this notion. Still, they are disappointed when a
test doesn't seem to work. One would think that they should feel confirmed,
but they don't.

On a HR Systems level, and with respect to leadership, I would think that
an organization that uses these devices is locked into a coercive control
model. It forces at the screening level, and will probably force again at
the performance appraisal level, etc. Thus, less directive/coercive
systems, e.g., participatory decision making, team management, etc., if
desired, will fail. In MacGregor's terms ... This is a clear Theory X
practice, isn't it?

In short, isn't much of this business about whether the sponsoring
organizations are building organization men and women [remember William
Whyte's Organization Man?] or self-knowing, risk takers?

I don't want to diminish these tests completely. I think that they could be
put to good use .. for example, the ones that I've looked at can cause some
interesting soul searching ... most are based on models of human
interaction or models of the human psyche that could enhance
self-awareness. Suppose that instead of using these tests to screen, we
allowed employees to take them and to use them as food for thought ... to
try to ask themselves whether there is information contained in them that
could help them become better problem solvers, better interactors, better
leaders, etc. Of course, such a practice, would have to be disconnected
from job screening, promotion screening, performance appraisal. It would
truly have to be a tool for self awareness.

On the subject of correlation and claims of accuracy, I had a PhD student
in Ed Admin recently tell me that a company he worked for claimed that
their tests were 60% accurate. He had become a true believer in helping to
sell the tests to client systems. He said that he could 'see' that the
tests worked. When I asked whether that was a correlation coefficient or
r-squared, he wasn't clear. When I asked what the research variables were,
he wasn't clear, but thought that it might be a supervisor's performance
appraisal ranking. When I asked him what the research findings were with
respect to performance appraisals and actual performance, he wasn't clear.
When I asked about the correlation between peer and supervisory performance
rankings, he wasn't clear. When I asked what would be the best dependent
measure, he wasn't sure. When I asked who did the research .. he said that
he thought it was the director of the consulting agency ... a person who
held a position at a local university, one not noted for its research, and
a name I had never read in any good quality research journal. He wasn't
aware as to whether there was independent research on the test.



Dr. R.V. (Ray) Rasmussen
Chair, Department of Organizational Analysis
http://www.bus.ualberta.ca/rrasmussen/
Director, Environmental Research & Studies Centre
http://www.ualberta.ca/~ersc/

Geeze...why does everyone on this list post such LONG messages? Here goes:

<< It's interesting to me to see that the debate on tests for screening
usually follows the line of how high the correlations are between test
results and subsequent performance, often as measured by a variable like
supervisory ratings.>>

Well, because most employers want to hire highly productive people. While we
can dream up lots of measure of productivity, supervisor ratings are usually
pretty reliable (between raters) and also correlate well with output measures
(like widgits per hour) and other important variables (absenteeism).

>>I've not seen a discussion of the less measurable impacts and wonder
whether any of you folks have.>>

By definition, if the impacts aren't measurable, how would you determine the
validity of the tests?

For example, I think that the interviewing/selection process is the initial
stage of a psychological contract that implicitly, if not explicitly,
defines the expectations of the employee. In addition, I think that a
device such as a pencil-and-paper test can have psychological consequences
that need to be considered.>>

There's a related concept called a realistic job preview (Wanous has written
extensively about this) that should be part of most selection systems. This
lets people know what they are in for in a particular job. Wanous
hypothesizes (and has found support) that this helps people who would
turnover in a job to select themselves out (this ain't for me). Those who
see the preview and still decide to hire on have already made a psychological
commitment and then would be more likely to stay.

>>Here's the way I see it:

The employee is in a trap, especially in a job scarce market s/he has no
choice of refusal.>>

At today's unemployment levels in the US? Especially in high-tech areas
where many young people are going? You must be joking.

>>S/he will be tempted to answer the test in the right way ... thus, there's
a motivation to lie. Even when given a test and supplied with an answer
key, my students tell me that they simply can't keep themselves from
answering the way the test demands [of course they may be wrong about
what's demanded]

The test-makers, of course, will screen for social desirability in the
response patters ... a complication within the test's reliability and
validity ... that is often not considered when promoting the usefulness of
the test.>>

Hough and others have done quite a bit of research on social desirability and
faking on personality tests. Their research indicates that this is not an
issue...faking/social desirability responses do not moderate validity. Of
course, on ability tests faking does not even come into the picture.

>>The employee MAY feel demeaned. This organization is telling the testtaker
that it can know about who s/he is by his/her responses to their test.
It's like being hooked up to a lie detector.>>

Kind of a cynical view of things. A personality test gives the applicant a
chance to tell the employer what kind of person s/he is. And does is a more
accurately than an interview.

>>The employee MAY buy in and yield to the power of tests -- thus, becomeing
more other directed or externally located with respect to feelings of
self-control/mastery. S/he MAY actually believe that the test-maker and
administrator can know more about him/her than s/he knows. That the rich,
inner experience that constitutes being human can be broken down into a set
of numbers and used to place people in jobs, careers, etc.>>

I just don't buy that at all. Next thing, you are going to insuinate that
people in white lab coats are hooking people up to strange machines. An
employer can't learn anything about a person that the person won't put on a
test. Only extra-paranoid people would seriously suspect otherwise. There's
plenty of research published on how people feel about employment tests. As
long as an employer communicates the nature and purpose of the test and gives
good feedback people have a generally positive response to them.

>>Along these lines, I notice that my students buy into these devices to an
alarming extent ... they believe that they can be defined by these tests. I
try to disabuse them of this notion. Still, they are disappointed when a
test doesn't seem to work. One would think that they should feel confirmed,
but they don't.>>

There are two issues here. One is the accuracy of a test (construct
validity). Then, there is criterion-related validity (ability to predict job
performance or some other outcome). Construct validity does not ensure
criterion-related validity.

>>On a HR Systems level, and with respect to leadership, I would think that
an organization that uses these devices is locked into a coercive control
model. It forces at the screening level, and will probably force again at
the performance appraisal level, etc. Thus, less directive/coercive
systems, e.g., participatory decision making, team management, etc., if
desired, will fail. In MacGregor's terms ... This is a clear Theory X
practice, isn't it?>>

What if you use tests to select people who are likely to practice Theory Y
management and be team decision makers?

>>In short, isn't much of this business about whether the sponsoring
organizations are building organization men and women [remember William
Whyte's Organization Man?] or self-knowing, risk takers?>>

It all depends on what you are hiring for.

>>I don't want to diminish these tests completely. I think that they could be
put to good use .. for example, the ones that I've looked at can cause some
interesting soul searching ... most are based on models of human
interaction or models of the human psyche that could enhance
self-awareness. Suppose that instead of using these tests to screen, we
allowed employees to take them and to use them as food for thought ... to
try to ask themselves whether there is information contained in them that
could help them become better problem solvers, better interactors, better
leaders, etc. Of course, such a practice, would have to be disconnected
from job screening, promotion screening, performance appraisal. It would
truly have to be a tool for self awareness.>>

I don't think you understand the nature of most tests that are used in
employment settings. Most of the valid ones are very focused on specific job
abilities or job related personality characteristics. I doubt very much that
creating self insight is related to job performance whatsoever. Also, if you
are selecting people based on a certain set of characteristics, and those
characteristics are not related to performance expectations, then both are
doomed to fail. Why wouldn't you want to select people and appraise them
(and provide training) on the critical competencies of the job?

>>On the subject of correlation and claims of accuracy, I had a PhD student
in Ed Admin recently tell me that a company he worked for claimed that
their tests were 60% accurate. He had become a true believer in helping to
sell the tests to client systems. He said that he could 'see' that the
tests worked. When I asked whether that was a correlation coefficient or
r-squared, he wasn't clear. When I asked what the research variables were,
he wasn't clear, but thought that it might be a supervisor's performance
appraisal ranking. When I asked him what the research findings were with
respect to performance appraisals and actual performance, he wasn't clear.
When I asked about the correlation between peer and supervisory performance
rankings, he wasn't clear. When I asked what would be the best dependent
measure, he wasn't sure. When I asked who did the research .. he said that
he thought it was the director of the consulting agency ... a person who
held a position at a local university, one not noted for its research, and
a name I had never read in any good quality research journal. He wasn't
aware as to whether there was independent research on the test.
>>

Just because your student is ignorant of industrial psychology research
methods is no reason to question the field. If your student was in a field
where people research employment tests (such as industrial psychology) s/he
would have no problem identifying a room full of research that documents the
validity (regression coefficients) of a large variety of pre-employment
tests. As well as research that demonstrates the return on investment of
such tests.

Ray, I would suggest that you take a look at some of that research as well.

Warren Bobrow, Ph.D.
The Context Group
www.contextgroup.com

At 04:04 PM 10/9/97 -0400, Warren Bobrow wrote:
>Geeze...why does everyone on this list post such LONG messages? Here goes:

Perhaps it's because we seem to feel a need to reply to every point?

> Ray Rasmussen had written: The employee is in a trap, especially in a
job scarce market s/he has no
> choice of refusal.>>
>

and Warren responded:

>At today's unemployment levels in the US? Especially in high-tech areas
>where many young people are going? You must be joking.

Warren: As the Academy internationalizes, and esp. as its members
participate in this global medium of the List, we will find that not all
participants live in the US of A. Conditions are different in other parts
of the world, occupational groups, etc.

Tom Bryant.

+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+
Prof. Thomas A. Bryant, Ph.D.
1997-98 Chair, Entrepreneurship Division, Admin. Sciences Assn of Canada
(ASAC)
Entrepreneurship Program, Faculty of Business, Brock University
St. Catharines, ON, L2S 3A1, CANADA
tbryant@peregrine.bus.brocku.ca
(905) 688-5550, ext. 4372 voice
(905) 984-4188 fax.
e-mail: tbryant@peregrine.bus.BrockU.ca