Discussion: View Thread

On 9 Mar 98 at 23:43, Garfield E. Lear, Jr wrote:

> Robert,
>
> I think you are mistaken. The DiSC instruments that are marketed by Carlson
> Learning Company have very high validation and reliability scores, and have
> been heavily researched with groups from all over the world. This is just one
> reason that I am impressed with their products.

From the study below:

"Claiming construct validity for an
instrument implies evidence that the instrument measures the construct
or trait. This review of empirical literature on the PPS found little
data concerning its reliability.

My experience is that no amount of disconfirming data will convince
typology proponents of anything...but on the off chance..here are
several independent studies that do NOT substantiate the claims made
by Carlson.

The issue, apart from Garfield's mention of Carlson's evidence, is
the degree to which independent third party research supports the
reliability and validity of any instrument, as published in
independent peer reviewed journals. Very few typology users bother to
look for other references on the subject, other than those supplied
by the vendors (are they really going to share negative findings?)

One issue here is that typology is a HUGE industry. You can purchase
a DISC mug, or little buttons, or pens with your type on it. That's
an indicator that we are in the realm of pop psych. and not research.

The personal seductiveness of all of the tools available, on a
personal basis seem to result in no amount of negative evidence being
relevant to people's opinions on the value of the instrument.



I did a quick check. Here are some abstracts.

ERIC_NO- EJ453804
-TITLE- Factor Analysis of the Personal Profile System.
-AUTHOR- Henkel, Thomas George; Wilmoth, James Noel
-JOURNAL_CITATION- Journal of Experimental Education; v60 n3 p271-80
Spr 1992 -LANGUAGE- English -DESCRIPTORS- Algorithms Behavior_Patterns
Construct_Validity Factor_Analysis Military_Personnel
Personality_Measures Profiles Scores -IDENTIFIERS- Orthogonal
Comparison Personal Profile System Rotation -ABSTRACT- Principal
components extraction with orthogonal and oblique rotations tested the
construct validity of the Personal Profile System (PPS) using data
from 1,045 senior noncommissioned Air Force officers. Four factors
accounted for 85 percent of the total variance, but the results do not
completely justify publisher claims for the PPS. (SLD)

***************************************************************
-ERIC_NO-
ED324344;
-TITLE-
Theoretical Considerations for Extracting Meaning from Personal
Profile System Data: The Need for Independent Construct Validity
Studies.; -AUTHOR- "Henkel, Thomas George; Wilmoth, James Noel";
-LANGUAGE- English; -DESCRIPTORS- Adults Behavior_Patterns
Factor_Analysis Personality_Measures Personnel_Evaluation
Test_Reliability Behavior_Rating_Scales Construct_Validity
Literature_Reviews Psychological_Testing Test_Validity
Theory_Practice_Relationship -IDENTIFIERS- Paper and Pencil Tests;
Self Scoring Tests; Personal Profile System; -ABSTRACT- "The Personal
Profile System (PPS) is a psychological testing instrument that has
been widely used. The construct validity of the PPS was studied
through a review of the literature. This paper organizes the
literature review into three broad categories: the background of the
PPS; the reliability of the PPS; and the validity of the PPS. The PPS
is a self-scoring instrument measuring the behavioral responses of
people along four dimensions: (1) dominance; (2) influencing; (3)
steadiness; and (4) compliance. The instrument is designed to provide
a systematic and comprehensive perception of an individual's
behavioral tendencies and the behavioral tendencies of those with whom
the individual comes in contact. Claiming construct validity for an
instrument implies evidence that the instrument measures the construct
or trait. This review of empirical literature on the PPS found little
data concerning its reliability. (SNIPPED FOR LENGTH)

*********************************************************************

Robert Bacal, Inst.For Cooperative Communication, rbacal@escape.ca
Visit our Resource Centre for articles on mgmt.,training,communication, and defusing hostility
at http://www.escape.ca/~rbacal (204) 888-9290

Robert,

I think you are mistaken. The DiSC instruments that are marketed by Carlson
Learning Company have very high validation and reliability scores, and have
been heavily researched with groups from all over the world. This is just one
reason that I am impressed with their products.

For someone who is supposed to be a communications expert, I am surprised that
you can not see any value in helping people see that not everyone uses the same
communication style that they use. The Personal Profile System is a great tool
for just this purpose.

Gary Lear
President

Resource Development Systems
Management & Training Consultants

P.O. Box 3185
Lake City, FL 32056

Phone: 904-754-0920
Fax: 904-754-0094

Web: www.atlantic.net/~lear-rds
e-mail: lear-rds@atlantic.net

A Carlson Learning Company Independent Performax(r) Consultant

Quality Training Increases Quality In Your Organization!


-----Original Message-----
From: Robert Bacal
Sent: Monday, March 09, 1998 5:26 PM
To: Garfield E. Lear, Jr; MG-ED-DV@MAELSTROM.STJOHNS.EDU;
MG-ED-DV@MAELSTROM.STJOHNS.EDU
Subject: Re: [MG-ED-DV] FW: [MG-ED-DV] FW: [MG-ED-DV] MBTI Exercises

On 9 Mar 98 at 21:56, Garfield E. Lear, Jr wrote:

My
> undergraduate degree was in psychology, and graduate work in
> Industrial/Organizational Psychology, so I do understand the implications of
> each. However, some of the reasons that you state you prefer the MBTI over
the
> DiSC is precisely why I like the DiSC.

Then you probably know that the DISC lacks the strong validation
studies needed (personally, I don't use tools like this for reasons I
have outlined in some articles in our newsletter).


Robert Bacal, Inst.For Cooperative Communication, rbacal@escape.ca
Visit our Resource Centre for articles on mgmt.,training,communication, and
defusing hostility
at http://www.escape.ca/~rbacal (204) 888-9290

On 11 Mar 98 at 9:50, Phillip Rutherford wrote:

> Re the DiSC discussion - we have experimented with this
> instrument, running it more than once with the same group, and
> have found that unless participants are 100% honest with their
> responses each ti> DiSC, MBTI, psychometric testing, and so on all
> suffer from the same limitation

The issue of test-retest reliability is critical. From Silberman et
al in The J. of Dental Education: (with reference to the MBTI)

"The study showed 75.8 percent of
students change type during their dental education using one
standard, and 11.2 percent showed changes in personality using
another standard. (GLR)"

If intstruments exhibit this level of unreliability, it is doubtful
that they measure anything meaningul or validly.

Another issue re: standardized types of tests is that when one does
validity studies, one has to use the exact same method of
administration (eg. protocal, or group based, or individual based).
One has to validate for each particular administration technique.

Hence a test that is normed and validated for use in a one-on-one
situation is not necessarily valid when administered via computer, or
in a group. That is why tests like the MMPI can only be administered
by trained personnel, so that the testing conditions will remain
"standard".

But let's face it. As people are indicating, the issue isn't
evidence, or whether the test measures what it says it does. What
matters to most is whether it is acceptable (plausible) to those
being tested. Whether partial truths are being taught to learners
doesn't seem to be of great concern to many practitioners; partly
because many don't understand the issues, which can be quite
technical.

Robert Bacal, Inst.For Cooperative Communication, rbacal@escape.ca
Visit our Resource Centre for articles on mgmt.,training,communication, and defusing hostility
at http://www.escape.ca/~rbacal (204) 888-9290

Re the DiSC discussion - we have experimented with this instrument, running it more than once with the same group, and have found that unless participants are 100% honest with their responses each time (oh yeah! and where do we find those participants?) then the results will differ markedly. Some people have in fact come out with a rating almost exactly opposite to their previous results.

DiSC, MBTI, psychometric testing, and so on all suffer from the same limitation - the person completing the instrument doesn't know the rules whereby the instrument was developed. He/she is a normal person who is often prone to trying to give answers the examiner is looking for (isn't that the way we have been conditioned to respond to examinations? If you've forgotten, go and relive your Sociology vs Philosophy lectures where each spends most of their time contradicting the other) and as such provides the one element of the experiment that will doom them all to failure.

We have one simple rule when developing and running such simulations or instruments - make them so simple an idiot can complete it, because one day an idiot will be asked to do just that!

Regards

Phil

P D Rutherford & Associates Pty Ltd
Competency-based systems specialists
0011 61 2 6230 4823
robnphil@ozemail.com.au

At 09:50 AM 3/11/98, you wrote:

>DiSC, MBTI, psychometric testing, and so on all suffer from the same
limitation - the person completing the instrument doesn't know the rules
whereby the instrument was developed. He/she is a normal person who is often

Certainly, effective use of instrumentation depends on good faith responses
from the persons completing the instrument. We have not had many problems
in this area, perhaps because there are no answers we "want" from persons
completing the MBTI or DiSC.

There is a companion to the Personal Profile System (PPS) called the
Interactive PPS (IPPS). It allows as many as six others to complete a PPS
on another person. When coupled with an individual's own PPS, it can be a
powerful tool to get individual's self-misconceptions. Needless to say
(but I'm saying it anyway), it must be used with great care and skillful
preparation of the administrator and of the participants.

Frank Bell Internet:
Project Leader fbell@nonamebbs.com
Amtrak
National Training and
Conference Center
110 S. French St.--Ste 200
Wilmington, Del. 19801

http://members.aol.com/frankwbell

On 13 Mar 98 at 21:02, Thomas Sullivan wrote:


> Good point Frank. These self-assessment profiles are meant to be used
> as "tools" among many other strategies to help people to become more
> effective and more successful in organizations today. They are intended
> for "self-examination", something to wonder about and consider in light
> of many other things. Questions of validity/reliability/construct, etc.
> is very important to an extent. In the long run, however, the person
> receiving the feedback/results is the ultimate judge of the instruments
> "truth-telling" ability.

Thomas, one problem which you may not have thought about is the
psychological "pressure" to accept the results of an instrument, and
then seek evidence to confirm those "results". So while you are right
that the ultimate issue is the person's judgment, that judgment,
itself is not always sound. The results from some of these
instruments are indeed non-specific enough (much like astrology
results), so that people can "fit" them and believe them.

I could write a fake personality typology instrument whose results
would be acceptable to a majority of people taking it. It isn't
exactly rocket science, particularly given the demand characteristics
of the "scientific" based instrument.

So, if personal interpretations are potentially flawed, and the
results still vague, the reliability and validity of the instrument
become critical.

Unless of course we simply don't care if we the instrument is
providing an adequate mirroring of someone's style. I suspect that
some users of the instruments don't care if it actually gives
accurate results or feedback. I happen to think that's unethical,
WETHER or not feedback recipients "believe" the results.

So does the psychology community.

Robert Bacal, Inst.For Cooperative Communication, rbacal@escape.ca
Visit our Resource Centre for articles on mgmt.,training,communication, and defusing hostility
at http://www.escape.ca/~rbacal (204) 888-9290

Frank Bell wrote:
>
> Certainly, effective use of instrumentation depends on good faith responses
> from the persons completing the instrument. We have not had many problems
> in this area, perhaps because there are no answers we "want" from persons
> completing the MBTI or DiSC.

Good point Frank. These self-assessment profiles are meant to be used
as "tools" among many other strategies to help people to become more
effective and more successful in organizations today. They are intended
for "self-examination", something to wonder about and consider in light
of many other things. Questions of validity/reliability/construct, etc.
is very important to an extent. In the long run, however, the person
receiving the feedback/results is the ultimate judge of the instruments
"truth-telling" ability.

Carlson's profiles happen to be some of the best "truth-telling"
instruments that I have ever experienced. Sometimes the best "test" of
any instrument is the person's testimony after scoring. And with
Carlson's DiSC, the reactions are usually something like "AH-HA",
"That's Me", "I Could have guessed it".

Tom Sullivan
Professional Growth Associates
412-835-7341