This is a very interesting thread. It seems to me that the question of
measuring the effectiveness of "training" is crucial to every individual,
community, business and society. We spend billions on education. U. S.
businesses spend more than the combined budgets of U. S. academic
institutions. Yet, there is little consensus on ROI.
Unfortunately, certain factions do not want to be measured. And I'm not
talking students, here.
Pardon the long posting that follows. I just could not control myself so
break out your empathy and humor me.
It seems to me that we should be more clear on what we are trying to
accomplish with "training." I find the following taxonomy useful. It
concerns the state of the "student" when they enter "training." There
appear to be six possible states:
What they know they know, but can't apply.
What they know, but will soon forget.
What they know, but don't know they know.
What they don't know, that would be valuable for them to know.
What they know, that ain't so.
What they don't know they don't know.
We can train humans and some animals (not including felines, it seems) to
behave in certain ways. I refer to that as embedding techniques, skills
(such as effective presentation), and certain "values" (such as tenacity).
We can measure the effectiveness of the trainer by interrogating the
"student" regarding what they encountered (ah, testing). We can measure
the effectiveness of the session by observation of before and after
behavior. In the case of guitarists our ear is our best measuring
instrument. It the case of the Olympics the stopwatch and tape measure are
appropriate instruments. The measurements can be taken during or shortly
after the session. In this case, Return Business is an effective indicator
of perceived value.
We help humans learn facts and ontologies and certain "values" (such as
empathy). We can measure the effectiveness of the learning environment by
interrogation. We can measure the subsequent effectiveness of the learner
only by observing 'in situ' achievements. Attitude surveys are useful,
here. An effective measuring instrument focuses on language, especially
the subject's ability to employ it. The single most telling factor in
successful people is their language ability. Fernando Flores takes this a
step further and refers to educating as the act of "languaging" a
community. Measures are a function of time. If taken immediately after
the experience the effectiveness index is typically modest and degrades
with time. However, the index may rise in a few weeks or months as the
learner integrates the new with the old and comes to appreciate what they
learned. Too often they forget, or deny, where they learned it so
testimony is not a good measurement device.
In my experience, helping people learn that "what they know, ain't so" is
one of the most interesting and touchy facets of management consulting. I
have a good time at this but I do not get much "repeat business" -- except
from those few with a high level of consciousness.
We can help humans learn how to learn. Donald Schon, MIT, in the book,
Educating the Reflective Practitioner, described this rather well. We can
measure the effectiveness of the mentor or coach by noticing the attitude
of the learner. Interestingly, the learner's attitude toward the source of
learning typically gets negative -- just before they understand. We can
measure the effectiveness of the subject by their testimony. But because
much of this learning is tacit rather than explicit, months to years may go
by before their understanding influences their judgement and behavior.
Note -- if you are a mentor or a coach -- so much for customer
satisfaction. In a "learning how to learn" situation, one should not seek
high customer satisfaction scores nor high peer popularity scores.
So, what you measure and how you measure and when you measure depends a
great deal on which "training" you are trying to accomplish and whether you
are measuring the "transmitter" or the "receiver" or the "effect". Once
you get more clear on those factors, measuring becomes much more
straightforward.
Have faith. One day, soon, when we can measure the secretion of endorphins
and enkaphalins in the "student's" brain, there will be no more mystery.
But don't wait.
Jack Ring
Innovation Management
32712 N. 70th St.
Scottsdale, AZ 85262-7143
602-488-4615