Fred,
Thanks, this helps. I wonder if you and others have seen the classification schemes created to "rank evidence." While there are a variety, here is one:
http://www.uspreventiveservicestaskforce.org/uspstf/grades.htm
From the US Preventative Services Task Force, this framework includes grades for both "level of certainty" (from high to low, derived from nature of evidence) and "suggestions for practice" (from A to D, or I as insufficient).
What would you and others on the list think about classifying management education findings using such a classification system? This would provide one possible mechanism to help separate wheat from chaff....
Ken
~~~~~~~~~~~~~
Kenneth G. Brown, Ph.D., SPHR
Associate Professor of Management & Organizations and Tippie Research Fellow
Editor, Academy of Management Learning & Education
Henry B. Tippie College of Business
The University of Iowa
Iowa City, IA 52242
Ph: 319.335.3812 Fx: 319.335.1957
From: Management Education and Development Discussion [mailto:MG-ED-DV@AOMLISTS.PACE.EDU] On Behalf Of Fred Nickols
Sent: Friday, October 14, 2011 1:36 PM
To: MG-ED-DV@AOMLISTS.PACE.EDU
Subject: Re: AOM 2012: Planning for a PDW/paper stream concerning the validity of using university students as a generalisable sample
Ken:
What I take away from the two articles in question is that unless we are capable of authoritatively and competently examining and critiquing the design and conduct of a research effort, we are not in a good position to question – or accept – its findings. My bottom-line conclusion is that a great deal of so-called scientific research isn't and its findings are not to be trusted. On the other hand, I'm equally sure there is a great deal of solid scientific research being conducted and reported. For me, as a non-scientist, the difficulty lies in knowing which is which and, frankly, I'm darned if I can tell. Even if I were qualified to evaluate research design, conduct and findings, I would hate to have to do that every time an interesting new or contradictory finding was reported. It seems to me there ought to be some kind of mechanism for sorting the wheat from the chaff.
Regards,
Fred Nickols
Managing Partner
Distance Consulting LLC
Home to "Solution Engineering"
1558 Coshocton Ave – Suite 303
Mount Vernon, OH 43050
www.nickols.us | fred@nickols.us
"We Engineer Solutions to Performance Problems"
From: Management Education and Development Discussion [mailto:MG-ED-DV@AOMLISTS.PACE.EDU] On Behalf Of Brown, Kenneth G
Sent: Friday, October 14, 2011 10:33 AM
To: MG-ED-DV@AOMLISTS.PACE.EDU
Subject: Re: AOM 2012: Planning for a PDW/paper stream concerning the validity of using university students as a generalisable sample
Fred, thanks for sending this particular blog link. I read it over lunch and came away disheartened. The author (whom I can't identify from the blog) criticized Lehrer for sweeping generalizations but then himself wrote the following:
"The problem isn't that the Decline Effect happens in science; the problem is that we think psychology and ecology and economics are sciences. They can be approached scientifically, but their conclusions cannot be considered valid outside of their immediate context."
"The problem isn't the Decline Effect, the problem is you believed the data had the force of F=ma. No one should be surprised when medical "truths" turn out to be wrong-- they were never true to begin with."
"And the moment you talk to The New Yorker, your misinterpreted statistical association becomes truth."
So I think the blogger here is ranting against the popular press (and scientists who talk to the popular press) and how it reports on findings too early and too flippantly. But he also seems to be dismissing all empirical research and, in particular, dismissing all research excepts physics.
Some of the studies described by Lehrer are rigourous meta-analyses, synthesized across multiple studies. They are not "n = 4 studies" (to use another quote from the blog post). In this post I do not see a substantive criticism of the studies that Lehrer discusses as a basis for his views.
As somone interested in the exchange of knowledge between science and practice, I come away thinking that the blog author believes there is no truth and no value in scientists trying to explain their findings to a broader public. This seems to me to be a nihilistic view of all social science, rather than a targeted criticism of the Decline Effect.
Maybe I missed something?
Ken
~~~~~~~~~~~~~
Kenneth G. Brown, Ph.D., SPHR
Associate Professor of Management & Organizations and Tippie Research Fellow
Editor, Academy of Management Learning & Education
Henry B. Tippie College of Business
The University of Iowa
Iowa City, IA 52242
Ph: 319.335.3812 Fx: 319.335.1957
From: Management Education and Development Discussion [mailto:MG-ED-DV@AOMLISTS.PACE.EDU] On Behalf Of Fred Nickols
Sent: Friday, October 14, 2011 8:13 AM
To: MG-ED-DV@AOMLISTS.PACE.EDU
Subject: Re: AOM 2012: Planning for a PDW/paper stream concerning the validity of using university students as a generalisable sample
In relation to the article Gary posted, here's one that takes a critical view of the claims made in it.
http://thelastpsychiatrist.com/2011/02/the_decline_effect_is_stupid.html
Regards,
Fred Nickols
Managing Partner
Distance Consulting LLC
Home to "Solution Engineering"
1558 Coshocton Ave – Suite 303
Mount Vernon, OH 43050
www.nickols.us | fred@nickols.us
"We Engineer Solutions to Performance Problems"