Discussion: View Thread

  • 1.  Cognitive Flexibility in Ill-Structured Problem

    Posted 12-04-1998 14:50
    Kim, the structured manner of your selection process is vital to
    obtaining reasonably predictive results from interviews. I agree that the
    interviews in the medical study mentioned must have been very unsystematic and
    arbitrary.

    However, the predictive validities you quote seem low. I believe Hunter
    & Hunter's study is old - about 1984?

    There is a more recent review:
    Schmidt, Fl. L., & Hunter, J. E. (1998). The validity and utility of
    selection methods in personnel psychology: Practical and theoretical
    implications of 85 years of research findings. Psychological Bulletin, 124,
    262-274.

    The validities of various personnel selection procedures in predicting
    overall job performance are calculated. Some of them are:

    General mental ability tests: .51
    Work sample tests: .54
    Integrity tests: .41
    Structured interviews: .51
    Unstructured interviews: .38
    Job experience (years): .18

    A meta-analysis [Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993).
    Comprehensive meta-analysis of integrity test validities: Findings and
    implications for personnel selection and theories of job performance. Journal
    of Applied Psychology, 78, 679-703.] concludes that supervisory ratings
    have a predictive validity of: .41.



    Pearl Hilliard
    HR Manager
    DPS, USC
    >philliard@dpsm.usc.edu<

    "What cannot be measured cannot be managed" (Anonymous)

    Kim Boal wrote:
    >John Naman's comments regarding the predictive validity of the interview
    >for GPA's should come as no surprise. Some years ago, Hunter & Hunter
    >reported in their meta-analysis of various predictors of job performance
    >the following criterion-predictor validities for the interview:
    >
    >1) When the criterion was supervisor ratings: .14
    >
    >2) When the criterion was promotion: .08
    >
    >3) When the criterion was training success: .10
    >
    >4) When the criterion was tenure: .03
    >
    >


  • 2.  Cognitive Flexibility in Ill-Structured Problem

    Posted 12-04-1998 14:50
    Kim, the structured manner of your selection process is vital to obtaining reasonably predictive results from interviews. I agree that the interviews in the medical study mentioned must have been very unsystematic and arbitrary.

    However, the predictive validities you quote seem low. I believe Hunter & Hunter's study is old - about 1984?

    There is a more recent review:
    Schmidt, Fl. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.

    The validities of various personnel selection procedures in predicting overall job performance are calculated. Some of them are:

    General mental ability tests: .51
    Work sample tests: .54
    Integrity tests: .41
    Structured interviews: .51
    Unstructured interviews: .38
    Job experience (years): .18

    A meta-analysis [Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance.     Journal of Applied Psychology, 78, 679-703.] concludes that supervisory ratings have a predictive validity of: .41.



    Pearl Hilliard
    HR Manager
    DPS, USC
    >philliard@dpsm.usc.edu<

    "What cannot be measured cannot be managed" (Anonymous)

    Kim Boal wrote:
    >John Naman's comments regarding the predictive validity of the interview
    >for GPA's should come as no surprise. Some years ago, Hunter & Hunter
    >reported in their meta-analysis of various predictors of job performance
    >the following criterion-predictor validities for the interview:
    >
    >1) When the criterion was supervisor ratings: .14
    >
    >2) When the criterion was promotion: .08
    >
    >3) When the criterion was training success: .10
    >
    >4) When the criterion was tenure: .03
    >
    >


  • 3.  Cognitive Flexibility in Ill-Structured Problem

    Posted 12-04-1998 17:18
    The interview is currently the most widely used selection tool. I agree with Kim that we should be trying to improve and systematize the selection / interview process. As I mentioned in my previous e.mail, there is a considerable difference in the predictive validity of structured (.51) as opposed to unstructured (.38) interviews. The current focus in personnel selection appears to be trying to combine various tools to improve predictive validity. According the Schmidt & Hunter (1998), combining a structured interview with GMA tests increases the predictive validity from .51 to .65.

    John Naman advocates a random lottery system for qualified candidates. In my opinion, I wouldn't be able to assess whether the candidates were equally qualified until I had interviewed them. Even if several candidates seem equally qualified on paper, I may not be able to assess whether they would fit into the organizational culture, or would have a good fit with myself (if I were the hiring manager). I don't believe that detailed references (if they are obtainable), or even a telephonic interview, would adequately assess a candidate's interpersonal skills.

    Perhaps ad hoc interviewing processes could be compared with measuring the skull for intelligence. However, it is very different if an adequate job analysis is done, the questions have been developed prior to the interview, all candidates are asked the same questions, acceptable answers are specified in advance, responses are rated for appropriateness of content on evaluation forms, and multiple interviewers are used.

    I'm interested in hearing if any organizations hire people without interviewing them first.

    Pearl Hilliard
    HR Manager
    DPS, USC
    >philliard@dpsm.usc.edu<

    "What cannot be measured cannot be managed" (Anonymous)

    John L. Naman wrote:
    >With all due respect, I disagree with Kim's conclusion that "Given the ubiquity
    >of the interview as the primary basis for selecting individuals, I doubt that
    >many people are willing to abandon it. Therefore, I think we have some
    >obligation to try to improve it."
    >
    >The cost of improving interviewing is not likely to have a large effect on
    >predictability. If a technique is flawed to the core, then change it. People
    >used to measure skull size and shape for intelligence and there is no way to
    >"fix" the flaw in such a technique. Reading tea leaves doesn't cut it for
    >forecasting.
    >
    >One Solution: A great psychologist, Bill Howell, proposed that when a qualified
    >pool exists for employment, promotion, admittance, etc., one should select
    >among them randomly (a lottery). The benefits include: less hard feelings
    >because the process is not personal, zero selection bias (gender, race, etc.),
    >and some subtle good effects of broadening diversity. To make it work for a
    >particular position, apply the same analysis that you would have to the
    >interviewing to be sure that the screen (GMAT, GPA, experience, etc.) for the
    >pool includes skills most relevant to success in the position.
    >
    >--
    >Prof. John L. Naman     naman+@pitt.edu


  • 4.  Cognitive Flexibility in Ill-Structured Problem

    Posted 12-04-1998 17:42
    Pearl said that the big problem is to "adequately assess a candidate's
    interpersonal skills". That is exactly what the Med. school profs claimed that
    they were doing in interviews: assessing student's interpersonal skills for
    working with patients and medical personnel (as in a hospital).

    What may make a small difference is that schools can blind admit and TRAIN the
    skills, whereas a business may already desire the skills. However, businesses
    that offer internal training are little different than schools, and can blind
    admit. Perhaps it is possible that professional interviewers could be trained
    to assess interpersonal skills, but my bet is that most managers and doctors
    will still insist on interviewing because they believe that they can do as well
    as professionals. Cynics may say that they are selecting out ambitious young
    people who might be a future political threat to them, which is of course
    another story.

    I am not trying to be an ideologue on the issue, but I do believe that thinking
    outside of the interview box might lead to some useful results and, dare I say,
    competitive advantage in HR resources.
    -- Prof. John L. Naman naman+@pitt.edu


  • 5.  Cognitive Flexibility in Ill-Structured Problem

    Posted 12-05-1998 22:39
    Just a reminder of what .51 means:  It means that there is some predictive value in about 25% of cases.  If you follow the Wall Street Journal's feature pitting randomization against experts' stock picks,  randomization more often than nor has a better predictive value.  Compare the effect of placebos in medicine. Not to be accusatory,  but on what grounds do you claim to be a better judge of an employment applicant having interviewed him or her?  You are not looking for a spouse and trying to identify personal compatibilities. So what are you looking for?  A recent expose' on one of the TV shows revealed just how susceptible interviewers are to subtle superficialities,  which had been manipulated by the show's director;  in each case of acceptance of one and rejection of the other,  the actual interviewee was the same person!
     

    Pearl Hilliard wrote:

    The interview is currently the most widely used selection tool. I agree with Kim that we should be trying to improve and systematize the selection / interview process. As I mentioned in my previous e.mail, there is a considerable difference in the predictive validity of structured (.51) as opposed to unstructured (.38) interviews. The current focus in personnel selection appears to be trying to combine various tools to improve predictive validity. According the Schmidt & Hunter (1998), combining a structured interview with GMA tests increases the predictive validity from .51 to .65.

    John Naman advocates a random lottery system for qualified candidates. In my opinion, I wouldn't be able to assess whether the candidates were equally qualified until I had interviewed them. Even if several candidates seem equally qualified on paper, I may not be able to assess whether they would fit into the organizational culture, or would have a good fit with myself (if I were the hiring manager). I don't believe that detailed references (if they are obtainable), or even a telephonic interview, would adequately assess a candidate's interpersonal skills.

    Perhaps ad hoc interviewing processes could be compared with measuring the skull for intelligence. However, it is very different if an adequate job analysis is done, the questions have been developed prior to the interview, all candidates are asked the same questions, acceptable answers are specified in advance, responses are rated for appropriateness of content on evaluation forms, and multiple interviewers are used.

    I'm interested in hearing if any organizations hire people without interviewing them first.

    Pearl Hilliard
    HR Manager
    DPS, USC
    >philliard@dpsm.usc.edu<

    "What cannot be measured cannot be managed" (Anonymous)

    John L. Naman wrote:
    >With all due respect, I disagree with Kim's conclusion that "Given the ubiquity
    >of the interview as the primary basis for selecting individuals, I doubt that
    >many people are willing to abandon it. Therefore, I think we have some
    >obligation to try to improve it."
    >
    >The cost of improving interviewing is not likely to have a large effect on
    >predictability. If a technique is flawed to the core, then change it. People
    >used to measure skull size and shape for intelligence and there is no way to
    >"fix" the flaw in such a technique. Reading tea leaves doesn't cut it for
    >forecasting.
    >
    >One Solution: A great psychologist, Bill Howell, proposed that when a qualified
    >pool exists for employment, promotion, admittance, etc., one should select
    >among them randomly (a lottery). The benefits include: less hard feelings
    >because the process is not personal, zero selection bias (gender, race, etc.),
    >and some subtle good effects of broadening diversity. To make it work for a
    >particular position, apply the same analysis that you would have to the
    >interviewing to be sure that the screen (GMAT, GPA, experience, etc.) for the
    >pool includes skills most relevant to success in the position.
    >
    >--
    >Prof. John L. Naman naman+@pitt.edu



  • 6.  Cognitive Flexibility in Ill-Structured Problem

    Posted 12-06-1998 12:02
    Leon,
     
    Do you have any idea which TV show had that segment on interviewing or when?  I'd love to try and get a copy.  Thanks.
     
    Kevin Wheeler
    President
    Global Learning Resources
     
    -----Original Message-----
    From: Leon Levitt <levitt@dhol.com>
    To: MG-ED-DV@MAELSTROM.STJOHNS.EDU <MG-ED-DV@MAELSTROM.STJOHNS.EDU>
    Date: Saturday, December 05, 1998 7:36 PM
    Subject: Re: Cognitive Flexibility in Ill-Structured Problem

    Just a reminder of what .51 means:  It means that there is some predictive value in about 25% of cases.  If you follow the Wall Street Journal's feature pitting randomization against experts' stock picks,  randomization more often than nor has a better predictive value.  Compare the effect of placebos in medicine. Not to be accusatory,  but on what grounds do you claim to be a better judge of an employment applicant having interviewed him or her?  You are not looking for a spouse and trying to identify personal compatibilities. So what are you looking for?  A recent expose' on one of the TV shows revealed just how susceptible interviewers are to subtle superficialities,  which had been manipulated by the show's director;  in each case of acceptance of one and rejection of the other,  the actual interviewee was the same person!
     

    Pearl Hilliard wrote:

    The interview is currently the most widely used selection tool. I agree with Kim that we should be trying to improve and systematize the selection / interview process. As I mentioned in my previous e.mail, there is a considerable difference in the predictive validity of structured (.51) as opposed to unstructured (.38) interviews. The current focus in personnel selection appears to be trying to combine various tools to improve predictive validity. According the Schmidt & Hunter (1998), combining a structured interview with GMA tests increases the predictive validity from .51 to .65.

    John Naman advocates a random lottery system for qualified candidates. In my opinion, I wouldn't be able to assess whether the candidates were equally qualified until I had interviewed them. Even if several candidates seem equally qualified on paper, I may not be able to assess whether they would fit into the organizational culture, or would have a good fit with myself (if I were the hiring manager). I don't believe that detailed references (if they are obtainable), or even a telephonic interview, would adequately assess a candidate's interpersonal skills.

    Perhaps ad hoc interviewing processes could be compared with measuring the skull for intelligence. However, it is very different if an adequate job analysis is done, the questions have been developed prior to the interview, all candidates are asked the same questions, acceptable answers are specified in advance, responses are rated for appropriateness of content on evaluation forms, and multiple interviewers are used.

    I'm interested in hearing if any organizations hire people without interviewing them first.

    Pearl Hilliard
    HR Manager
    DPS, USC
    >philliard@dpsm.usc.edu<

    "What cannot be measured cannot be managed" (Anonymous)

    John L. Naman wrote:
    >With all due respect, I disagree with Kim's conclusion that "Given the ubiquity
    >of the interview as the primary basis for selecting individuals, I doubt that
    >many people are willing to abandon it. Therefore, I think we have some
    >obligation to try to improve it."
    >
    >The cost of improving interviewing is not likely to have a large effect on
    >predictability. If a technique is flawed to the core, then change it. People
    >used to measure skull size and shape for intelligence and there is no way to
    >"fix" the flaw in such a technique. Reading tea leaves doesn't cut it for
    >forecasting.
    >
    >One Solution: A great psychologist, Bill Howell, proposed that when a qualified
    >pool exists for employment, promotion, admittance, etc., one should select
    >among them randomly (a lottery). The benefits include: less hard feelings
    >because the process is not personal, zero selection bias (gender, race, etc.),
    >and some subtle good effects of broadening diversity. To make it work for a
    >particular position, apply the same analysis that you would have to the
    >interviewing to be sure that the screen (GMAT, GPA, experience, etc.) for the
    >pool includes skills most relevant to success in the position.
    >
    >--
    >Prof. John L. Naman naman+@pitt.edu