0470861630_01_pre01.fm Page iii Thursday, December 2, 2004 8:30 PM
Psychological Assessment in the Workplace A Manager’s Guide
Mark Cook and Barry Cripps
0470861630_01_pre01.fm Page ii Thursday, December 2, 2004 8:30 PM
0470861630_01_pre01.fm Page i Thursday, December 2, 2004 8:30 PM
Psychological Assessment in the Workplace
0470861630_01_pre01.fm Page ii Thursday, December 2, 2004 8:30 PM
0470861630_01_pre01.fm Page iii Thursday, December 2, 2004 8:30 PM
Psychological Assessment in the Workplace A Manager’s Guide
Mark Cook and Barry Cripps
0470861630_01_pre01.fm Page iv Thursday, December 2, 2004 8:30 PM
Copyright © 2005
John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777
Email (for orders and customer service enquiries):
[email protected] Visit our Home Page on www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to
[email protected], or faxed to ( + 44) 1243 770620. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The Publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Library of Congress Cataloging in Publication Data Cook, Mark, 1942– Psychological assessment in the workplace:a manager’s guide / Mark Cook and Barry Cripps. p. cm. Includes index. ISBN 0-470-86159-2 (hbk.:acid-free paper) — ISBN 0-470-86163-0 (pbk.:acid-free paper) 1. Employees—Psychological testing. I. Cripps, Barry, 1938– II. Title. HF5548.8.C596 2005 658.3′001′ 9—dc22 2004018409 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 0-470-86159-2 (hbk) ISBN 0-470-86163-0 (pbk) Typeset in 10/12pt Times by Integra Software Services Pvt. Ltd, Pondicherry, India Printed and bound in Great Britain by Antony Rowe Ltd, Chippenham, Wiltshire This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.
0470861630_01_pre01.fm Page v Thursday, December 2, 2004 8:30 PM
Contents
About the Authors
vii
Preface
ix
Chapter 1
Assessment in the Workplace
1
Chapter 2
Using Psychometric Tests
25
Chapter 3
Tests of Mental Ability
47
Chapter 4
Personality Tests
67
Chapter 5
Sifting and Screening
99
Chapter 6
References and Ratings
121
Chapter 7
Competence Analysis
139
Chapter 8
Assessment and Development Centres
159
Chapter 9
The Interview
185
Chapter 10
Structured Interviews
205
Chapter 11
Other Special Assessment Methods
223
Chapter 12
Using Assessment to Arrive at a Decision
245
Chapter 13
Workplace Counselling
271
Chapter 14
Performance Appraisal
281
Chapter 15
Training for Testing and Assessment
311
Chapter 16
Professional and Ethical Issues
319
Chapter 17
The Future of Assessment
329
References
341
Index
349
0470861630_01_pre01.fm Page vi Thursday, December 2, 2004 8:30 PM
0470861630_01_pre01.fm Page vii Thursday, December 2, 2004 8:30 PM
About the Authors
Mark Cook is a Chartered Occupational Psychologist. He completed his first degree and D.Phil. at Oxford University and has been a practitioner in occupational psychology and psychological assessment in the workplace since 1968. He directed the standardization of three important personality tests—California Psychological Inventory, Myers Briggs Type Indicator and FIRO-B—in the UK, and is a former Director of Oxford Psychologists Press, a leading UK test publisher. He has researched extensively on personnel selection and psychological testing, including psychological profiles of managers in formerly communist countries and health care workers, and is the author of Personnel Selection, 4th edn, published by John Wiley & Sons in 2003. Mark Cook teaches psychology of work at Swansea University. Barry Cripps is a Chartered Occupational Psychologist consulting independently in industry, business, Higher Education and sport. His interests are in organisational learning and development, assessment and coaching. Having served as a main board Director of Training, External Examiner in HR to top ranking Business Schools, and verifier of test competence in the British Psychological Society, Barry Cripps brings his extensive experience of people assessment in the workplace to this book. Barry Cripps’ practice is in Dartington, Devon.
0470861630_01_pre01.fm Page viii Thursday, December 2, 2004 8:30 PM
0470861630_01_pre01.fm Page ix Thursday, December 2, 2004 8:30 PM
Preface
We have both been engaged in psychological assessment in the workplace for some years, in a variety of sectors, and also have extensive experience of using psychological tests. We perceived a need for a book that would describe the best techniques for psychological assessment, in a form accessible to managers. Our account is intended to be research based and critical. There are some good techniques for psychological assessment available; unfortunately there is also quite a lot that is not really worth using. We aim to sort the wheat from the chaff, for the guidance of line and HR managers. Our style of writing is intended to move away from academic language to explain concepts in a language comprehensible to managers. In selecting research to describe, we have focused on ‘real’ workers and ‘real’ managers, and have avoided citing research that uses college students to simulate managers or other workers, or uses laboratory simulations of work. We would like to thank the many people who have helped us prepare this book. First, we would like to thank the many researchers in the selection area who have generously sent us accounts of research in press or in progress. Second, we would like to thank the managers and students on training courses whose questions and comments have helped us write this book. Third, we would like to thank Karen Howard for her help with the illustrations used. Finally, we would like to thank the Psychology team at John Wiley & Sons for their support and help. Mark Cook Barry Cripps
0470861630_01_pre01.fm Page x Thursday, December 2, 2004 8:30 PM
0470861630_02_cha01.fm Page 1 Thursday, December 2, 2004 8:31 PM
CHAPTER 1
Assessment in the Workplace
OVERVIEW • • • • • • • • • •
Nine general headings in what is assessed. A preliminary list of ways of assessing people. An overview of what is assessed, and how. How assessments should themselves be assessed. The concepts of consistency (or reliability) and accuracy (or validity) in assessment. The correlation and what it means. Outlining some reasons why validity of assessment is sometimes low. The concepts of fairness and adverse impact in assessment. How to identify good work performance. How selection is currently practised in the UK, the USA and Europe.
INTRODUCTION Consider the ideal employee, from the employer’s point of view. Mr/Ms Ideal is never late, never ill, never argues, never refuses to undertake an assigned task, doesn’t join a union, so doesn’t ‘go slow’ or ‘work to rule’ and certainly never goes on strike. Mr/Ms Ideal works 100% of the time, always does every thing exactly right, but also does everything very quickly. Where will the employer find such a paragon? In the showroom of the nearest industrial robot supplier. But suppose we need a human Mr/Ms Ideal. How do we set about finding him/her? Think of someone in your organisation whom you consider is a real liability and ask yourself: • Why is he/she a liability? • What does he/she do to make you think that?
0470861630_02_cha01.fm Page 2 Thursday, December 2, 2004 8:31 PM
2
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
• How did we come to employ him/her in the first place? • What selection assessments did we make? Now let us take a more positive slant. Think of someone who is a real asset, the sort of person you would definitely like more of, and ask yourself: • • • •
Why is he/she an asset? What does he/she do to make you think that? How did we come to employ him/her in the first place? What selection assessments did we make?
Helping you get these decisions right is what the rest of this book is about. There are four main issues: • • • •
Why we assess. What to assess. How to assess it. How to assess the assessment.
WHY WE ASSESS We can distinguish four main reasons for assessing people in the workplace: • • • •
selection promotion downsizing formal appraisal.
Selection is difficult. We are trying to predict how well someone will work, over a period of perhaps 10 years, on the basis of the information we can collect in a period lasting from 30 minutes to at most 3 days. We are limited by ethics, the law and the natural desire of applicants to present themselves in the best light. Promotion and downsizing are logically the same as selection: we compare what we need with what people have to offer. The main difference is we have— or should have—much more, and much better, information. If someone has been doing the job for 5 years, we should have a pretty clear idea of what they are like and what they can do. Downsizing can be trickier than promotion for obvious reasons. These days most organisations in North America and the UK have performance appraisal systems. Employees are formally assessed at regular intervals, and the assessments used either to plan their training and development, or to determine their pay and promotion.
0470861630_02_cha01.fm Page 3 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
3
WHAT TO ASSESS The short answer to this is: ability to do the job. A more detailed answer is provided by a competence analysis, which will list the main competencies that successful employees need (see Chapter 7). We give here a general list of the main headings for assessing staff: • • • • • • • • •
knowledge work skills social skills organisational fit required work behaviours physical characteristics mental ability personality interests and values.
Assessing Staff: General Attributes Knowledge Every job requires some knowledge. It may be high-level expertise, such as knowledge of current employment law, of human anatomy and physiology, or of aerodynamics and materials science. Or it may be something very limited and basic that can be acquired in a few minutes: where to find the broom and what to do with the rubbish when you have swept it up. Sometimes the knowledge element is implicit: employers tend to assume applicants can read and write, use telephones, give change etc. Knowledge can be acquired by training; so need not necessarily be a selection requirement. Mastery of higher level knowledge may require higher level mental ability.
Work Skills A skill is the ability to do something quickly and efficiently: bricklaying, heart surgery, driving a vehicle, valuing a property etc. Skills can mostly only be acquired by practice. Reading a book on driving does not enable you to drive, nor does watching someone else drive. Employers sometimes select for skills, and sometimes train for them. Mastery of some skills may require levels of mental or physical ability not everyone has. Some approaches to selection aim to assess only knowledge and skill(s), because they are very specific, fairly easy to assess and generally uncontroversial. Some psychologists and some managers see limitations in this, because some jobs change so fast that specific knowledge and skills may soon become out of date.
0470861630_02_cha01.fm Page 4 Thursday, December 2, 2004 8:31 PM
4
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Social Skills Social skills are important for many jobs, and essential for some. Here is a selection of words and phrases from job adverts: good communicator, gets on well with others, persuasive, able to control or influence people, teamwork, leadership. These skills are also sometimes called social intelligence, or currently emotional intelligence.
Organisational Fit The term ‘organisational fit’ refers to whether the applicant’s outlook or behaviour matches the organisation’s requirements. These can be explicit: soldiers are expected to obey orders instantly and without question. Fit is often implicit: the applicant does not sound or look ‘right for us’, but there is no written list of requirements, or even a list that selectors can explain to you.
Required Work Behaviours Examples of such behaviours are punctuality, health (avoiding) absence, theft, disorder, and drink and drug abuse. These are often overlooked by psychologists and competence analysts, but all employers need people who come to work on time, and who do not steal, start fights etc. Most organisations also want people who believe that what the organisation does is useful and important, or who at least behave as if they do.
Physical Characteristics Some jobs need specific physical abilities: strength, endurance, dexterity. Others have more implicit requirements for height or appearance. Managers are sometimes suspicious of formal competence or job analyses, because they seem to leave important things out. Formal qualification in HRM, they argue, is not the whole picture when selecting a good HR manager. Psychologists like to think that what is left out is ability, personality, interest and values, which psychological tests can assess. We should always be careful that ‘what’s left out’ does not turn out to be ‘people like me’ or ‘white males like me’. There is always the risk that something you cannot articulate very clearly might be something you are not allowed to select for, for example, gender.
Mental Ability Mental ability divides into general mental ability (or ‘intelligence’), and more specific applied mental skills, for example problem solving, practical judgement, clerical ability, mechanical comprehension etc.
0470861630_02_cha01.fm Page 5 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
5
Personality ‘Personality’ is a word that means different things to different people (discussed in more detail in Chapter 4). Psychologists tend to use it to describe from 5 to 30 underlying dispositions, or personality traits, to think, feel and behave in particular ways. An extravert person likes meeting people, feels at ease meeting strangers etc. An anxious person worries a lot, may cope with stress or pressure less well etc. Personality traits are thought to be fairly deep-seated parts of the person, that underlie many aspects of their behaviour, and which might be difficult to alter. The employer will probably find it easier to select someone who is very outgoing to sell insurance, rather than trying to train someone who is presently rather shy.
Interests, Values and Preferences Someone who thinks drinking alcohol is wrong will normally not seek out work in a bar; someone who wants to help others may find social work or religious ministry more rewarding than selling potato crisps; someone who believes that people should obey all the rules all the time may enjoy being a traffic warden. People often have to take work that does not match their ideals and values, but work that does may prove more rewarding. Applicants have sometimes already assessed themselves in this area, if they have sought career counselling.
Explicit versus Implicit Assessment Some attributes are very explicitly required, especially knowledge and skill. There is no point applying for a post as a doctor without a medical degree. Other requirements are more implicit, sometimes barely even articulated: personality, interest and values, organisational fit, sometimes aspects of physical appearance.
Specific versus General Some requirements are very specific, e.g. skill in comptometer operation. Other are much broader, e.g. adaptable, flexible or outgoing. Very specific skills are easy to assess, or can be trained for if lacking. The assessment is accurate, simple and creates few legal problems. Assessing broader requirements like adaptability or calmness tends to involve psychologists and tests, is often much less accurate, and creates more legal and PR issues. Our example illustrates the problem with very specific requirements: who uses comptometer operators today? It is an obsolete skill. In a time of rapid change we may need to assess broader underlying characteristics.
0470861630_02_cha01.fm Page 6 Thursday, December 2, 2004 8:31 PM
6
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Surface specific
Bricklaying
More general underlying
Strength
Dexterity
Figure 1.1 Underlying attributes or specific skills
Underlying Attributes versus Surface Skills This issue is linked to the previous one. We can assess surface knowledge and skills, such as bricklaying or knowledge of employment law, or we can assess characteristics that underlie the ability to learn to lay bricks, such as strength and manual dexterity, or to learn employment law, such as general and verbal mental ability (Figure 1.1).
Recommendation: Ask yourself what assessments of staff you currently make, under each of the nine headings: knowledge, work skills, social skills, organisational fit, required work behaviour, physical characteristics, mental ability, personality, interests and values. Are these assessments explicit or implicit? Do you assess anything that our list has left out?
HOW TO ASSESS Traditional Selection Methods Figure 1.2 summarises the traditional approach to selecting staff in the UK and the USA. The advertisement attracts applicants, who return an application form. Applicants with satisfactory references are shortlisted and invited for interview. The employer seeks to attract a large pool of applicants, then pass them through a series of filters, until the number of surviving candidates equals the number of vacancies. The traditional filters are: • application form • letter of reference • interview. It became apparent early in the twentieth century that these traditional methods are not always very effective, starting a search for something better. Newer methods include: • psychological tests, of mental ability, personality, interests and values etc.; • work sample tests, such as typing tests or bricklaying tests;
0470861630_02_cha01.fm Page 7 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
7
ADVERTISEMENT
APPLICANTS Reject Consider further
REFERENCES Reject Consider further
INTERVIEW Reject Select
Figure 1.2 Successive stages in selecting staff
• biodata, or biographical questionnaires, which look for common themes in the background of, e.g. successful salespersons; • structured interviews; • assessment centres, which include group exercises and discussions, as written assessments. Most ‘new’ methods have actually been around a long time. Work samples can be traced back to the Boston, Massachusetts streetcar system in 1913; the first personality questionnaire dates from 1917; assessment centres were first used during World War II. The only significant major new development in the past 25 years is the structured interview. Perhaps we have already found every possible way of assessing people that exists, or could exist. We will return to this question in Chapter 17.
Recommendation: Ask yourself which methods of assessing staff you currently use. Distinguish if necessary between different types and level of job. Do you use any assessment methods that we have not mentioned?
What × How We have a list of things we want to assess, and another list of ways to assess them. Different methods are used for different aspects of applicants. Table 1.1 tries to indicate which methods are used for assessing what aspects. An X means that assessment is a main way of assessing that aspect: for example, we assess social skills by interview or assessment, and mental ability by test. We cannot assess social skills
0470861630_02_cha01.fm Page 8 Thursday, December 2, 2004 8:31 PM
8
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Table 1.1 How to assess main employee characteristics What
How Q AF R I
Knowledge X X Work skills X X Social skills Organisational fit Required work behaviours Physical ability Mental ability Personality Interests and values
X X (x) X X
SI AT PI B AVI WS AC PR Other
X X U X X X X X (x) (x)
(x) X U X U X
(x) (x) (x) X (x) (x) (x) (x) (x) (x)
(x) X (x) (x) X
(x) (x) X X X X HR records (x) Physical test (x) (x) (x) (x) (x) (x)
Notes: Q, Qualifications; AF, Application Form; R, Reference; I, Interview; SI, Structured Interview; AT, Ability Test; PI, Personality Inventory; B, Biodata; AVI, Attitudes and Values Inventory; WS, Work Sample; AC, Assessment Centre; PR, Peer Rating.
from the application form, nor mental ability by worksample test. An (x) in Table 1.1 means the aspect can be assessed by that method, but in more incidental way: personality questionnnaires may provide some information about social skill. A U in Table 1.1 indicates that the assessment may assess, or try to assess, the underlying basis of that aspect. Personality tests might pick out the sort of people who will be absent a lot, or drink too much or otherwise be deficient in required work behaviour. One thing that leaps out from Table 1.1 is the shortage of Xs for some aspects of applicants. There are not very many core ways of assessing social skills, or required work behaviours, or interests and values.
Recommendation: Draw up your own version of Table 1.1, indicating how you assess, and what you assess.
HOW TO ASSESS THE ASSESSMENT Assessment methods need to be assessed against six main criteria. An assessment should be: • • • • •
reliable, meaning it gives a consistent account of the person being assessed; valid, meaning it selects good applicants and rejects bad ones; fair, meaning it complies with equal opportunities legislation; acceptable, to applicants as well as to the organisation; cost-effective, meaning the assessment saves the organisation more than it costs to use; • easy to use, meaning it fits into the selection process, and allows the selectors to appear organised and prepared to the applicants.
0470861630_02_cha01.fm Page 9 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
9
Selection methods do not automatically possess all these qualities; we need research to tell us which possess what. Few assessment methods meet all six criteria so our choice of assessment is always a compromise. We will return to this issue in a later chapter, after reviewing assessment methods in turn.
RELIABILITY: DOES THE ASSESSMENT GIVE CONSISTENT RESULTS? Reliability means consistency. Physical measurements, e.g. the dimensions of a piece of a furniture, are usually so reliable their consistency is taken for granted. Selection assessments often are not so consistent. At their worst they may be so inconsistent that they convey little or no information. The Rorschach test asks people what they see in inkblots, and tries to interpret this as evidence of personality, especially conflicts and defences. The Rorschach is not often used in the UK because it is very unreliable, in two ways: • what a person sees in the blot on Monday may be quite different from what they see on Wednesday; • what one expert says about what the person’s Rorschach means may be quite different to what a second expert says. A test that gives inconsistent results from day to day, and that you cannot agree about is not very useful. Unfortunately there are a lot of assessments about that do fail these two simple tests. There are several types of reliability in general use: • Re-test reliability is calculated by comparing two sets of information—Rorschach interpretations, or interview ratings, or IQs—obtained from the same set of people, on two separate occasions, typically a month or so apart. If the test is assessing some enduring aspect of the person, as selection assessments are intended to, the two sets of information should be fairly similar. • Inter-rater reliability is calculated by comparing two sets of ratings given by two assessors to a set of applicants they have both interviewed, or observed in a group discussion, or written a reference for. If the assessors do not agree, one at least of them must be wrong, but which?
Recommendation: Ask yourself what information you have about the reliability of the assessments you presently make of staff.
CORRELATION Selection research uses correlation very extensively. If you are familiar with correlation, you can skip this section. Correlation is the extent to which two characteristics
0470861630_02_cha01.fm Page 10 Thursday, December 2, 2004 8:31 PM
10
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE Tall Tall thin people
9 8 7
XX
XX
X
X
XX
XX
XX XX
X
X
XX
XX XX
XX XX
XX
XX
6
X
XX
XXX XX
XXX XXX
XX XX
XX
5
XX
XX XX
XXX XXX
XXX XX
XX
X
X
Height
4
XX
XX
XX XX
XX XX
XX
3
X
XX
XX
X
2
X
XX XX XX
XX
2
3
4
Short fat people
Short 1
Light
5
6 Weight
7
8
9
10 Heavy
Figure 1.3 Height plotted against weight, showing a positive correlation of 0.75
go together. In selection, we are usually looking at the correlation between the assessment and work performance, but let’s first look at a more concrete example. Height and weight are correlated: tall people usually weigh more than short people, and heavy people are usually taller than light people. However, height and weight are not perfectly correlated—there are plenty of short fat and tall thin exceptions to the rule. Figure 1.3 shows height plotted against weight, for 100 people in their twenties. The correlation summarises how closely two measures like height and weight go together. A perfect one-to-one correlation gives a value of +1.00. Height and weight correlate about 0.75, meaning a strong relationship, but not perfect.
Zero Correlation If two measures are completely unrelated, the correlation is zero—0.00. For example, we know that extraversion and conscientiousness are not correlated. Knowing how extravert someone is will tell us nothing about how conscientious they are, whereas knowing someone’s weight gives us a generally fairly good idea of their likely height.
Weak Correlations In selection, we are looking at the correlation between selection assessment and work performance. It would be nice if we could find selection methods that correlated 0.75 with work performance, like the results in Figure 1.3. Unfortunately, most selection
0470861630_02_cha01.fm Page 11 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
11
High 9 8
Interview rating
Poor applicants selected
X
X
X
X
X
XX
XX
XX
XXX X
7
X
XX
XXX
XXX
XXX
XXX X
6
XX
XXX XXX
XXXX
XXX
XX
5
X XX
XX
XXXX
XXX
XXX
XX
4
X XX
XX
XXXX
XXX
XX
X
3
X XXX XX
XX
XX
X
X
2
X
X
X
3
4
5
Good applicants missed
Low 1 Poor
2
X
6
7
Work performance
8
9
10 Good
Figure 1.4 Interview rating plotted against work performance, showing a correlation of 0.30
methods do not work that well, and we very rarely find a link as close as that between height and weight. Figure 1.4 shows the sort of results that are more typical. Overall there is some relationship between, e.g. interview rating and work performance, but there are a lot of cases of good performers being rejected and poor performers being accepted. Figure 1.4 shows a correlation of 0.30, which is fairly typical of what you find when comparing selection rating with work performance ratings. Note how many good applicants are being rejected, and how many poor applicants are being accepted.
VALIDITY: IS THE ASSESSMENT ACCURATE? Validity means accuracy. A valid selection method accepts good applicants and rejects poor ones. This is clearly the most important thing to check for in any assessment. There is little point using an assessment method that is completely inaccurate. Unfortunately most ways of assessing staff are less than perfectly accurate, and many are so inaccurate they convey little or no useful information.
Validation Research The basic building block of selection research is the validation study. A validation study collects two sets of information, predictor and criterion:
0470861630_02_cha01.fm Page 12 Thursday, December 2, 2004 8:31 PM
12
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Table 1.2 The information we need to calculate the validity or accuracy of a selection assessment Employee number
Name
Interview rating
Performance appraisal rating
1 2 3 4 ⯗ 100
JS MM BK SM
4 3 1 2
4 4 2 3
AB
3
4
• the predictor is the assessment: interview, test, reference etc.; • the criterion is some index of work performance. Let us take the example of the interview. Interviewer ratings form the predictor, while the criterion is a performance appraisal rating, after one year on the job. Table 1.2 shows the information we need to collect. We have 100 people to study in this example. We then calculate a correlation between predictor (interview rating) and criterion (appraisal rating), which is the validity coefficient. In our example, this gives 0.32, meaning there is some link between interview rating and performance appraisal—in other words, that the interview is working, and is a valid selection method. Note that the interviewer’s opinion of the candidates has to be quantified, typically as a rating. You can correlate interview ratings, but not interview impressions, such as ‘good chap’ or ‘I didn’t think much of him’. For some types of assessment, hundreds, even thousands, of validation studies have been carried out. Work psychologists summarise these to find how well each assessment does on average, and whether it does better or worse in particular circumstances (for example, with particular sorts of work or particular types of people). Basic validation research is fairly easy to follow: the bigger the correlation, the better. Two linked aspects of validation—construct validity and incremental validity—are a little more complicated, but very important for any one planning a selection programme. Both concern what an assessment is actually assessing, and how it relates to other assessments.
Recommendation: Ask yourself what information you have about the validity of the assessments you presently make of staff.
Construct Validity When a new selection system is devised, people sometimes ask themselves: ‘What is this assessing?’ ‘What sort of person will get a good mark from it?’ One answer to
0470861630_02_cha01.fm Page 13 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
13
this question should always be ‘People who will do the job well’. But it is worth going a bit deeper and trying to get some picture of what particular aspects of the applicant the test is assessing: abilities, personality, social background, specific skills etc. The technical name for this is construct validity. There are several reasons why it is important to explore construct validity: • If your new test is mostly assessing personality, and you already use a personality test, you may well find that your new test is not adding much to the personality test. Your new test may not be called a ‘personality test’; it may be labelled ‘emotional intelligence’ or ‘sales aptitude’. • If your 2-day assessment centre measures the same as a 30-minute ability test, it would be much cheaper to use the 30-minute test. • If applicants complain about your selection methods, and you have to defend them in court or tribunal, you may want to be able to say exactly what your test assesses, and what it does not. You may be made to look very silly if you cannot! • If you understand what you are doing, you can often devise ways of doing it better. • If your new test turns out to be mostly assessing intellectual ability, you will be alerted to the possibility of adverse impact on certain groups. Construct validity is usually assessed by comparing one selection method, e.g. interview ratings, with other methods, e.g. psychological tests, by calculating correlations. This is fairly easy to do if you have an ongoing assessment programme, and you keep good records. Construct validity tells you what a particular method is actually assessing (which is not necessarily that same as what it is intended to assess). For example, the traditional interview turns out to be assessing mental ability to a surprising extent.
Recommendation: Ask yourself what information you have about what the assessments you presently make of staff are actually measuring.
Incremental Validity Suppose you are using an interview and a mental ability test to select office workers. Someone sells you an expensive computer simulation of key office skills. The people who sell the simulation produce lots of evidence that it predicts good office work very well. After a year or so, management start to complain that the expected improvement in the quality of office workers has not materialised. Did the people who sell you the test mislead you about its validity? No: the simulation does predict good work, but so do the tests you were already using. The new simulation covers exactly the same ground as your existing interview and mental ability test, so it does not improve your overall selection procedure at all. The technical term for this is incremental validity. This problem always arises when sets of selection tests are used. You can never assume that the validities of all the tests in the set can be simply added, to give overall
0470861630_02_cha01.fm Page 14 Thursday, December 2, 2004 8:31 PM
14
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
validity. Many selection tests cover the same ground, so adding new tests often does not improve overall validity. If you use two equally valid tests, there are three possible outcomes: • both tests together do twice as well as either alone; • both tests together do a little better than either alone; • the second test adds nothing at all. All three outcomes are possible, but a little better is probably the most likely. The fact that tests have different names or claim to be measuring differing aspects of employee effectiveness does not mean they will not turn out to be duplicating each other’s efforts. This problem is now being analysed and researched more and more, so advice can be given on combinations of tests that are likely to improve accuracy, and combinations of tests that, while effective in their own right, will not build on each at all.
REASONS FOR POOR VALIDITY Why do selection assessments sometimes fail to give good results? • Low reliability. An assessment that does not succeed in measuring much cannot predict much either. Low reliability can sometimes be improved by training, in e.g. interviewing, or by being careful to use the assessment correctly. • Imperfect measures of work performance. We calculate a correlation between the selection assessment and some index of work performance. The most frequent index is performance appraisal rating. Managers who do these will know why they can sometimes be poor reflections of true performance. Some organisations even today lack clear ideas about good and poor work performance, so cannot define it accurately. If we have defective work performance measures, we cannot achieve valid selection. If we do not know what we are looking for, we cannot know if we have found it. For this reason, the organisation should make sure it has an effective performance appraisal system, or some other way of assessing peoples’ work. If necessary, the organisation should devote some time to clarifying what it exists to achieve, and how it could be certain it has been achieved, and who by. • Selection is not the whole story. Employees’ work performance is influenced by many other factors, besides those characteristics we can assess during selection. Employees’ performance is affected by how they are managed, by the organisation’s culture, by the employment market and by the economy as whole. • Impression management or faking good. Selectors expect applicants to present themselves well. One sometimes hears interviewers complain that an applicant ‘didn’t make much of an effort’ (to describe his/her achievement and potential at sufficient length and with sufficient enthusiasm). Presenting oneself well unfortunately often shades into exaggeration, and from there into lying. Many selection methods rely on information the applicant provides, and often have no way of
0470861630_02_cha01.fm Page 15 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
15
verifying it. We will come across this limitation quite frequently in our more detailed review of selection assessments. • Quality of information. Table 1.3 is Table 1.1 again, but with some indication of the nature of the information each assessment provides. This divides into: – self: information provided by the applicant, on the application form, in the interview, and when answering questionnaires. – reported: information provided by other people about the applicant, principally through the reference, but also in ratings of him/her. – recorded: the applicant has obtained a qualification, which indicates he/she has knowledge or a skill. – demonstrated: the applicant performs a task or demonstrates a skill, in a work sample, or on a test with right and wrong answers. Assessments that demonstrate a skill are clearly better evidence than ones where the applicant claims to possess it. If someone says he/she can use a computer, we may doubt him/her, but if they succeed in doing specified tasks with it, there is no room for doubt. We noted previously that some of our nine main headings of things to assess were short of good means of assessment. Table 1.3 adds a further problem. There is a far greater shortage of assessments that clearly demonstrate that quality, as opposed to relying on someone’s word for it—usually the applicant’s. Only knowledge, skill and mental and physical ability can be assessed by demonstration.
Table 1.3 How to assess main employee characteristics, indicating the nature of the information obtained What
How Q AF R
I
SI
Knowledge rec self rep S/D S/D Work skills rec self rep self self Social skills (rep) S/D S/D Organisational fit rep (self) (self) Required work rep self self behaviours
AT
PI
B
AVI WS AC
PR
Other
(rep) (rep) dem rep self rep self rep HR records: rec dem (rep) Physical test: dem (dem) (rep) (dem) (rep) self rep dem
(self) (self) self self self
Physical ability
(rep) (self) (self)
Mental ability Personality Interests and values
(rep) (self) (self) dem (rep) (self) (self) self (self) (rep) (self) (self) (self)
Notes: Q, Qualifications; AF, Application Form; R, Reference; I, Interview; SI, Structured Interview; AT, Ability Test; PI, Personality Inventory; B, Biodata; AVI, Attitudes and Values Inventory; WS, Work Sample; AC, Assessment Centre; PR, Peer Rating. rec, recorded; rep, reported by others; self, reported by self; dem, demonstrated; S/D, reported by self and/or demonstrated.
0470861630_02_cha01.fm Page 16 Thursday, December 2, 2004 8:31 PM
16
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
FAIRNESS: WILL THE ASSESSMENT CREATE ADVERSE IMPACT? Fairness in selection means conforming with equal opportunities laws and codes of conduct. All developed countries have these. The USA led the way with the Civil Rights Act (CRA) of 1964, which prohibited discrimination in employment on grounds of race, colour, religion, national origin or gender. Later laws in the USA brought in age and disability. Similar laws followed in the UK: the Race Relations Act (1976), the Sex Discrimination Act (1975) and the Disability Discrimination Act (1995). Figure 1.5 shows how fair employment laws work in the USA; other developed countries, including the UK, have followed the same general model and adopted many of the key concepts. • An assessment that excludes too many of a protected minority, such as non-white persons, is said to create adverse impact. • The employer can remove the adverse impact by quota hiring to ‘get the numbers right’, i.e. equal number of men and women, correct proportion of ethnic minorities etc. • Or else the employer can argue that the adverse impact is justified because the selection test is job related, meaning accurate or valid.
Choose an assessment
No No problem
ADVERSE IMPACT? Yes Yes
No problem
QUOTA HIRING? No No JOB RELATED? Yes Yes ANY ALTERNATIVE? No Test is fair
Figure 1.5 Stages in deciding whether a test is legally ‘fair’
Use it
0470861630_02_cha01.fm Page 17 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
17
• The employer who succeeds in proving the test job related faces one last hurdle— proving that there is no alternative test that is equally valid but does not create adverse impact. Note that adverse impact is not what the layperson thinks of as ‘discrimination’. Adverse impact does not mean turning away minorities or women, to keep the job open for white males, or otherwise deliberately treating women or minorities differently. This is clearly unacceptable, and not likely to done by any major employer these days. Adverse impact means that the selection method results in more men or majority persons getting through than women or minority persons. Adverse impact means an employer can be proved guilty of discrimination, by setting standards that make no reference to race or gender, and that may seem well-established, ‘commonsense’ practice. • Height and strength tests for police, fire brigade and armed forces create adverse impact, because they exclude more women. • In Britain some employers sift out any applicants who have been unemployed for more than six months, in the belief they will have lost the habit of working. The (UK) Commission for Racial Equality (CRE) argues this creates adverse impact on ethnic minorities, who are more likely to be unemployed. • The important Griggs case in the USA ruled that high school diplomas and ability tests created adverse impact, because fewer African American applicants had diplomas or reached the pass mark set on the ability test. • The Green v. Missouri Pacific Railroad case in the USA showed that excluding applicants with criminal records created adverse impact because more non-whites than whites had criminal records. Adverse impact is very easy to prove; all you need is a breakdown of the workforce by gender and ethnicity. If there are too few women or minorities, there has been adverse impact. Table 1.4 shows the composition of the UK House of Commons after the 2002 election. There are clearly far ‘too few’ women and minority MPs. In the USA, the Uniform Guidelines introduced the four-fifths rule. If the number selected divided by the number of applicants for a protected minority is less than four-fifths of the highest ratio for any group, adverse impact applies. Table 1.4 shows how this would apply to the House of Commons: there are still ‘too few’ women and Table 1.4 Composition of the British House of Commons in 2002, actual and expected Actual Male Female White Minority
530 120 641 9
Expected
Expected (four-fifths rule)
325
260
39
31
Note: ‘Expected’ composition is based on the assumption that MPs are selected regardless of race and gender, and that 6% of the population belong to an ethnic minority.
0470861630_02_cha01.fm Page 18 Thursday, December 2, 2004 8:31 PM
18
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
minority MPs. (In fact the law does not apply to the House of Commons, because MPs are not classed as employees.) Adverse impact assesses the effect of selection, not the intentions of the people who use it. Adverse impact is a very serious matter for employers. It creates a presumption of discrimination, which the employer must disprove, possibly in court. This will cost a lot of time and money and may create damaging publicity. Selection methods that do not create adverse impact are therefore highly desirable, but unfortunately not always easy to find. If the employer is convinced that the assessment is necessary, even though it creates adverse impact, they must prove it is job related or valid. For example, a police force that thought all officers needed to be at least six feet tall would have to prove that shorter men and women were unable to do the job as well. Some American police forces tried to do this, without success. They were unable to demonstrate any relationship between height and any indicator of effective performance in police work. The risk of legal challenge, and possible poor publicity, means that the HR department often has to act defensively and may make our account of assessment sound a little negative or pessimistic in places. US law covers age discrimination, but British law does not—yet. The British government plans to comply with European Directives about age discrimination and introduce legislation by 2007. Britain has recently—December 2003—added religious belief and sexual orientation to the list of legally protected minorities.
Recommendation: Keep careful records of the gender and ethnicity of all the people you assess at work, and of everyone who applies for employment with you. Check these periodically to ensure that you are not selecting out too many of any group at any stage of the assessment process.
ACCEPTABILITY: WHAT WILL APPLICANTS THINK OF THE ASSESSMENT? Selection methods should be acceptable to applicants. In times of high unemployment, employers may feel they can ignore applicants’ reactions. In times of labour shortage, unpopular methods could drive applicants away. Applicants’ views of selection methods may influence their decision on whether to accept a job offer. A recent Belgian survey even reports that applicants who do not like an organisation’s assessment methods may stop buying its products (Stinglhamber et al., 1999). Surveys indicate that some assessment methods are more popular with applicants than others: they like interviews, work samples and assessment centres, but do not like biodata, peer assessment or personality tests. Applicants see more job-related approaches as fairer: simulations, interviews and more concrete ability tests (vocabulary, maths). They see personality tests, biodata and abstract ability tests (letter sets etc.) as less fair because they seem less job related. Preferences in the USA and France are generally similar, but personality tests and graphology are more
0470861630_02_cha01.fm Page 19 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
19
acceptable in France, although still not very popular. Research indicates that people like selection methods that are job related, but do not like being assessed on things they cannot change, such as personality.
Recommendation: Ask applicants what they think of your assessment procedures. Ask yourself how you would feel about completing the assessments you use.
COST: DOES THE ASSESSMENT JUSTIFY ITS COST? A good selection method saves the employer more in increased output than it costs to use. Employers can calculate the costs of an assessment (and usually do—so many tests at £5 each, so many psychologists at £500 a day etc.). It is more difficult to calculate the benefits of good selection. Some employers have estimated the cost of getting it wrong, of hiring the wrong person; the US Corporate Leadership Council in 1998 generated estimates based on wasted salary and benefits, severance pay, headhunter fees, training costs and hiring time, for three levels of full-time employee: Entry level: $5–7K $20K salary: $40K $100K salary: $300K However, the problem goes deeper than avoiding obvious mistakes. Work performance is not either/or; psychologists argue that it is normally distributed, like the data in Figure 1.6, for people making socks (‘hosiery loopers’). The best
Percentage of employees
30
20
10
0
1
2
3
4
Dozens per hour
Figure 1.6 Distribution of output in 199 hosiery loopers Source:
From Tiffin (1943).
5
6
7
0470861630_02_cha01.fm Page 20 Thursday, December 2, 2004 8:31 PM
20
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
worker does seven dozen pairs an hour, while the worst worker apparently manages only four pairs an hour. The great majority of loopers, however, are in between, falling either side of an average of four dozen pairs an hour. They are not good enough to be outstanding, nor poor enough to be considered for termination. Distributions of this type have been found for other occupations. The distribution in Figure 1.6 means that HR’s job is more than just avoiding employing the handful of very poor workers. HR should maximise the number of really good workers. If the sock factory can find workers whose output is five or six dozen pairs per hour, rather than two or three, they can double output (or halve the workforce). Critics have seen a flaw in this argument. Every organisation in the country cannot employ the best employees; someone has to employ the rest. Good selection cannot increase national productivity, only the productivity of employers who use good selection methods to get more than their fair share of talent. Employers are still free to do precisely that. The aim of this book is to explain how.
GOOD WORK PERFORMANCE Selection research compares a predictor, meaning an assessment, with a criterion, meaning a index of the person’s work performance. The criterion side of selection research presents problems of its own because it requires us to define good work performance. This can be very simple when the work generates something we can count: widgets manufactured per day or sales per week. The problem can also be simple if the organisation has an appraisal system whose ratings can be used. The supervisor rating criterion is widely used, because it is widely available, simple and hard to argue with. How do you know X is a good worker? Because management says so. On the other hand, the criterion problem can get very complex, if we want to explore rather deeper into what constitutes effective performance. Questions about the real nature of work, or the true purpose of organisations, soon arise: • Is success better measured objectively by counting units produced, or better measured subjectively by informed opinion? • Is success at work unidimensional or multidimensional? • Who decides whether work is successful? Different supervisors may not agree. Management and workers may not agree. The organisation and its customers may not agree.
Recommendation: Ask yourself what definition of good work performance your organisation presently uses.
0470861630_02_cha01.fm Page 21 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
21
CURRENT SELECTION PRACTICE We have a lot of surveys to tell us how selection is currently done in North America, the UK and Europe. We know rather less about the way it is done in the rest of the world.
The USA Figure 1.7 summarises a recent survey of 250 US employers (Rynes et al., 1997). Reference checks are the most frequently used assessment, followed by structured interviews, drug tests then unstructured interviews. Biodata, assessment centres and psychological tests, of either personality or ability, are rarely used. Another survey (Harris et al., 1990) probed deeper, and asked why personnel managers choose or do not choose different selection methods: • the most important factor was accuracy; • the least important factors were cost and fairness; • factors of middling importance were fakability, offensiveness to applicant, and how many other companies use the method. Another survey, by contrast, asked personnel managers why they did not use particular methods (Terpstra and Rozell, 1997): • some techniques they did not think useful: structured interviews, mental ability tests; • some they had not heard of: biodata; • some they did not use for fear of legal problems, notably mental ability tests.
Frequency of use
5 4 3 2 1
ie w in te rv ie w A bi lit y Pe te rs st on al ity te st Bi od A ss ata es ce sm nt e r n W e t or k sa m pl e Dr ug tes t W or kt ria l
en
rv te
St ru
ct
ur
ed
In
er Re f
G
ra
de
s
ce
0
Figure 1.7 Survey of 251 American employers, showing methods used to select experienced staff (1 = never; 3 = sometimes; 5 = always) Source:
Data from Rynes et al. (1997).
0470861630_02_cha01.fm Page 22 Thursday, December 2, 2004 8:31 PM
22
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Table 1.5 The US Merit Principles Survey Method
Extent used Great
Moderate
71 69 39 36 38 34 20 15 10 7
25 26 46 41 44 28 37 28 19 19
Prior work experience Interview Quality of application Reference Level of education Major field of study Personal recommendations College GPA Reputation of college Written test Source:
Data from US Merit Systems Protection Board (2000).
The most recent survey (US merit Systems Protection Board, 2000) describes the US government sector, and lists methods used to ‘a great’ or ‘a moderate’ extent. Table 1.5 shows that interviews are still the most popular assessment, with application form and reference also widely used. The US public sector avoids written tests, and gives little weight to which university people went to, or how well they did there.
The UK The most recent UK survey (Hodgkinson et al., 1996) finds the traditional combination of application, reference and interview used by most employers, but being joined by other methods, including ability and personality tests and assessment centres (Figure 1.8).
Frequency of use
5 4 3 2
ta se ss m ce en nt t re
da
As
te ity al on
Bi o
st
st te y lit Pe rs
bi
in ed ur ct
A
te
te rv
rv
ie
ie
w
w
e In
Re fe
re
nc
St ru
A
pp
lic
at
io n
fo
rm
1
Figure 1.8 Selection methods used by 176 UK employers (1 = never; 3 = sometimes; 5 = always) Source:
Data from Hodkinson et al. (1996).
0470861630_02_cha01.fm Page 23 Thursday, December 2, 2004 8:31 PM
ASSESSMENT IN THE WORKPLACE
23
Other UK surveys have looked at more specialised areas: • For graduate recruitment, employers use application form, interview and reference at the screening stage, then interviews or assessment centres for the final decision. • Recruitment agencies, used by many employers to fill managerial positions, always use interviews, usually use references, sometimes use psychological tests, but rarely use biodata or graphology. • University staff are selected by application form, reference and interview, and virtually never by psychological tests or assessment centres. Most universities also ask applicants to make a presentation to existing academic staff—a form of work sample test. • One-third of the British workforce work for small employers, with fewer than 10 staff, where employers rely on interviews (where they try to assess honesty, integrity and interest in the job, rather than ability).
Europe The Price Waterhouse Cranfield survey (Dany and Torchy, 1994) covers 12 Western European countries and 9 methods. The survey reveals a number of interesting national differences: • the French favour graphology, but no other country does; • application forms are widely used everywhere except in the Netherlands; • references are widely used everywhere, but are less popular in Spain, Portugal and the Netherlands; • psychometric testing is most popular in Spain and Portugal, and least popular in West Germany and Turkey; • aptitude testing is most popular in Spain and the Netherlands, and least popular in West Germany and Turkey; • assessment centres are not used much, but are most popular in Spain and the Netherlands.
Elsewhere We know much less about selection in other parts of the world. Surveys of New Zealand (Taylor et al., 1993) and Australia (Di Milia et al., 1994) find a very similar picture to the UK; interview, references and application are virtually universal, with personality tests, ability tests and assessment centres used by a minority. The same seems true of Africa: selection in Nigeria and Ghana generally uses interviews and references, occasionally paper and pencil tests, work samples or work simulations.
0470861630_02_cha01.fm Page 24 Thursday, December 2, 2004 8:31 PM
0470861630_03_cha02.fm Page 25 Thursday, December 2, 2004 8:31 PM
CHAPTER 2
Using Psychometric Tests
OVERVIEW • The need for standardising of assessment procedures, and how to achieve it. • Understanding the test administration process. • Guidance on running a Test Administration Workshop. • Stages in handling test results, from raw scores, through standard score systems, norming scores, recording and reporting scores. • A case study covering the mechanics of a selection centre.
INTRODUCTION What is a psychological test? The British Psychological Society (BPS) provides the following definition: a procedure for the evaluation of psychological functions. Psychological tests involve those being tested in solving, performing skilled tasks or making judgements. Psychological test procedures are characterized by standard methods of administration and scoring. Results are usually quantified by means of normative or other scaling procedures but they may also be interpreted qualitatively by reference to psychological theory. Included in the term psychological test are tests of varieties of: intelligence; ability; aptitude; language development and function; perception; personality; temperament and disposition; and interests, habits, values and preferences.
There are two main types of psychometric test used in workplace assessment: • tests of maximum performance, or mental ability tests; • tests of typical performance, or personality tests. Tests of mental ability in turn divide into general mental ability, aptitude and achievement tests:
0470861630_03_cha02.fm Page 26 Thursday, December 2, 2004 8:31 PM
26
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
• Achievement or attainment tests assess how much someone knows about a particular body of knowledge, e.g. use of Microsoft Excel. Conventional school and university exams are another type of achievement test. So too are professional exams such as CIPD. • Aptitude tests assess how easy it would be for someone to acquire knowledge or skill they do not presently possess, e.g. of computer programming. • Tests of general mental ability assess how good the individual is at understanding and using information of all types.
THE BENEFITS OF USING PSYCHOMETRIC TESTS When seeking to calculate the benefits to an organisation of using psychometric tests we should ask several questions: • • • •
What is your organisation doing now about testing? How is testing being used? Does the organisation need to use tests? Does the organisation have a written policy document on using tests in the workplace?
Testing can benefit an organisation in a range of ways: • • • • • • • • • •
helping with selection by identifying staff with potential; helping with development by identifying staff with potential; improving morale; demonstrating commitment to people; increasing retention; reducing costs and saving time, which can be converted into bottom-line profit; producing in-house benchmarks and norms for good performance; objectively assessing people against known standards and tests; demonstrating fairness to all; ensuring consistency over time, through the application of valid and reliable ways to assess people.
ADMINISTRATION OF TESTS The Need for Standardisation One of the greatest influences on test results, besides candidate performance, is the way that tests are administered. Variable, erratic and inconsistent administration practice is reflected in inaccurate test results in candidates. An obvious example would be varying the recommended time in an ability test; another would be rushing through administration instructions and practice examples, leaving candidates unclear about how to approach questions and give their answers. One reason the traditional
0470861630_03_cha02.fm Page 27 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
27
interview does not work very well is precisely the lack of standardisation; interviewers are inconsistent in what they ask different candidates. Besides making the interview less accurate, this can create grounds for complaints of unfairness. Test administrators should be trained to conduct every testing session following the same, rigorous standard routine. Every person who takes a particular test anywhere should experience the same administration, because this will ensure their responses are comparable. With personality and interest inventories, test takers should be able to focus on themselves in a relaxed, purposeful way, to give their own preferred responses free from outside distraction. The process of making everything the same, and thus fair, for each test taker is called standardisation.
Test Administration Certificate In the UK the BPS plays a key role in monitoring test standards and reviewing test materials. The BPS (2003) has published Psychological Testing: A User’s Guide, which gives valuable guidance about using psychological tests and the principles of good test use, including a summary of the areas covered by tests and testing in the test use guidelines, published by the International Test Commission (2001). The BPS offers a Test Administration Certificate (TAC), which qualifies people to administer, score and norm psychological tests, but not to interpret them or explain the results. The TAC covers the principles of test planning, and the administration and scoring of all types of test (ability, personality, motivation, stress indicators and interest inventories). Holders of the TAC are not qualified to purchase tests or give feedback to candidates regarding test results, and must be supervised by someone who is a fully qualified test user. The TAC is useful for any tester who wishes to delegate the actual test administration. Any verified Assessor, usually a Chartered Psychologist verified by the BPS, can provide training leading for the TAC. The BPS Psychological Testing Centre keeps a register of verified assessors. This chapter breaks the testing process down into the five main areas recommended by the BPS’s TAC. A training course or workshop dealing with test administration will usually begin with an introduction to psychometric testing and tests before moving on to the administration process. What follows is based on a typical workshop programme, as provided by B.C.
Recommendation: Make sure that anyone who administers psychological tests in your organisation has been trained to do it correctly, and that they have a BPS Certificate at the right level.
Standardised Administration Ability testing assumes that test scores of candidates assessed in Penzance will be comparable with those of candidates assessed in Aberdeen, or anywhere. It is
0470861630_03_cha02.fm Page 28 Thursday, December 2, 2004 8:31 PM
28
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
important to try to ensure that every test administrator operates according to the same rules, wherever they are located. The way in which tests are administered can have considerable impact on candidates’ results and is a key element of the reliability of test results. There are six key stages in test administration: • • • • • •
preparing for testing; briefing the candidates; administering the test; debriefing the candidates; scoring the test; managing the results.
PREPARING FOR TESTING Personal Preparation The person giving the test should be thoroughly familiar with it; they should know the administration instructions, be able to explain the practice questions, and ideally have completed the entire test themselves. They should also have practised the full administration of the test on someone who is not applying for a job. It is very useful to draw up a preparation checklist, like the one in Figure 2.1.
Test administration preparation checklist Freedom from distractions ● ● ● ●
Telephone removed or disconnected. Notice on door Please do not disturb, testing in progress. Adequate distance between candidates. Candidates facing a wall if possible.
Good working conditions ● ● ● ● ● ●
Quiet, inside and outside room. Good lighting. Comfortable temperature. Flat, even working surface. Adequate working area. Clock where candidates can see it.
Resources required to administer tests ● ● ● ● ● ●
Stopwatch. Question books, answer sheets. Paper for rough working, if used. Pencils etc. Calculators, if required. Computer equipment, if required.
Figure 2.1 Checklist for preparing for test administration
0470861630_03_cha02.fm Page 29 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
29
Preparing the Test Session Allow plenty of time before the session; testing should never be rushed. Chose the time and location carefully. Time of day is important; morning is best. It is also best to use the same time on each testing occasion. An acceptable number of candidates for one administrator is 12 (although less able or less well motivated people might need to be tested in smaller groups). Make sure you have all the test material you will need; remember it can only be obtained by mail order. Check the test material for marks; the last time it was used someone may have written the correct answers on the question books (or incorrect answers!). Also prepare your testing log (Figure 2.2), for recording date, time, who is tested, with which tests, any problems that arise, general comments and observations. Always use a stopwatch and time to the second; never try to time tests using the minute hand of an ordinary watch.
The Testing Room The room used for testing should be quiet, and have enough space for the number of candidates you plan to test. Candidates should have enough room to spread out question books, answer sheets and rough paper, and should be far enough from each Test Session Log
Date:
Administrator:
Number of Candidates:
Names of Candidates 1 2 3 4 5 6
7 8 9 10 11 12
Position recruited for: Name of test:
Location of Test Centre: Start time Finish time
1 2 3 4 5 Materials checklist Administration instructions Test booklets Test answer sheets Report, record forms Norm tables Calculators Stopwatch General comments about session
Erase stray marks Scrap paper Pencils (2 each) Erasers
Figure 2.2 A typical tester’s test log
Provide spares
0470861630_03_cha02.fm Page 30 Thursday, December 2, 2004 8:31 PM
30
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
other to avoid distraction and the possibility of copying. It is surprisingly difficult in many organisations to find a suitable room; the boardroom is often suitable.
Recommendation: Always plan your testing session thoroughly and well in advance. Use checklists of everything you will need.
BRIEFING THE CANDIDATES Letter of Invitation Every test session requires the informed consent of candidates; ideally they should agree, in writing, to be tested. Figure 2.3 shows a typical letter sent to candidates about two weeks beforehand. With the letter you can send the leaflet, supplied with most good tests, that explains the test and gives some practice questions. Candidates should know what is expected of them when they turn up on the day. This makes your task much easier by preparing them well in advance. A surprising number of employers neglect to tell candidates that the ‘interview’ includes psychological tests. On the day, as candidates arrive, you should explain to them once again the arrangements for the day. This will help establish rapport, put the candidates at their ease, and allay any fears they have. In order to standardise what you say to everyone on each occasion, it is a good practice to prepare a checklist. A useful checklist is outlined in Figure 2.4.
Dear Ms Jones, Selection/Development Centre I have much pleasure in inviting you to attend a selection/development interview day for the Newco Organisation on Tuesday 31st June, at 10:30 a.m. in the Company Training Centre. You do not need to bring anything with you: all materials will be supplied and lunch provided. The day will include a 20-minute interview, a brief presentation from you about how you see your role in this organisation, and some psychometric tests of ability and personality. Please find enclosed some leaflets that describe the tests, and give some practice questions you can try, as well as information about your rights as a test taker. Oral feedback on the tests and a brief written summary will be given to you before you leave. The day should finish around 5 p.m. Please let me know if you have any special needs or requirements before the day begins so that I can make suitable arrangements. Please confirm your availability as soon as you can. If you telephone then you can of course ask any questions. I look forward to hearing from you, Yours sincerely,
Figure 2.3 A typical letter inviting someone to a test session
0470861630_03_cha02.fm Page 31 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
Introduction checklist ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●
Introduce yourself Arrangements for the day Purpose of testing Skills measured, relevance to job Number of tests, how long each Check received invitation letter and practice leaflets Ask if any candidates have done these tests before. If so, enter on log and continue Explain need to read administration instructions from card Spectacles? Special needs? Calculator? Turn off mobiles and pagers Location of toilet; use now Location of fire exit, drill No smoking Water The rest of the selection procedure Feedback Any questions? Get going!
31
Check off ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ ⵧ
Figure 2.4 A typical checklist for test administration
Recommendation: Make sure that people know they will be given tests, both when you invite them and again when they arrive on the day.
ADMINISTERING THE TESTS Test Administration Card It is essential that everyone who does a particular test gets the same instructions. If they do not, their scores may not be comparable, nor can you be certain your data will be comparable with the normative data in the test manual. A test administration card is designed to standardise conditions for that particular test, and to ensure consistency over all test sessions at a single centre, as well as between different test centres. Some tests are provided with administration cards as part of the testing kit; for others it is necessary to prepare your own. Distribute test material, and check every candidate has everything they require. Then read out the test instructions from the administration card. Answer questions by repeating from the administration card as far as possible.
0470861630_03_cha02.fm Page 32 Thursday, December 2, 2004 8:31 PM
32
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Practice Questions At the beginning of most tests, there are a number of example questions. Give candidates adequate time to answer these and do not pressurise them in any way. Walk quietly around the room to check that candidates are putting their answer marks in the correct place and following instructions. It is better to avoid using tests that do not have example or practice questions, as plunging straight into the test proper can put some candidates at a disadvantage, and give less accurate results. When the candidates are ready, give the answers to the practice questions. Candidates should understand the examples themselves without becoming flustered or feeling they are being criticised. Invite questions and give your answer to the whole group. It is of course essential that you are thoroughly familiar with the practice questions. Judge for yourself when to move on. It is important to press on quickly.
Timing the Test Face the candidates, be vigilant. Then start the test, start your stopwatch and record the finish time. Walk around checking that candidates are putting their marks in the right place. During the test sit in full view of the candidates and walk around occasionally. Maintain an atmosphere of quiet urgency. When the time is up, say in a firm tone ‘Pencils down, please. Stop writing’. Collect all test papers from the front, walking to the back of the room. (Some people think they should give 5 minutes’ warning of when time will be up: this is not part of the standard administration procedure for most ability tests, and should not be done.)
Recommendation: Always plan your testing session thoroughly on the day. Use checklists of everything you will need.
Difficult Situations During Testing Here is a list of possible problems that might arise during testing, and suggested ways of dealing with them. Any of these should be noted in the test log. • cheating: decide about disqualification; • fire alarm: local rules as instructed during introduction; note time of interruption; restart if/when possible; • candidate taken ill: stop, deal with emergency, restart if time permits; decide on a re-test for that person; • candidate talking: be firm; • people entering the room: escort out quietly; check ‘Do not disturb’ notice; • candidate opting out: let them go quietly;
0470861630_03_cha02.fm Page 33 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
33
• candidate arriving late: start them at the end of current time or arrange another date; • power cut: stop everyone; restart when power returns; • disruptive candidate: be firm; call security if necessary; escort out quietly; • candidate carrying on after time: repeat instructions loudly, to their face; consider disqualification. Can you think of any other eventuality?
DEBRIEFING CANDIDATES It is important to leave candidates with a good impression of your organisation and testing programme by effective debriefing at the end of a test session. First, collect everything into separate piles before candidates leave the room, to ensure no question books or answers are taken away. Tear up rough work, to reassure candidates it will not be assessed. Next, thank the candidates, and tell them what is going to happen next and how they will receive feedback. This is most important. Then say goodbye to each candidate. Check all reusable material for stray marks. If possible erase the marks, otherwise destroy any defaced material. Remember, test materials are expensive! After candidates have left the room and you have collected all the materials, it is advisable to begin marking their test papers. This allows you to make sure that all the papers are in order and that answers have been marked in the correct place. Marking at this time allows you an element of recovery time before candidates leave the building, during which some errors could be corrected.
Recommendation: Always offer people feedback on the tests they have completed. Make sure they know what the feedback arrangements are before they leave.
SCORING TESTS Test scoring instructions should be followed precisely. Most tests come with scoring keys or templates. Check you are using the correct answer key and that it is lined up correctly. Use a different colour pencil to the writing on the answer form. Until you are entirely familiar with the scoring, check three times and ask a colleague to check as well. Be systematic when scoring: go round the template the same way every time; count correct answers, count incorrect answers, then sum to total as another check. A further check is to reverse your marking order. Double check the alignment of template and totals. Remember, ‘rubbish in equals rubbish out!’ This may all seem very elaborate, but it is surprising how easy it is
0470861630_03_cha02.fm Page 34 Thursday, December 2, 2004 8:31 PM
34
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
to make mistakes in scoring, and how often even very intelligent people make them. Stepping errors occasionally occur, when candidates lose their place on an answer sheet and complete the answer sheet out of sequence. Where five or more consecutive questions are answered incorrectly, move the scoring key up or down two lines to see if this results in a string of correct answers. If so adjust accordingly, correcting when error stops. Make a note in the test log for that candidate.
MANAGING RESULTS Recording Results Candidates’ results should be entered onto a standard report form. If the testing has been done by someone who is only qualified as a test administrator, they should pass on the data to someone who is qualified to interpret the results, give feedback to candidates, or make decisions based on the results.
Confidentiality and Security The purpose of testing is to find something out about people. Such findings are sensitive and need to be kept secure. Confidentiality must be maintained and information kept secure under lock and key. This also applies to blank test materials, answer books and test manuals. Access by unauthorised persons to any test materials including confidential data must be restricted on a ‘need to know’ basis. As a rule, candidate data should be shredded after 12 months, which is the generally accepted ‘shelf life’ for test data. (After 12 months, it is better practice to test the candidate again.)
Feedback Always remember that the results of testing belong to the candidate, as well as to the organisation commissioning the testing. It is important that candidates are given feedback as soon as possible after testing. The organisation should frame a policy about this. Usually the Level A test user will give feedback orally or in writing; giving candidates feedback about test results needs skill, and should not be attempted by unqualified persons. Feedback should be factual, and should avoid making any judgements about the candidate. In the past some employers have followed a policy of not offering feedback, to save time or to avoid arguments. This is very poor practice; it is both unethical and bad public relations. The test administrator should explain to the whole group that decisions made about candidates will not be made solely based on test results. Other information will also be used, from interview, CV and application form, references, a presentation, group exercises etc.
0470861630_03_cha02.fm Page 35 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
35
Recommendation: Prepare a policy document about access to test material, and get it agreed by the organisation.
INTERPRETING TEST SCORES Ability tests produce raw scores—how many correct answers the applicant gives— which mean very little in themselves. The raw score is usually interpreted by comparing it with normative data, e.g. scores for 1000 applicants, which tells test users whether the score is above average or below, and how far above or below average. Normative data—norms for short—allow us to compare one person’s score with another, and to take account of age, job, experience, educational background etc.
Selecting a Norm Group Choice of norm group can make a big difference to how we view a raw score. Imagine that the same raw score is achieved by: • a 16-year-old school leaver; • an experienced 35-year-old middle manager; • a graduate management trainee. Is it fair to compare these three scores? Clearly no, so the raw scores must be normed, i.e. interpreted using an appropriate norm table. The reference point used in psychometric testing to compare scores between people is known as the norm group. The norm group is a set of scores obtained on the same test by a group of people comparable to the candidate. Test administrators should choose the norm group that best fits the candidates being tested (or be advised by the qualified test user). Considerations when choosing an appropriate norm group from the test manual include: • • • • • •
age gender ethnicity educational level work experience job function.
The normative sample should be large, relevant and recent. Comparing applicants with 2000 people who have applied for the same job within the past three years is better than comparing them with only 50 people doing a roughly similar job in Czechoslovakia in the 1930s. (Normative data for some tests are as old as this,
0470861630_03_cha02.fm Page 36 Thursday, December 2, 2004 8:31 PM
36
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
and can be very misleading. Scores on some tests, e.g. Raven Progressive Matrices, have increased considerably since the 1940s, so using 1940s normative data would systematically overestimate candidates’ performance relative to people today.) Choosing normative data often involves an element of compromise. If you are testing bank managers in Bristol, the ideal normative group clearly is Bristol bank managers. However, the available normative data may offer nothing closer than (British) managers (in general), or Australian bank managers. This poses some interesting questions. Are Australian bank managers likely to differ from British managers? Are they a better match for Bristol bank managers than British managers in general? Collecting your own normative data is a very good way of ensuring you can make the most relevant comparisons. Once you have tested 100 persons, for a particular post or group of posts, you can compute your own provisional local norms. Most normative data for ability tests is occupation specific: bank managers, motor mechanics, call centre workers etc. Few tests can offer any broader comparisons. It would be useful for some purposes to have norms based on representative cross-sections of the whole population of the country. The organisation could then select people in the top 25% or screen out people in the bottom 25%, of ‘everyone’. Getting a representative cross-section of everyone—representative for age, gender, ethnicity, education, location, social background etc.—is very expensive, and can never be entirely successful because a large proportion will decline to be tested.
Normative Score Systems Raw scores should be converted into some form of normative score, using information in the test’s manual. Various normative score systems are used (Figure 2.5): • • • • •
mental age and IQ percentiles z scores and T scores stens and stanines grades or bands.
Mental Age and Intelligence Quotient (IQ) A person with a mental age of 8 does as well on the test as the average 8-year-old. If that person is actually aged 16, they are somewhat behind for their age. IQ was originally a mental age system, calculated by dividing mental age by actual (chronological) age, and multiplying by 100. The 16-year-old with a mental age of 8 has an IQ of 8/16 × 100, which works out at 50. Mental age systems are not very useful for workplace assessment because intelligence stops increasing rapidly at around age 16, just about the time people start work.
0470861630_03_cha02.fm Page 37 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
37
z scores
–2.5 –2.0 –1.5 –1.0 –0.5
T scores
25
Stens
30
1
35
2
Stanines
1
IQ
70
40
3
2
45
4
3 85
0
0.5
1.0
1.5
2.0
2.5
z scores
50
55
60
65
70
75
T scores
5
4
6
5 100
7
6
8
7 115
9
8
10
9 130
Stens
Stanines IQ
Figure 2.5 Distribution of mental ability scores, showing mean, standard deviation, z and T scores, stens, stanines and IQs
Percentiles Percentiles interpret the raw score in terms of what percentage of the norm group scores lower than the candidate. Percentiles are easy for the layperson to understand. To give an example, for a sample of 466 British health service managers, a raw score of 7 on the Graduate & Managerial Assessment—Numerical translates into a percentile of 30, meaning that someone who scores 7 gets a better score than 30% of health service managers. The 50th percentile is the average. Percentiles are quite easy to calculate. If a candidate is ranked 15th out of 50, then their percentile is: 35 × 100 50 − 15, i.e. 35, so: --------------------- = 70th percentile 50 Percentiles have a problem, which Figure 2.6 illustrates. Percentiles bunch together in the middle of the scale and spread out at the ends. On most tests, a lot more people score close to the average than score very high or very low. In Figure 2.6, there are a lot of scores between 16 and 22, and far fewer between 0 and 15, or between 22 and 40. This means that in the middle range of scores, a difference of 2 in raw score corresponds to a big difference in percentiles—31 to 69. But for very
0470861630_03_cha02.fm Page 38 Thursday, December 2, 2004 8:31 PM
38
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
−2½ Standard deviations or z scores
−2
−1½
−1
−½
0
+½
Percentiles
1
2
7
16
31
50 69
Raw scores
0–10
11–13 14–15 16–17 18
+1
+1½
+2
+2½
84
93
98
99
19 20–21 22–23 24–28 29–33 34–40
Figure 2.6 The relationship between percentiles and raw scores
high (or very low) scores a difference of 2 in raw score corresponds to a much smaller difference in percentiles. In fact all the raw scores between 34 and 40 fall at the 99th percentile. If you have two or more scores and wish to calculate an average, do not calculate this with percentiles; use z or T scores. Percentile is not the same as percentage of correct answers. Candidates often think that their performance is evaluated by calculating what percentage of the questions they got right. In fact, this is hardly ever used to interpret test scores, because it actually tells us very little.
Standard Scores Standard scores are based on standard deviations and the normal distribution, so need a degree of statistical knowledge. They still serve the basic purpose of telling us whether the candidate is above or below average, and how far above or below. The three most widely used standard score systems are: • z scores • T scores • stens.
z Scores The raw score is converted into a z score using the formula raw score – sample mean z = -------------------------------------------------------------sample standard deviation Here is a numerical example. On the AH4 test, candidate A’s raw score is 101. The AH4 manual gives normative data for 1183 naval ratings. The normative sample’s mean is 75.23, and its standard deviation is 14.58. Calculating z gives a value of +1.77. A z score of +1.77 shows that candidate A scores 1.77 standard deviations above the norm group mean. Here is a second numerical example. Candidate B’s raw AH4 score is 63, which gives a z of −0.83, which means candidate B scores 0.83 standard deviations below the norm group mean.
0470861630_03_cha02.fm Page 39 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
39
The z score system is not very user friendly, because it includes plus and minus signs and decimals, and because two-thirds of z scores will be between +1 and −1.
T Scores The T score system avoids decimals and signs, using the formula raw score – sample mean T = 50 + 10 × -------------------------------------------------------------- sample standard deviation Here is the same AH4 numerical example. Applicant A’s T score is (50+(10×(101− 75.23)/14.58)) which is 67.67. T scores are always rounded to the nearest whole number when used to describe individuals, so A’s T score is given as 68. Applicant B’s T score is (50 + (10 × (63 − 75.23)/14.58)), which works out at 42. T scores and z scores give exactly the same information, but T scores are more user friendly, avoiding minus signs or decimal points. T scores have a mean of 50 and a standard deviation of 10. T scores usually run from 20 to 80. Table 2.1 shows how many people will fall in different T score ranges in a normal distribution. Are naval ratings a good comparison group? Could they be taken as a crosssection of the British population of the day? Probably not. Apart from the fact that all 1183 are men, the navy, even in the days of conscription, could choose its recruits, and may not have chosen the less able. This would mean that the naval rating average was higher than the general population average. IQs were originally derived from mental ages, but where they are used today, they have been turned into standard score systems, where the mean is set at 100 and the standard deviation at 15. In fact IQs are not used much for several reasons: • IQs need very good normative data based on large cross-sections of the entire population, which few tests have. • IQs create a misleading impression of precision, by appearing to place people on a 60-point scale from 70 to 130. Ability tests are nowhere near accurate enough to achieve such precision. • The name ‘IQ’ tends to alarm people unnecessarily. Table 2.1 Proportion of people tested falling into different T score ranges, in a normal distribution T score
Percentage
Interpretation
Over 80 71–80 61–70 51–60 41–50 31–40 21–30 20 or under
Less than 1 3 13 33 33 13 3 Less than 1
Very high High High average Low average Low Very low
0470861630_03_cha02.fm Page 40 Thursday, December 2, 2004 8:31 PM
40
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Sten or Standard of Ten A very useful 10-point scoring system, a sten has a mean of 5.5 and a standard deviation of 2. To calculate a sten, use the formula: sten = (2 × z) + 5.5 Here is a numerical example, for the AH4 raw score of 101, where we have already calculated the z score as +1.77. The sten is (2 × 1.77) + 5.5, which is 9.04. Stens are rounded to the nearest whole number, giving 9. Many test manuals give tables to convert raw scores to stens, saving you the trouble of calculating them. A variation on the sten is the stanine score or standard of nine. These are not used often but in case you come across one, they are a nine-point system, calculated using a variation on the sten formula: stanine = (2 × z) + 5.
Grades or Score Bands Scores are divided into grades or bands, typically on a five-point scale, A to E, based on percentiles (Figure 2.7). Grades are easy for the layperson to understand. A grade or sten system may seem to be wasting some of the information the test provides, by dividing candidates into no more than 5 or 10 levels of ability. In fact the typical ability test does not assess ability all that precisely, and cannot reliably distinguish more than 12 levels of ability, so the sten system does not really lose that much information. Recommendation: Make sure you are familiar with the manual for the test(s) you are using, and that you know how to interpret the raw scores, or else make sure the person interpreting the test scores has the correct expertise, and has completed the appropriate training.
COMPUTER ADMINISTRATION Many tests can now be administered by computer. For one-to-one administrations, this can save an enormous amount of time, not only in administration itself, but also Five grades of mental ability E = bottom 10% (of norm group) D = next 20% C = middle 40% B = next 20% A = top 10%
Figure 2.7 A typical five grade or band categorisation of scores. Note that % refers to percentage of the norm group, not to percentage of correct answers
0470861630_03_cha02.fm Page 41 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
41
in scoring and norming. The computer is perfectly designed to handle administration. For a group administration each candidate will need their own computer, which has obvious cost implications. Computers can be relied on to score the test accurately, and to calculate normative scores. Care needs to be taken regarding computer-generated reports, which can sometimes lack the subtlety of human interpretation.
EQUAL OPPORTUNITIES The guiding principle in legislation is fairness for all candidates, at every stage of selection decision making. We are all entitled to this fundamental right, regardless of gender, race, religion, colour, ethnic origin and disability. The law supports this principle in the UK, the USA, Europe and many other countries. As a guide to managers, testing should be based on: • a thorough job analysis identifying competencies and job demands; • establishing a matching person specification to map personal attributes across to the job analysis; • using valid and reliable tests to assess those personal attributes in the applicants; • demonstrating equal opportunity principles throughout the procedure.
Agencies In the UK, there are three bodies which help enforce equal opportunities law, and issue codes of conduct: • Equal Opportunities Commission, which issued its Code of Practice in 1985. • Commission for Racial Equality, which issued its Code of Practice in 1984. • Disability Rights Commission, which issued its Code of Practice in 1996.
Gender, Ethnicity and Age Employers should record information on gender, ethnicity and age, at every stage of selection, to check for adverse impact. This information should be kept separately and not used to make any decisions about individual candidates.
Disability Discrimination legislation has some specific implications for psychological testing: • Testers must accommodate disabled candidates as far as possible. For example, a candidate with a visual impairment might be provided with large print or Braille materials.
0470861630_03_cha02.fm Page 42 Thursday, December 2, 2004 8:31 PM
42
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
• Testers need therefore to ascertain if any candidates have disabilities that require accommodation. Note, however, that disability discrimination laws generally prohibit direct enquiries about disability during the selection process, to avoid open discrimination. It is necessary, therefore, to employ a form of words enquiring about candidates’ ‘special needs’ (and explaining what sort of assessments will be made, so the candidate can mention, e.g. visual disability if relevant). The fifth paragraph of the letter in Figure 2.3 shows how this is done. • Accommodation is often a problem with ability tests because they generally have time limits, so any change in format may make the test more difficult. Suppose we use a Braille form for visually handicapped candidates. Is Braille slower to read than conventional print? So does the candidate also need more time? How much more? • Time limits also present difficult decisions when assessing people with dyslexia. Schools and universities often use a blanket extra 50% time allowance in written exams, but this is very crude and unlikely to result in accurate assessment. • The normative data for most tests assume a particular time limit has been used, and are not valid otherwise. • In the long term, we need a new generation of ability tests that do not depend on time limits. One currently available test, the Raven Progressive Matrices, does have normative data for untimed administration.
The implications for people involved in using psychometric tests are extremely important and wide ranging. Test administrators need to be aware of the relevant legislation, and that every stage of using tests could potentially have a discriminatory effect on those taking tests.
Recommendation: Be careful to observe the laws and codes of practice covering assessment of applicants, especially those concerned with accommodating people with disabilities. Keep up to date with the constantly changing law.
DATA PROTECTION The final unit in the TAC standards deals with data protection, particularly when data are stored on a computer. The various UK Data Protection Acts (1984, 1988, 2001) have implications for employers who use tests (and for other selection methods as well).
0470861630_03_cha02.fm Page 43 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS
43
• Candidates should be told how their results will be used. • Organisations should appoint a data controller, and include their name in the consent paragraph on application forms. • For ‘sensitive personal data’, including psychometric tests, the Act requires the explicit consent of the individual. However, it appears that even explicit consent need not be in writing, so long as it is unequivocal. Data controllers should ‘consider the extent to which the use of personal data by them is or is not reasonably foreseeable’. Anyone sent an introductory letter, like the one in Figure 2.3, should know that they are taking a psychometric test, which may be used for selection or development. However, if there is doubt about this, candidates should be told again, before doing the test. • Employees have the right to see their paper-based records, at reasonable intervals and immediately.
Certification: Where Next? All delegates taking training as test administrators are encouraged to continue to the further levels of competence, as part of their professional continuing development and to enhance the good name of testing in the United Kingdom. Holders of the TAC will be exempted from the test administration units of those further qualifications.
CASE STUDY IN TEST ADMINISTRATION Transpo is a distribution company, running heavy goods deliveries in the UK. Transpo has recently advertised in the national and trade press for a Key Accounts Executive. There were 47 enquiries for the application pack and 29 returned applications. After careful consideration of the job analysis and pre-screening of the applications, eight candidates were selected by the HR team to attend a selection centre. The HR manager, Susan Harris, has the BPS Level A and B Certificates of Competence in Occupational Testing. The HR manager is under pressure currently to produce a detailed annual report for the end-of-the-month Directors’ meeting. Accordingly she delegates the setting up and administration of the selection centre to her assistant, Alan Jones, who has recently qualified as a test administrator via the BPS Test Administration Certificate. This will be Alan’s first live test administration exercise so he is very keen to impress his boss that spending money on his TAC was worthwhile. Alan was involved with his boss in the initial job analysis exercise, advertising for the position and pre-screening, but is looking forward to operating independently. Susan has asked him to tackle the task as a project, and to map out all the steps needed, from sending letters of invitation to handing over the completed test results to her at the end of the selection centre. Alan prepares a draft project plan, with allocated days and timing.
0470861630_03_cha02.fm Page 44 Thursday, December 2, 2004 8:31 PM
44
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Memo: Selection Centre for Key Accounts Executive From: Alan Jones To: Susan Harris Dear Susan, Please find enclosed my first draft project plan [Table 2.2] to run the Selection Centre for the new post of Key Accounts Executive. I welcome your comments. I understand that the Selection Centre will be in 20 working days time on Tuesday December 2nd. Sincerely, Alan
Table 2.2 Selection centre for Key Accounts Executive, Tuesday December 2nd Day 1
Date
Actions
Nov. 5th
Post letter of invitation to 8 shortlisted candidates inviting acceptance of interview date for selection centre. Invite responses via email, telephone or letter. Enclose practice leaflets and map to training centre. Book accommodation in training centre, 10 desks, refreshments on arrival, December 2nd. Check test materials, answer books, answer forms, report forms, scoring templates, norm tables. Erase any stray marks in answer books. Order new answer forms for verbal and numerical reasoning tests. Send order form to Susan for authority and signature. Copy test session log. Open materials file for selection centre.
2
6th
3
7th
4
10th
5
11th
6
12th
7
13th
8
14th
9
17th
10
18th
Receive acceptance letters Receive acceptance letters Receive acceptance letters Receive acceptance letters Receive acceptance letters Telephone non-repliers requesting intentions
Send order form to test publisher requesting 24-hour delivery of test materials. Check testing kit for pencils, erasers, stopwatch and calculators. Receive and check test materials from test publisher.
Revise introduction checklist; make notes relevant to job and selection day, check for fire alarm practice December 2nd. Review administration card, refreshing memory as to test procedure, example questions and question completion.
0470861630_03_cha02.fm Page 45 Thursday, December 2, 2004 8:31 PM
USING PSYCHOMETRIC TESTS 11
19th
12
20th
13 14
21st 24th
15
25th
16
26th
17
27th
18 19
28th Dec. 1st
20
2nd
Practice scoring and marking a practice test. Print off 10 candidate report forms. One candidate, B.C., requests large-print materials. Order from test publisher requesting 24-hour delivery. Receive and check large-print materials. Telephone B.C. to reassure. Assemble all materials in test flight case. Sharpen pencils. Check every item against numbers of candidates. Check with training and catering over room allocation and refreshments. Print off schedule for day for each candidate. Include times for tests, interview, presentations and feedback. Progress report to Susan. Final checks, room, refreshment, timing, lunch, interviews and presentations. Print off schedule for interview panel members. 09:30 meet candidates, briefing 10:00 commence test administration 12:00 dismiss and thank candidates 12:10 score tests 1:00 prepare report forms 2:00 hand over test data to Susan.
Memo: Selection Centre for Key Accounts Executive From: Susan Harris To: Alan Jones Thank you, Alan, for your schedule for the Selection Centre. You have put a lot of time in here and it looks as if you have covered everything. Well done. We mustn’t forget candidates’ expenses so would you mind please collecting some expense forms from accounts and handing them out during your introduction? Thank you once again, your TAC is paying off. I look forward to receiving the completed candidate results on Tuesday at 2:00 p.m. Regards, Susan
45
0470861630_03_cha02.fm Page 46 Thursday, December 2, 2004 8:31 PM
0470861630_04_cha03.fm Page 47 Thursday, December 2, 2004 8:31 PM
CHAPTER 3
Tests of Mental Ability
OVERVIEW • • • • • • • • • • •
Ability tests divide into achievement, aptitude and general ability. Some tests seek to assess more specialised abilities, such as creativity. All tests of mental ability tend to go together, i.e. correlate positively. We are not yet sure what emotional intelligence tests measure. Mental ability tests predict work performance fairly well. The link between mental ability is continuous and linear: the brighter someone is, the more likely their work performance is to be good. Research on mental ability and team, rather than individual, performance suggests more complex relationships. Low scorers may hold the whole team back. Research on why mental ability tests predict work performance suggests that mental ability leads to improved job knowledge, which in turns leads to better work performance. Mental ability tests create adverse impact on some ethnic minorities in the USA; there is a dearth of information on this issue elsewhere. Attempts to deal with the adverse impact problem include score banding, which defines a range of scores as equivalent, thus allowing selection within the band to be based on achieving diversity. Validation research tends to underestimate validity of assessment, because it is limited by small samples, unsatisfactory measures of work performance and restricted range; research needs therefore to be analysed carefully.
INTRODUCTION If you only have an hour to assess an applicant, some psychologists argue that the best way to use your time is to give an ability test. Ability tests give a reasonably accurate assessment of one key aspect. The more usual choice of a one hour
0470861630_04_cha03.fm Page 48 Thursday, December 2, 2004 8:31 PM
48
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
assessment—the interview—may try to cover much more ground, but is much less accurate. It is of course better to set aside more time to assess candidates: after all, they will be working for you for a lot longer than one hour! Figure 3.1 gives some varied (but fictitious) test items divided into achievement, aptitude and ability tests, which were defined in Chapter 2 (p. 26). Most tests use the multiple choice format shown in Figure 3.1, which makes scoring quick and Achievement 1. Which size of fuse should be used for a 3 kW electric fire? 3 amp 5 amp 13 amp
26 amp
2. What did Sigmund Freud describe as the ‘royal road to the unconscious’ alcohol
dreaming
3. Which of the following is/are not a disease? multiple regression metonymy
homosexuality
malaria
toilet training
multiple sclerosis
Aptitude 4. Check whether the names are exactly the same or differ in any way. Smith, John E Smith, John E same different Johnson, Edwin P Jonson, Edwin P same different 5. Which sections of this sentence contain errors of spelling or grammar? In reply to your letter / of the 14th June / I feel it incummbent on me / to strongly advise you / to withdraw the allagation of impropriety / implied by the statement / in the pennultimate paragraph 6. Which seat in a single deck bus gives the smoothest ride? one directly over the back axle one directly over the front axle one midway between front and back axle? Ability 7. What does the word ‘impeach’ mean? preserve
accuse
give a sermon
propose
8. How far is it from New Orleans to New York in miles? 50 800 1300
5000
9. How much is 15 plus 16? 1
51
31
41
10. What is the rate of UK income tax for incomes over £40 000 a year? 25% 40% 75%
90%
11. What do you get if you divide the largest number by the next to smallest number, and then multiply the result by the next to largest number? 20 15 11 14 5 2 110 60 10 300
Figure 3.1 Twelve questions from various types of mental ability test. NB: These are not ‘real’ questions, from ‘real’ tests, but are typical of mental ability test questions
0470861630_04_cha03.fm Page 49 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
49
accurate. Most tests have time limits, which helps ensure the test presents the same level of difficulty to everyone, and makes it easier to plan testing sessions. (However, time limits also create problems for candidates who have dyslexia.)
Tests of General Mental Ability General mental ability is sometimes called cognitive ability or intelligence. Mental ability tests use many different types of question, that vary in content and difficulty. The questions in Figure 3.1 look very diverse, and ‘common sense’ would expect ability to answer one to have little to do with ability to answer another. ‘Common sense’ says knowing what a word means depends on education and home background, whereas mental arithmetic questions, such as question 12, need good concentration and short-term memory. The questions in Figure 3.1 are fictitious but results with similar problems in real tests show fairly high positive correlations; people who are good at vocabulary questions tend to be good also at mental arithmetic. Early on in the study of human intelligence, it emerged that people who are good at one intellectual task tend to be good at others. This makes it possible to talk about general mental ability.
Aptitude Tests Traditionally aptitude means how good someone is likely to be at acquiring a new skill or body of knowledge, such as learning a foreign language. Some aptitude tests seek to assess suitability for a particular profession or professional training, e.g. management, medical school, sales. Aptitudes that are widely assessed include: • • • • •
Clerical: General Clerical Test Mechanical: Bennett Mechanical Comprehension Test Programming: Computer Programmer Aptitude Battery Numerical: Graduate & Managerial Assessment—Numerical Language usage: Differential Aptitude Tests—Language Usage.
Sometimes a number of aptitude tests are combined to form a multiple aptitude battery, such as the Differential Aptitude Tests, which assess verbal reasoning, numerical ability, abstract reasoning, clerical speed and accuracy, mechanical reasoning, and spelling and language usage. In practice the distinction between ability and aptitude is often blurred, especially because most aptitude tests correlate fairly well with tests of general mental ability.
More Specialised Ability Tests Psychomotor or Dexterity Tests These are not paper and pencil tests; applicants assemble small fiddly components or sort larger ones, as quickly as possible. These are useful for assembly work and
0470861630_04_cha03.fm Page 50 Thursday, December 2, 2004 8:31 PM
50
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
some skilled manual work. Fine dexterity is needed for assembling small components using the fingers, while gross dexterity is needed for assembling or handling larger objects using the whole arm. Dexterity can also be assessed by work sample tests.
Culture-Free Tests Few people in the UK, however able, know the distance from New Orleans to New York, but adding 15 to 16 should be possible for anyone whose culture uses numbers. The geography question is more culture bound than the addition question. For years some psychologists looked for culture-free questions, that any one regardless of cultural background could answer. A test that would work equally well in the UK, communist China, and the Amazonian rain forest would be very useful, but has proved difficult to find.
Creativity Critics say ability tests favour ‘convergent’ thinkers, people who can remember textbook answers to questions, but cannot think creatively. They argue that we often need ‘divergent’ thinkers, who can find many equally good answers to a problem, not just one ‘right’ answer. Many tests of creativity have been devised, for example Uses of Objects. The applicant has 2 minutes to list possible uses of a brick, other than building a wall: a door stop, a weapon, a pendulum etc. Or applicants may be asked to design an advertisement for a new product, e.g. a 3D television set. Creativity tests often correlate with standard tests of mental ability very highly, so they cannot prove they are measuring anything new.
Video-Based Testing Several video-based selection tests have been devised, for example for bus drivers and insurance sales staff. Including sound and moving pictures creates a richer test, especially suitable perhaps for work involving a lot of contact with the public, or for trying to assess social/emotional intelligence.
Computerised Testing Paper and pencil tests are inflexible. They irritate able candidates by making them plough through long lists of easy items, and dishearten the less able by asking too many questions they cannot answer. Computerised testing does not have to present a fixed sequence of questions but can be tailored to the individual’s performance. If the candidate does well, the questions get harder; but if the candidate does poorly the questions get easier, until the candidate reaches his/her own limit. Computerised testing seems especially popular in military testing programmes.
0470861630_04_cha03.fm Page 51 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
51
Internet Testing Many Internet recruiting systems include online assessment of ability or personality. Testing people over the Internet is fast and cheap, but creates a number of problems: • Is the person doing the test really John Smith or someone else? • How can you stop people applying several times under different names and so giving themselves unfair practice on the test? • How can the test’s authors protect their copyright, and prevent unauthorised use of the test by unqualified users? Experts have suggested supervised testing centres can solve these problems, but that would lose the convenience and economy of Internet testing. A good test should (a) be reliable, (b) be valid, (c) have useful normative data and (d) not create adverse impact. There are large numbers of dubious looking tests on the Internet that probably fail to do all or even any of these four things. (It is equally easy to generate dubious tests in paper form, but the Internet allows them to be distributed widely, and possibly to cause more damage by giving people information that is not true or which they cannot handle.)
Recommendation: Remember that computer or Internet tests are still tests, and should be critically evaluated before use.
EMOTIONAL INTELLIGENCE Being able to understand others and get along with them is a vital skill in some work which, it is sometimes claimed, intelligent people often lack. Tests of social or emotional intelligence have been devised to fill this gap. They have used a variety of formats: • Identifying emotions expressed in facial expression. • Memory for names and faces. • Written descriptions of interpersonal problems. For example, what is the best way to deal with a noisy neighbour: complain to the police/appeal to his better nature/ threaten him etc. • Sequences of photographs showing an encounter between two people, where the applicant identifies the likely outcome from a choice of endings. • Assessment of ‘tacit knowledge’: who is the best person to ask in an organisation to get a particular task done quickly, and how best to phrase the request. This is information that will not be found in the organisation’s official handbook. • Recent tests of emotional intelligence often resemble personality tests rather than ability tests. There is no time limit, and candidates are describing themselves, not looking for right and wrong answers.
0470861630_04_cha03.fm Page 52 Thursday, December 2, 2004 8:31 PM
52
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Daniel Goleman (1995) has claimed that success at work is ‘80% dependent on emotional intelligence and only 20% on IQ’. On closer inspection, this claim turns out to derive from an analysis of job descriptions, not of test validity. It is certainly true that many job descriptions mention things like being able to work in a team, but it does not follow that emotional intelligence will account for 80% of successful work performance, or that emotional intelligence tests can assess it. Social intelligence tests went out of favour because they were mostly so highly correlated with other tests of general intelligence that there was no reason to suppose they were measuring anything different. Sceptical psychologists ask themselves whether emotional intelligence tests are falling into the same trap. How do we know they are not just assessing familiar aspects of personality, such as sociability or tolerance, under a new name? What we need is research showing that emotional intelligence: • can be separated from personality as well as from conventional intelligence; • predicts work performance; • improves on the prediction of work performance that tests of personality and mental ability can make. Social/emotional intelligence probably is very important in the workplace, but probably cannot be assessed by paper and pencil tests. We may need tests of ‘real life’ behaviour, which are much more difficult to devise, usually much less accurate, and much more time consuming to use. Group exercises and role plays, found in many assessment centres (see Chapter 8), probably assess social/emotional intelligence to some extent.
Recommendation: Think carefully before using tests of emotional intelligence, and ask what evidence you have they will improve on your present assessment.
ERROR OF MEASUREMENT An ability test is not very useful if it says someone is bright today, but two weeks later says the same person is average or dull. A good ability test can achieve high re-test reliability, represented by a correlation between the first and second sets of scores of 0.90. If an ability test cannot achieve a re-test reliability of 0.85, you should probably avoid using it. Calculating re-test reliability means persuading the same set of people to do the same test twice, which is not always that easy, so some test publishers omit this important step. Error of measurement estimates how much test scores might vary on re-test. An ability test with a re-test reliability of 0.90 has an error of measurement of five IQ points, meaning one in three re-tests will vary by five or more points. Clearly it would be a mistake for Smith who scores IQ 119 to regard himself as superior to Jones who scores 118. If they take the test again in three months time, Smith might get 116 and Jones 121. If Smith and Jones are candidates on a shortlist, they should
0470861630_04_cha03.fm Page 53 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
53
be regarded as having the same score on the test, and the decision which to employ should be based on some other consideration such as personality or experience. One reason untrained people should not use psychological tests is that they may not understand error of measurement, and will tend to think test scores are much more exact than they really are.
Recommendation: Never use a test without evidence of its reliability. Do not use an ability test which has a reliability less than 0.85. Always think of a test score as x plus or minus y, where x is the actual score, and y is the error of measurement.
VALIDITY: DO ABILITY TESTS PREDICT WORK PERFORMANCE? The best way of answering this question is research. We need a workforce who have done a mental ability test, and some quantifiable information about work performance. We then compute a correlation between the two sets of numbers. Thousands of researches of this type have been done, and their results summarised by work psychologists. A lot of the research is American but enough has been done in Europe to be sure the same results are found this side of the Atlantic too. Research shows ability tests can select fairly well for most broad types of work. Table 3.1 shows tests achieve corrected correlations of 0.40 to 0.50 for every type of work except sales assistants, where the correlation is less than 0.30. Unskilled and casual work has not been researched, so we do not know whether ability tests are useful. Table 3.1 shows clearly that ability tests do predict performance quite well for most types of work, so are definitely worth using. There are some specific types of work where the link between mental ability and work performance may not be so strong: achievement in science and technology, police officers, pilots and sales ability.
Table 3.1 Summary of validity of mental ability tests for nine broad classes of work Class of job Manager Clerk Salesperson Protective professions Service jobs Trades and crafts Elementary industrial Vehicle operator Sales assistants Source:
Ghiselli (1966).
Validity 0.53 0.55 0.62 0.43 0.49 0.50 0.47 0.46 0.28
0470861630_04_cha03.fm Page 54 Thursday, December 2, 2004 8:31 PM
54
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
The ‘Weakest Link’? Selection persists in assessing the individual, and trying to predict the individual’s success at work. Yet most work is done by teams of people, who often depend on each other to get the job done. So should selection consider the team as a whole? And ask some of these questions: • Does a successful team consist of people who are all able, or will a mixture of ability be sufficient, or even better? • Can one person of low ability hold everyone back? • Will one very able person be able to get the group moving forwards, or will he/ she be submerged and frustrated? Only a few studies have tried to answer these questions, and they do not point to a single simple conclusion: • Units in the Israeli army with higher average mental ability perform better. • Groups of assembly workers with higher average ability produce more, and are less likely to fail as teams. • In motor manufacturing, poorer team performance is linked to more variation in ability, and to the presence of low scorers. The smaller the group, the stronger the effect. These results are consistent with a ‘weakest link’ hypothesis, that a few low scorers, or even just one, can hold the whole team back. Research on mental ability and team performance will undoubtedly be a growth area, and may uncover complex relationships.
Setting Cut-Off Scores Selectors are often asked: is this candidate appointable? In other words, what is the minimum level of mental ability necessary to function in this job? The commonest approach to setting cut-offs is: do not appoint anyone who falls in the bottom third of existing post-holders. Strictly speaking, the idea of a fixed cut-off is simplistic. The relationship between test score and performance is linear, meaning the lower the test score, the poorer the person’s performance is likely to be. This implies that any cut-off is arbitrary. If the employer has sufficiently good records, it may be possible to construct an expectancy table (Table 3.2) showing the level of work performance expected for people with different test score ranges. An applicant who scores 35 on the test has a 21% chance of being very good, and a 58% chance of being good or very good. An applicant who scores 8 on the test has only a 2% change of being very good, and a 20% chance of being good or very good. The percentages come from an analysis of existing employees. The employer can decide the level of performance required and set the cut-off accordingly. In practice the decision also depends on how easy it is to fill the vacancy, and how urgently it needs to be filled.
0470861630_04_cha03.fm Page 55 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
55
Table 3.2 Expectancy table, showing relationship between test score and work performance, assessed by performance appraisal Test score
30–40 25–29 20–24 11–19 Under 11
Performance appraisal (%) Very poor
Poor
Average
Good
Very good
7 6 11 13 26
15 20 18 36 31
20 32 44 33 23
37 31 20 15 18
21 11 7 3 2
Necessary But Not Sufficient Some people argue that mental ability is necessary but not sufficient for good work performance. Table 3.3 shows that few accountants had an IQ more than 15 points below the accountant average, whereas quite a few lumberjacks had an IQ well over their average of 85. Assuming the latter had not always wanted to be lumberjacks, the data imply they could not or did not use their mental ability to find more prestigious work. Perhaps they lack some other important quality: energy, social skill, good adjustment or luck.
Incremental Validity Mental ability tests by and large predict work performance quite well. What other assessments are worth using alongside them that will give a better prediction still? There is no point adding new tests to a selection battery if they cover the same ground as tests already in use. Personality tests, work samples and structured interviews seem likely to offer incremental validity on mental ability tests, whereas assessment centres may not. Traditional interviews may not add much to ability tests, because the traditional interview tends to assess intellectual ability, or be strongly influenced by it.
Table 3.3 Average IQ scores of accountants and lumberjacks, and scores for highest and lowest 10%
Accountants Lumberjacks
Lowest 10%
Average
Highest 10%
Average— low
Average— highest
114 60
129 85
143 116
15 25
14 31
0470861630_04_cha03.fm Page 56 Thursday, December 2, 2004 8:31 PM
56
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Predicting Different Aspects of Work Performance Supervisor ratings make a global estimate of work performance, and can be fairly crude. Some employers have developed more precise and differentiated accounts of work performance. During the 1980s the American armed services carried out the world’s largest and most expensive selection research. They started by analysing military performance, to identify five general themes, the first four of which apply equally well to civilian employment: • • • • •
general proficiency technical proficiency effort and leadership personal discipline fitness and military bearing.
They went on to show that the two core measures of job performance, technical proficiency and general proficiency, are best predicted by general mental ability. The other three aspects—effort and leadership, personal discipline, fitness and military bearing—have less to do with mental ability, but are better predicted by personality measures. These results suggest that mental ability tests tell us what people can do, while personality tests tell us what they will do.
Mental Ability, Job Knowledge and Work Performance More sophisticated analysis shows that more able people are better workers primarily because they learn more quickly what the job is about. In high-level work this may mean learning scientific method, scientific techniques and a large body of knowledge. In low-level work it may mean learning where the stock is kept and what it costs. There seems to be no direct link between mental ability and work performance (Figure 3.2).
0.05
Mental ability
0.50
Work performance
0.45
Job knowledge
Figure 3.2 Schematic diagram showing the paths from mental ability to work performance
0470861630_04_cha03.fm Page 57 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
57
ONE ABILITY OR MANY? So far we have been talking about tests that produce a single score, for general mental ability. An alternative approach assesses from 6 to 12 abilities or aptitudes. Table 3.4 lists the eight aptitudes assessed by the Differential Aptitudes Test (DAT). Assessing eight abilities takes longer than assessing just one; administering the entire DAT will take all morning or all afternoon. In theory multiple aptitude batteries should give more accurate predictions of work performance, on the assumption that each job requires a different profile of abilities. Table 3.5 gives a fictitious example for three professional jobs. The architect needs spatial ability and some numerical ability. The accountant needs numerical ability and some verbal ability to write reports. The lawyer needs verbal ability but also some numerical ability. The multiple aptitude battery can be used to generate weighted prediction equations for different jobs, in which each aptitude score is given a different weight according to how well it predicts performance. However, it is important to remember that the abilities assessed by the DAT are highly correlated; people who get high scores on the verbal test tend to get high scores on the numerical test. Large differences between scores within one person are the exception rather than the rule. This suggests that the extra time needed to administer
Table 3.4 The eight aptitudes measured by the Differential Aptitudes Test Aptitude
Example
Verbal reasoning Numerical reasoning Abstract reasoning
Verbal analogies, e.g. cat is to kitten as dog is to . . . Simple arithmetic A series of four diagrams in which, e.g. a line is inclined in an increasing angle from the vertical, the task being to select the correct fifth diagram Reading which of five pairs of letters is underlined in the question book, and marking the same pair on the answer sheet, working to a tight time limit Understanding mechanical principles and devices Being able to say which of four boxes a two-dimensional cut-out would create Detecting spelling errors Detecting incorrect grammar or punctuation
Clerical speed and accuracy Mechanical reasoning Space relations Spelling Language usage
Table 3.5 Weights that might be given to three abilities for three professional jobs
Verbal Spatial Numerical Notes:
Lawyer
Architect
Accountant
×2 — ×1
— ×2 ×1
×1 — ×2
×2, double weighted; ×1, single weighted; —, that ability is not needed for that job.
0470861630_04_cha03.fm Page 58 Thursday, December 2, 2004 8:31 PM
58
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
the whole of the DAT may not add much to the prediction of work performance. Several large-scale analyses have generally confirmed this; we do not get a better prediction of work performance by using an aptitude battery and developing different scoring systems for different jobs. One researcher even tried using the ‘wrong’ scoring system, e.g. selecting mechanics using the electricians’ scoring, and found it gave just as good results as using the ‘right’ scoring. (Test publishers have a habit of ignoring these findings, and selling ranges of aptitude batteries allegedly tailored to different occupations, e.g. Psychological Technician Aptitude Battery, but probably achieving most of their success by measuring general mental ability. This is good marketing but not very good psychology.) Assessing multiple aptitudes may be useful sometimes, where a specific ability proves vital in a particular job. Trainee military pilots with poor visuo-spatial ability tend to fail pilot training, regardless of general mental ability. Visuo-spatial ability means seeing spatial and three-dimensional relationships, and is assessed by tests such as indicating which cubes in a 3D array numbering seven to ten cubes are touching a designated cube. Flying is very definitely a visuo-spatial task so this is not surprising. Fuel tanker drivers with poor selective attention make more mistakes, again regardless of general mental ability. Higher level jobs may be more likely to need specific abilities, so testing may sometimes need to include more than just general mental ability.
THINGS WE PROBABLY DO NOT NEED TO WORRY ABOUT There are some plausible suggestions about ability tests for which there is no evidence, so we may not need to worry about them.
Do We Need Both Motivation and Ability? It is plausibly argued that people need both ability and motivation to succeed in work; lazy geniuses achieve little, while energetic but dim people are just a nuisance. Recent research tested this for 22 different jobs and found no evidence for it (Sackett et al., 1998).
Social Class and Education Sociologists have argued that any apparent link between occupation and mental ability is a creation of the class system, or, as we would say today, is an example of social exclusion. Children from better-off homes get better education, so do better on mental ability tests, which are in any case heavily biased towards the middle classes; better-off children go on to get better-paid jobs. On this argument there is no true link between mental ability and work performance; psychological tests are merely class-laden rationing mechanisms. The social class argument is undermined by analyses that study the same group of people over time (Wilk and Sackett, 1996). Ability scores in 1980 predict whether people move up or down the occupational
0470861630_04_cha03.fm Page 59 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
59
ladder between 1982 and 1987; brighter persons move up into work of greater complexity, while less able persons move down into work of lesser complexity. This implies that the less able find they cope less well with complex work, and gravitate to work more within their intellectual grasp—something which would not happen if the test was just an arbitrary class-based way of keeping some people out of better jobs. Other research (Barrick et al., 1994) finds that lower mental ability goes with poorer job rating, which in turn goes with losing one’s job during downsizing.
Threshold or Linear? A widely held ‘commonsense’ view claims that, above a certain minimum level, most people are capable of most jobs. All that tests can accomplish is to screen out the unfortunate minority of incompetents. This view implies a threshold or step in the relation between test scores and job proficiency (Figure 3.3). An early study (Mls, 1935) found a clear break in truck driving proficiency at approximately IQ 80; any Czech soldier with an IQ over 80 was equally competent to drive a truck, while all those whose IQ fell below 80 were equally unfit to be trusted with an army vehicle. Linearity means work performance improves steadily as test score increases, with no step or threshold. The threshold versus linearity issue has important fair employment implications: • linearity implies candidates should be placed in strict order of test score, because the higher the test score, the better their job performance; • threshold implies all applicants within a broad band of scores are equally suitable, so the employer can select for diversity as well as for effective performance.
Performance
Linear
Threshold
Ability
Figure 3.3 Linear versus threshold models of mental ability and work performance
0470861630_04_cha03.fm Page 60 Thursday, December 2, 2004 8:31 PM
60
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Several large analyses of test data show test–performance relationships are generally linear. This means selecting for diversity and effectiveness may be incompatible (and implies that the Czech army truck drivers were atypical).
PROBLEMS IN THE USE OF MENTAL ABILITY TESTS Bias Item 10 in Figure 3.1 may be biased against people who earn too little to care how much tax their income attracts. Tests are usually checked for obvious sources of bias, such as gender, as part of the development process. Less obvious biases could still creep in, so it is always important to read a test carefully before using it, and consider whether any items might create a problem.
Recommendation: Before using a test, read it through carefully, and ask of every question whether it might create problems in your organisation or with the people you intend to assess. If possible, complete the test yourself.
Coaching Practice and coaching improve ability test scores by about seven or eight IQ points on the same test, and about three to four points on similar tests. This is quite enough to affect selection decisions if some candidates have had practice and coaching and some have not. Practice is a potential problem in graduate recruitment where applicants may be repeatedly tested. Large employers can avoid practice and coaching effects by using closed tests, ones they produce themselves and do not allow anyone else to use. The problem is far worse in the USA, where practice and coaching are provided on a commercial basis. One pessimistic psychologist says American employers should not use the same test more than once; if they do, ‘the only applicants who will not receive a near perfect score would be those who couldn’t afford the $1,000 . . . for a “test-preparation seminar”’ (Barrett, 1997). Apparently some US professionals will misuse their access to test material to sell test takers an unfair advantage. The same observer says even closed tests can be compromised; the first set of people to take the test conspire to reconstruct it by memorising items, then give or sell their reconstruction to subsequent intakes. European test users should be on the alert in case abuses like this cross the Atlantic.
Adverse Impact Selection tests that find large differences between key sections of the population can cause serious problems. A selection test that creates adverse impact requires careful justification, possibly in tribunal or court, which will cost the employer time and
0470861630_04_cha03.fm Page 61 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
61
money and may generate bad publicity. It is easier to use tests that do not find any differences between men and women, or whites and other ethnic groups. Differences in intellectual ability are much more controversial than differences in, for example, height. Gender differences are not generally a problem, although some research reports women score higher on verbal tests while men score higher on numerical. Age differences are probably not important within the 16–65 range. In the USA, quite large differences between white American and certain minority groups are found, which create major problems for employers. Ethnicity differences in ability tests remain a big unknown in the UK and most of the rest of Europe. Research in the Netherlands has found ethnicity differences in DAT scores, probably related to how long people have lived in the Netherlands and whether Dutch is their first language. All British and European employers would be wise to allow for the possibility of group differences. At the minimum they should keep careful records of test scores, and analyse them periodically to check whether gender or ethnicity differences are present. Recommendation: If possible, and when numbers permit, analyse your test data to see if any gender, ethnicity or disability differences are arising.
Differences between groups create major problems when using tests in selection, and many systems of using test scores have been proposed.
Simple Quota If 10% of the population are Fantasians, the employer reserves 10% of vacancies for them. This is direct discrimination, and so is illegal.
Top-Down Quota The top-down quota is a possible compromise. The employer decides what proportion of persons appointed shall come from a minority, then selects the best minority applicants even though their test scores may be lower than those of majority persons not appointed. This is effectively a formal quota for minorities, but one which selects the most able minority applicants.
Separate Norms The US Employment Service tried separate norms for white and minority Americans in the 1980s; a raw score of 300 on their test translated into a percentile of 45 for white Americans, compared with a percentile of 83 for African Americans. Separate norms have the advantage of avoiding setting a formal quota, which often proves a focus of discontent.
0470861630_04_cha03.fm Page 62 Thursday, December 2, 2004 8:31 PM
62
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Both top-down quota and separate norms represent a reasonable compromise between selecting the best applicants and ensuring a representative workforce. However, both were forbidden in the USA by the Civil Rights Act 1991. Neither system is formally prohibited in the UK but both could be viewed as direct discrimination, so are considered unsafe.
Score Bands Score banding means raw scores between, e.g. 25 and 30 are regarded as equivalent. Current banding systems are based on error of measurement. If a test has an error of measurement of, e.g. five raw score points, one can argue that all candidates within the five-point band should be regarded as equal. The employer can then give preference to minority persons within the five-point band without engaging in illegal reverse discrimination. Unfortunately, this will not look fair at all to the unsuccessful candidate who scores one point outside the band, and who does not differ reliably from most of those appointed, using exactly the same reasoning and calculation as are used to define the band.
TECHNICALITIES OF VALIDITY RESEARCH Validity research can be very misleading to the lay reader, because it sometimes seems to give depressingly poor results, suggesting tests do not predict work performance very well. This impression can be misleading, because we need to take account of the limitations of workplace selection research: • We only have successful candidates to research on. The technical term is restricted range. • We rarely have a good measure of work performance. The technical term is unreliable criterion. • We need large numbers for selection, which we rarely have. The technical term is sampling error.
Restricted Range We use the test to decide who to hire, so will tend not to hire low scorers. Suppose only the top 50% of test scorers are hired: the other 50% who are not hired cannot contribute data to the validity coefficient, because they did not get their job and we therefore have no data on their work performance. When, some time later, researchers attempt to associate high test scores with good job performance, it often comes as a surprise to HR managers that the correlation coefficient is very low. Figure 3.4 shows the effect on the validity coefficient; a clear relationship between test score and work performance is largely obscured. Validity should ideally be calculated from all applicants, but usually has to be estimated from successful applicants. Few organisations can afford to employ people judged unsuitable simply to allow psychologists to
0470861630_04_cha03.fm Page 63 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
63
Restriction of range to high test scorers
High No correlation 0.01 between performance and test scores
x x
x Performance at work
x x
x x x x x x x
Low
x
x x
x x x x
x
x
x
x
x x
x x x x x x x x x x
x x x
x x
x x
x x
x
Positive correlation with performance over whole range 0.05
Low
High Test scores
Figure 3.4 The effect of range restrictions in validity research
calculate better estimates of validity. Validity coefficients can be statistically corrected for restricted range, which usually increases them considerably.
Criterion Reliability Research on selection needs a criterion—a quantifiable index of successful work performance—which is difficult to find. Supervisor rating is the most widely used, but has fairly poor reliability; one supervisor’s opinion of John Smith will not agree all that well with another supervisor’s. If we cannot decide for sure who is and is not a good worker, it is obviously difficult to achieve highly accurate selection. An unreliable criterion necessarily reduces the correlation between assessment and criterion. Validity coefficients can be statistically corrected for unreliable criteria, which usually increases them considerably. Correcting for both restricted range and criterion reliability usually has the effect of doubling the size of the correlation and so the reported validity of the test, so if you are shown validity data it is a good idea to ask whether it has been ‘corrected’.
Sampling Error Calculating the relationship between test and work performance needs large numbers. Estimates based on small numbers can be misleading because a few unusual people can affect the results; ideally we should have data from 200 people. However, most employers do not have this many workers doing the same work. Work psychologists advise them to rely on evidence from large employers doing similar work, and not to rely on estimates from their own smaller work force, which could be misleading.
0470861630_04_cha03.fm Page 64 Thursday, December 2, 2004 8:31 PM
64
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Psychological tests are never perfectly reliable; if the same person does the test twice at an interval of three months, he/she will not give exactly the same answers and will not get exactly the same score.
CASE STUDY This case study considers an application of psychological testing in selection. The results of a set of ability scales are used, together with other information about the applicant from application form, CV, interview, references and a brief presentation.
Background to the Selection Exercise Newpharm, the pharmaceutical company based in the south east, needs to recruit a Learning Resources Officer, to promote and deliver learning and development among the 750 employees working at the production site, which also doubles as administrative and distribution headquarters. The HR Manager, John White, has prepared a person specification, shown below in the left-hand column. He next uses the person specification to construct a battery of tests and selection methods, shown in the right-hand column. Person specification
Assessment method
Physical make-up Presentable, confident appearance
Observe at interview
Education Graduate level
Job training CIPD graduate CPD in learning and development Experience 5–7 years in industrial, business learning development and training environment Ability General mental ability Verbal ability
Verify application form and CV Request certificate originals Reference check CIPD check
Reference check with last employer
Creativity
Advanced Raven Progressive Matrices Graduate Managerial Assessment—Verbal DAT language usage Guilford divergent thinking test
Special aptitudes IT skills Presentation skills Budgeting and cost analysis Driving
AC exercise on IT skills AC exercise: 10-minute presentation Graduate Managerial Assessment—Numerical Interview
[to be continued in the next chapter]
0470861630_04_cha03.fm Page 65 Thursday, December 2, 2004 8:31 PM
TESTS OF MENTAL ABILITY
65
The results are as follows: Newpharm Selection Centre for position of Learning Resources Officer Candidate: Alec Newman Assessment results Person specification
Assessment method
Result
Physical make-up Presentable Appearance
Interview
Appropriately dressed Confident approach
Education Graduate level
Application form
BA Business Studies MA in HRM
Job training CIPD graduate
CIPD check
Satisfactory
Experience 5–7 years
Reference check
Satisfactory
Ability General mental ability Advanced Raven Progressive Matrices Verbal ability Graduate Managerial Assessment—Verbal DAT language usage Creativity Guilford divergent thinking test Special aptitudes IT skills Presentation skills Budgeting and cost analysis Driving
AC exercise on IT skills AC exercise: presentation Graduate Managerial Assessment — Numerical Interview
95th percentile 70th percentile 80th percentile 60th percentile Rated 4 (out of 5) Rated 3 30th percentile Has clean licence
[to be continued in the next chapter]
The assessment data for Alec Newman indicate so far that he is generally a good match for the post of Learning Resources Officer at Newpharm, although his numerical ability, assessed to meet the budgeting and cost part of the person specification, is rather low.
0470861630_04_cha03.fm Page 66 Thursday, December 2, 2004 8:31 PM
0470861630_05_cha04.fm Page 67 Thursday, December 2, 2004 8:31 PM
CHAPTER 4
Personality Tests
OVERVIEW • There are various ways to assess personality, of which the questionnaire is the most popular. • There are various ways of writing personality questionnaires. • The ‘big five’ model describes five broad themes in personality. • Personality questionnaires have limited validity for predicting work performance. • Personality questionnaires do better at predicting other aspects of work behaviour, including effort, citizenship, leadership and career success. • Personality questionnaires are also used to screen applicants for dishonesty and other undesirable behaviours. • There is not enough research on team personality and team performance. • ‘Faking good’ is a problem with personality questionnaires, and various ways have been tried to solve the problem. • Gender differences are found in some questionnaire scores. • Some items in personality questionnaires may be considered intrusive.
INTRODUCTION ‘Personality test’ usually means questionnaire or inventory: a list of questions or statements, from 12 to 600 long. Questionnaires are very economical; instead of watching someone to see if he/she talks to strangers, which could take anything from 5 minutes to a day, we ask: ‘Are you afraid of talking to strangers?’, which takes about 15 seconds. In 15 minutes we can ask 100 questions, to 20 applicants at once. The questions can cover thoughts and feelings, as well as behaviour: ‘Do you often long for excitement?’ or ‘Do you sometimes worry about things you wish you had not done?’. This makes the questionnaire versatile as well as economical.
0470861630_05_cha04.fm Page 68 Thursday, December 2, 2004 8:31 PM
68
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Personality tests use ‘closed’ questions; the answer to ‘Do you mind being interrupted?’ has to be ‘yes’ or ‘no’, never ‘it all depends’. Personality inventories use a range of ways of phrasing their questions, various ways of interpreting the answers and two main ways of choosing the questions to ask.
Types of Questions Figure 4.1 shows the three forms of questions used in inventories: true/false (or yes/no), rating and forced choice, between equally attractive or unattractive answers.
True/false
1. I always arrive early for meetings.
true
false
2. I liked Gulliver’s Travels by Jonathan Swift.
true
don’t know
3. Do you ever tease animals?
yes
no
Forced choice
4. On your day off would you rather play football OR watch football? 5. Would you rather be Dr Crippen OR Jack the Ripper? 6. Which of these words do you prefer?
money
7. Circle the word MOST like you and Circle the word LEAST like you pushy ruthless well-liked smooth
Rating 8. I wait to find out what important people think before offering an opinion. never
occasionally
sometimes
usually
9. I like to spend my weekends with my family. very much 5 4 3 2 1
always
not at all
10. I feel tired after a day’s work. never
7
6
5
4
3
2
1
always
Figure 4.1 Three types of question in personality inventories
mystery
false
0470861630_05_cha04.fm Page 69 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
69
Interpreting the Answers Each question in the questionnaire is keyed to a personality trait. The answer true to question 1 in Figure 4.1 is keyed for conscientiousness. The first step therefore is to count how many ‘conscientious’ answers the person has given, which is the raw score. The second step is to compare the raw score with normative data. Scores on a personality inventory need to be related to a clearly defined group— people in general, bank managers, bus drivers, students etc. Good normative data are vital, and are the first key difference between a useful test and worthless imitations of a personality test. Most inventories offer some occupational data, usually for managerial or professional groups. Some inventories also have normative data based on large, representative cross-sections of the general population—16PF, Occupational Personality Questionnaires (OPQ) and California Psychological Inventory (CPI)—so they can make statements about, for example, how the person tested compares with British people in general. Various systems of interpreting raw scores are used, which have been described in Chapter 2. Some inventories such as CPI, use T scores; others such as 16PF and OPQ use sten scores.
Recommendation: Check the normative data for any inventory you might consider using. Is it relevant, up to date, and based on a large enough sample?
Choosing the Items It is very easy to assemble twenty questions, and call the result a leadership test; magazines are full of ‘tests’ like that. A proper personality test goes one crucial stage further: checking that the questions really assess leadership, and discarding the ones that do not. There are two main ways of doing this: • Empirical: the questions are included because they predict ability to lead. The inventory is empirically keyed using criterion groups of people of known characteristics. The CPI developed its Dominance scale from answers that differentiated college students nominated by their classmates as leaders from those seen as followers. Only questions that distinguish leaders from followers are kept in the final list. This empirical keying is done twice to make sure the questions in the finished test consistently distinguish leaders from followers. • Factorial: the questions have a common theme of ability to lead. Statistical analysis checks whether the items all relate to a common theme or factor. Questions that look as if they ought to assess leadership, but in fact do not, are discarded. Cattell’s 16PF research was one of the earliest uses of the factorial approach; Cattell (1965) found 16 factors or themes in his original set of questions.
0470861630_05_cha04.fm Page 70 Thursday, December 2, 2004 8:31 PM
70
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
In practice most tests use a mix of both approaches. Factorial approaches tend to be more popular because they are quicker and cheaper. However, there is no real substitute for showing that your leadership scale actually can distinguish leaders from followers.
Internal Reliability There is a third type of reliability, often calculated for personality questionnaires. Consider the statements in Table 4.1. Do they have anything in common? Do they all relate to one aspect of personality? No: they have been chosen precisely because they do not. There is no logical connection between liking taking charge of people and paying bills on time, and unlikely to be any in practice. There is unlikely to be any correlation between any of the items. The answer to one question will give us no clue to the likely answer to any of the others. Suppose we regarded the five statements as a scale that measured, for example, sales aptitude, scored the answers, and added them up to give a sales aptitude score. The resulting number would be completely meaningless. Internal reliability checks whether the items in a scale do hang together and relate to a common theme. All modern personality questionnaires have been checked for internal reliability. Home-made questionnaires may turn out to have very low internal reliability, meaning they are not really measuring much. Test publishers like internal reliability, and usually include it in their manuals, because it is easy to calculate, and because they do not need to find people prepared to complete the inventory twice.
THE BIG FIVE Different theories of personality have proposed different numbers of personality traits or factors. Hans Eysenck’s approach includes only two—extraversion and anxiety (he later added a third, ‘psychoticism’, to describe people who do not relate well to others and may be indifferent to others’ feelings)—while some versions of SHL’s OPQ assess as many as 32. Cattell is famous for his 16-factor model. Table 4.1 Set of personality inventory items and answers Statement
Answer
1. 2. 3. 4. 5.
True True True True True
I worry a lot about the future. I like taking charge of people. I like learning new facts. I like to be pleasant to people. I always pay my bills on time.
0470861630_05_cha04.fm Page 71 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
71
Table 4.2 The ‘big five’ model of personality Big five factor
Alternative names
Alternative names (opposite)
Neuroticism
Anxiety Emotionality Surgency Assertiveness Culture Intellect Likeability Friendly compliance Will to achieve Diligence
Emotional stability Emotional control Introversion
Extraversion Openness Agreeableness Conscientiousness
Dogmatism Closedness Antagonism Negligence
Recently the big five personality model (Table 4.2) has become very popular. Statistical analysis of personality data consistently finds five separate personality factors. A model that includes more than five factors will find that one or more of them overlap; thus most scales on the 16PF correlate with one or more other scales. The big five personality factors emerge from personality tests in many different cultures—the USA, the UK, Germany, the Netherlands, Israel, Russia, Japan and China—so the model has more promise of providing a culturally universal account of personality (although less developed countries are not well represented). The best known big-five questionnaire is the NEO.
Personality Profiles and Error of Difference Figure 4.2 shows a CPI profile. The profile presents all 20 scores in a way that shows whether each is above or below average, i.e. 50, and how far above or below, so it is easy to compare them at a glance. The score for Dominance is the highest, whereas the score for Socialisation is far below the average, indicating a person with a very low level of social maturity and integrity. The combination of someone very keen to exert influence on others but lacking any moral standards suggests someone liable to lapse into dishonesty. The profile belongs to the proprietor of a dating agency which was accused of taking money but failing to provide a worthwhile service. Score profiles are a neat, quick way of presenting the data, but may encourage over-interpretation—reading more into the difference between two scores than is justified. Personality tests are less reliable than ability tests; the difference between two scores is doubly unreliable, so the difference between two scores on a CPI or 16PF profile has to be quite large to merit interpretation (Figure 4.3). It would be a mistake to give someone career advice on the strength of the difference between two scores, if the difference was liable to disappear, or reverse itself, when the person did the test again.
0470861630_05_cha04.fm Page 72 Thursday, December 2, 2004 8:31 PM
72
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE 80
70
60
50
40
30
20 Do Cs Sy Sp Sa Wb Re So Sc To Gi Cm Ac Ai Ie Py Fx Fm Em In Do Cs Sy Sp Sa Wb Re So Sc To
Dominance Capacity for status Sociability Social presence Self-acceptance Well-being Responsibility Socialisation Self-control Tolerance
Gi Cm Ac Ai Ie Py Fx Fm Em In
Good impression Communality Achievement via conformance Achievement via independence Intellectual efficiency Psychological-mindedness Flexibility Femininity/masculinity Empathy Independence
Figure 4.2 CPI profile for ‘John Smith’
Taking 0.80 as a general estimate of CPI scale reliability gives a general estimate for error of difference for a pair of CPI scores of 6.3 T score points. This means that differences of more than 6 or 7 points will arise by chance in 1 in 3 comparisons, and differences of 12 or 13 points will arise by chance in 1 in 20 comparisons. If we are making an important decision based on the difference between two CPI scores, we might want a difference of 13 points or more. In Figure 4.2, ‘Mr Smith’s’ Dominance and Socialisation differ by much more than 13 points, so we can interpret the difference with confidence.
Figure 4.3 Error of difference
Recommendation: Be careful when comparing scores between personality traits, and check whether the difference is large enough to base decisions on.
USING INVENTORIES IN PERSONNEL SELECTION Opinion on the value of personality inventories in selection has shifted over time. Once they were widely dismissed as having little contribution to make. Critics
0470861630_05_cha04.fm Page 73 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
73
talked of a ‘0.30 barrier’, meaning tests never correlate with work performance better than 0.30, and so never predict more than a tenth of the variation of employee performance. Since about 1985 personality tests have grown steadily more popular, and more and more new measures have appeared. Does the evidence justify this new optimism? Inventories are used to answer four main questions: 1 2 3 4
Does the applicant have the right personality for the job? Will the applicant be able to do the job? How will the applicant behave in the workplace? Does the applicant have any problems that will interfere with work?
The first two questions look similar but are not the same; question 1 is answered by comparing bank managers with people in general, while question 2 is answered by comparing successful and less successful bank managers. Questions 1–3 are discussed in the following subsections; question 4 is discussed later, in the section on Personality Screening Tests.
Question 1: The Right Personality? Different sorts of British managers have different personalities, on average: R&D managers score higher on 16PF Conscientiousness, while finance managers score higher on 16PF Astuteness. American research finds similar differences with the Myers Briggs Type Indicator, especially Thinking/Feeling (Schneider et al., 1998). Some organisations attract, select or retain people who base decisions on logic (Thinkers), while others are largely staffed by people who base decisions more on their own or others’ feelings. Some employers seem to want a book of perfect personality profiles for managers, sales staff, engineers etc. Test manuals meet this demand to some extent, by giving normative data for different occupations. The perfect profile approach contains several pitfalls: • Most perfect profiles derive from people doing the job, taking no account of how well they do it. • A perfect profile may show how well people have adapted to the job’s demands, not how well people with that profile will do the job. • The perfect profile approach encourages cloning—selecting as managers only people who resemble as closely as possible existing managers. This may create great harmony and satisfaction within the organisation, but could make the organisation very vulnerable when faced with the need to change. Diversity of personality is often an advantage.
Recommendation: Do not assume that choosing new staff with the same personality as present staff is automatically a good idea.
0470861630_05_cha04.fm Page 74 Thursday, December 2, 2004 8:31 PM
74
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Interests and Values Career or vocational interests can be assessed by questionnaire. These are more frequently used in career counselling than in selection: the applicant assesses him/ herself to decide what jobs to apply for. Interest inventories may be less useful in selection, because they are easily faked. Extensive analysis of interest measures indicates there are six broad themes in peoples’ work interests (Table 4.3).
Question 2: Ability To Do the Job? The second, more important, question is whether inventories can select people who will do their job well. A very large number of researches have been carried out, for many jobs and many personality tests. Most recent reviews have employed the big five framework. Figure 4.4 shows that conscientiousness best predicts work performance in general. The other four of the big five—neuroticism, extraversion, openness and agreeableness—do not predict work performance to any significant extent. At first sight Figure 4.4 presents a disappointing picture. Even allowing for the limitations of selection research, personality tests cannot offer a better prediction of work performance than a correlation of 0.23. They cannot even reach the ‘0.30 barrier’, let alone break it.
Lighthouse Keepers and Holiday Reps Perhaps some jobs need extraverts, while others are better done by introverts, which we miss by pooling all types of work. We have no research on lighthouse keepers, but comparing five very broad classes of work reveals some differences:
Table 4.3 Six general themes in career interests Theme
Description
Typical careers
Realistic
Enjoys technical material and outdoor activities
Investigative
Interested in science and investigation
Artistic
Enjoys self-expression, enjoys music, drama or art
Social
Interested in helping others and work that involves other people Interested in power and political strength
Armed services Skilled trades Scientist Engineer Art teacher Music teacher Teacher Social worker Sales manager HR director Office worker Accountant
Enterprising Conventional
Likes to be well organised, happy with rules and regulations
0470861630_05_cha04.fm Page 75 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
75
(Low) Neuroticism
Factor
Extraversion Openness Agreeableness Conscientiousness 0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Correlation with work performance
Figure 4.4 Personality and proficiency at work Source:
Data from Barrick et al. (2001).
• (Low) neuroticism gives a better prediction of success in police work and skilled/semi-skilled work. (Manual jobs are often stressful, a point managers do not always appreciate! This would explain why very anxious people do them less well.) • Extraversion gives a better prediction of success in management and police work. • Extraversion and conscientiousness predict sales performance. • Agreeableness and openness give a better prediction of success for customer service work. Agreeableness seems important for work where cooperation and teamwork are important (as opposed to work which simply involves interacting with clients, e.g. hotel work.) • While conscientiousness is generally desirable in employees, research suggests it may be a drawback in some jobs, encouraging indecision and bureaucracy. Some UK data on managers find high conscientiousness scores associated with lower ratings of promotability, confirming that high conscientiousness may not always be an asset in management. The evidence we have looked at so far indicates that personality questionnaires do not seem very good at answering the question ‘Will X be able to do this job well?’. Possibly questions about ability are better answered by ability tests.
Recommendation: Do not rely on personality questionnaires to select staff with the ability to do the job well.
Question 3: Behaviour at Work There may be other aspects of work performance where personality is more relevant, and which are just as important to the organisation as proficiency. These include:
0470861630_05_cha04.fm Page 76 Thursday, December 2, 2004 8:31 PM
76
• • • • •
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
leadership dependability citizenship absence training performance.
Leadership Can personality tests help select good leaders? Reviews of research find modest relationships for (low) neuroticism and extraversion. In civilian management, openness is linked to effective leadership, while in military and government, conscientiousness is linked (Figure 4.5). This makes sense. In business a leader has to be open to new ideas, while in the government and military it may be more important to follow rules and procedures carefully. Other research has linked the big five to transformational or charismatic leadership. (Transformational leaders inspire their followers and create a vision for them to follow. This can be very important in some sorts of managers.) Figure 4.6 shows that transformational leaders are higher on extraversion, openness and agreeableness, but not (low) neuroticism and conscientiousness. The striking difference is on agreeableness: conventional leaders do not have to be likeable, but transformational leaders do need to be able to win trust and make people like them.
Business
Govt/military
(Low) Neuroticism
Factor
Extraversion Openness Agreeableness Conscientiousness 0
0.1
0.2
0.3
0.4
0.5
0.6
Correlation with leadership
Figure 4.5 Personality and leadership in business and in government/military Source:
Data from Judge and Ilies (2002).
0.7
0470861630_05_cha04.fm Page 77 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
77
(Low) Neuroticism Extraversion Factor
Openness Agreeableness Conscientiousness
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Correlation with transformational leadership
Figure 4.6 Personality and transformational leadership Source:
Data from Judge and Bono (2000).
Dependability Dependability has several strands: • Commendable behaviour is defined by letters of recommendation, letters of reprimand, disciplinary actions, demotions, involuntary terminations, and ratings of effort and hard work. • Non-delinquency means avoiding actual theft, conviction or imprisonment. • Non-substance abuse means not abusing alcohol and drugs. Dependability is sometimes called (absence of) counterproductive behaviour. Personality questionnaires predict commendable behaviour and non-substance abuse fairly well (correlations of 0.20–0.39), and non-delinquency very well (correlations up to 0.52). As might be expected, it is the conscientiousness factor in personality questionnaires that best predicts dependability at work. These results indicate that personality inventories can be genuinely useful in selection.
Citizenship Organisational citizenship means volunteering to do things not in the job description, helping others, following rules willingly and publicly supporting the organisation— all highly desirable behaviour in employees. In fact most organisations probably could not function at all without ‘citizenship’; industrial action such as ‘working to rule’ often demonstrates this. Citizenship shows modest correlations with extraversion, agreeableness and conscientiousness, giving the employer some indication of how to select better ‘citizens’.
0470861630_05_cha04.fm Page 78 Thursday, December 2, 2004 8:31 PM
78
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Absence Absence is a major problem for many employers, so the possibility of selecting applicants less likely to be absent is attractive. Research on university staff finds one ‘obvious’ result—conscientious employees were absent less, showing their ‘dutiful, rule-bound and reliable nature’—and one less obvious result—extraverts were absent more, consistent with their ‘carefree, excitement seeking, hedonistic nature’ (Judge et al., 1997). The relationship between absence and conscientiousness is small, but potentially valuable in the longer term; the employer who reduces absence by 5% by selecting highly conscientious employees could achieve major savings across a large work force.
Training Performance The openness personality factor is linked to training performance, which makes sense: open-minded people welcome the opportunity to gain new knowledge and skills. Extraversion also predicts rated training performance, which is more surprising. Perhaps extraverts like new experiences, or meeting new people, and are better able to profit from them.
Recommendation: Check the validity data for any inventory you might consider using. Is there any? What has the inventory succeeded in predicting in the way of workplace behaviour?
Can Do versus Will Do? Much of the research on personality and work can be summed up by the can do/will do distinction. Conventional measures of work proficiency emphasise the can-do side of work. This is not well predicted by personality questionnaires. Reliability, citizenship and absence, on the other hand, are will-do aspects: how motivated employees are to try hard, follow rules and systems, and simply to be there. These aspects of work behaviour are better predicted by personality questionnaires. The can-do/will-do distinction is best illustrated by the US Army’s massive testing programme (Table 4.4). Their personality questionnaire, called ABLE, predicts Table 4.4 ‘Can-do’ and ‘will-do’ at work: personality, and mental ability ABLE (personality test) ‘Will-do’ Effort and leadership Personal discipline Physical fitness and military bearing ‘Can-do’ Technical proficiency General soldiering proficiency
ASVAB (ability test)
✓ ✓ ✓ ✓ ✓
0470861630_05_cha04.fm Page 79 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
79
three will-do criteria—effort and leadership, personal discipline, physical fitness and military bearing—but not the two can-do criteria—technical proficiency and general soldiering proficiency. The two aspects of proficiency are better predicted by the Army’s ability test (Armed Services Vocational Aptitude Battery).
Recommendation: Consider using personality tests to select for motivational ‘will-do’ aspects of behaviour at work.
PERSONALITY AND TEAMWORK While there are hundreds of studies linking personality to work performance in individuals, there is a serious lack of research that has looked at teams. We have only two pieces of research on team personality and performance, one in HRM teams in an American department store chain (Neuman and Wright, 1999), and one in assembly workers (Barrick et al., 1998). These studies relate team personality to team effectiveness. Team personality can be defined simply as the average, e.g. the average of the conscientiousness scores of the individual team members. At the average personality level, team performance relates most strongly to team agreeableness, then to team conscientiousness. Note that team agreeableness is linked to team performance, even though agreeableness is not related to work performance at the individual level. In assembly work average (low) neuroticism also goes with team effectiveness (Figure 4.7). However, there is more to team personality than just its average. An average level of agreeableness could mean that everyone is averagely agreeable, or could conceal a mixture of very agreeable and very disagreeable people. We need to look at variation within the team, and especially at very high or low scores. In assembly work, teams
HR
Assembly
(Low) Neuroticism
Factor
Extraversion Openness Agreeableness Conscientiousness 0
0.1
0.2
0.3
0.4
0.5
0.6
Correlation with team performance
Figure 4.7 Personality and team performance in HR and assembly work Source:
Data from Barrick et al. (1998).
0.7
0470861630_05_cha04.fm Page 80 Thursday, December 2, 2004 8:31 PM
80
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
with no low scorers on extraversion, conscientiousness or agreeableness performed better. One disagreeable person in the team reduces its effectiveness, as does one introvert or one person low in conscientiousness. Where this happens, testing personality may be able to make a far bigger contribution to productivity. Work psychologists now have a full agenda of further research, to find out where extreme scores matter, and why. Another approach to teamwork argues that certain combinations of personalities create a more effective team. The Belbin Team Role Self Perception Test assigns people to one of eight team roles, e.g. shaper or company worker, and argues that an effective team needs the right mix of roles. If everyone wants to take charge and no one wants to be a ‘company worker’, the team will fail. Preliminary research has shown that balanced teams are more effective.
Recommendation: Where teamwork is important, consider the personality of the team as a whole.
CAREER SUCCESS Some research has linked the big five to career success in managers. Career success is defined by salary level and by status—the number of rungs of the corporate ladder between you and the Managing Director. Career success reflects performance over a much longer time span than conventional indices of job proficiency. A large survey in America and Europe finds some differences between the two sides of the Atlantic (Boudreau et al., 2000). Career success in the USA is linked to low neuroticism and low agreeableness, but not to conscientiousness. In Europe, however, high extraversion and low agreeableness go with career success, but again not with conscientiousness (even though conscientiousness is the best predictor of work performance overall) (Figure 4.8). USA
Europe
(Low) Neuroticism
Factor
Extraversion Openness Agreeableness Conscientiousness –0.5
–0.3
–0.1
0.1
0.3
Correlation with career success
Figure 4.8 Personality and career success in Europe and the USA Source:
Data from Judge et al. (1999), Boudreau et al. (2000).
0.5
0470861630_05_cha04.fm Page 81 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
81
The most convincing study is a long-term follow-up in California (Judge etal., 1999). Personality was assessed in childhood and adolescence, then related to occupational status and income when the people had reached their fifties. Conscientious and extravert people were more successful, while anxious people tended to be less successful. The four factors combined correlated 0.54 with career success, suggesting personality is definitely important for success in work. All the research finds that less agreeable people were more successful, suggesting perhaps that it does not pay to be too nice to others if you want to get ahead. Openness seems completely unrelated to career success. The studies differed on the role of conscientiousness in career success; the study of mature managers found no link, but the long-term follow-up found a strong link. Perhaps conscientiousness is important at earlier stages of a manager’s career.
INCREMENTAL VALIDITY Two analyses suggest personality inventories will have considerable incremental validity over mental ability tests (Salgado, 1998; Schmidt and Hunter, 1998). (Recall that incremental validity means that a second test improves on the prediction made by a first test, which will not happen if both tests measure the same thing.) Both conscientiousness and mental ability are related to work performance, but not to each other, so it follows logically that conscientiousness will have incremental validity on mental ability (and vice versa). Conscientiousness should increase validity by around 18%, making the combination of conscientiousness and mental ability one of the more useful. Review of European data confirms this, and concludes that neuroticism too will have incremental validity.
Recommendation: Try to check the incremental validity data for any inventory you might consider using. Will the inventory add anything to your other selection methods?
PERSONALITY SCREENING TESTS Question 4: Does the Applicant Have Any Problems that Will Interfere with Work? Personality questionnaires can be used like the driving test: not to select the best, but to exclude the unsuitable: • In the USA personality inventories are widely used to screen out candidates who are psychologically unfit for police work. • Many American employers try to screen employees for dishonesty. • Many American employers also try to screen employees for violent or criminal tendencies in case they cause harm to the organisation’s customers.
0470861630_05_cha04.fm Page 82 Thursday, December 2, 2004 8:31 PM
82
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
• American employers once tried to use personality tests to screen out union activitists—‘thugs and agitators’—working on the curious assumption that joining a union was a sign of poor adjustment. • British Government reports, the Walner Report and the Waterhouse Report, have recommended psychological testing for anyone whose work brings them into contact with children, to identify possible child abusers. Personality questionnaires cannot detect sexual interest in children specifically, but could screen out poorly adjusted persons who may be less able to resist deviant impulses.
Honesty Testing Surveys indicate that employee theft is a widespread problem, which may contribute to 30% of company failures in the USA. Questionnaire-format honesty tests have become popular in the USA in recent years, to try to screen out potential thieves, especially since use of the polygraph or lie detector was restricted. A massive analysis of 665 honesty test studies in the North America, covering over 500 000 persons, produced some surprising results: • Honesty tests work: they predict dishonesty fairly well (a correlation of 0.39). • Honesty tests also predict general job proficiency fairly well. • Mainstream psychological tests are no more successful than specialist honesty tests. These results are disturbing for professional psychologists, who have been warning the public for years that writing psychological tests is a specialised task best left to the experts, i.e. psychologists. • Honesty tests work for all levels of employee, not just for lower paid workers, as is sometimes supposed. Critics express some cautions about honesty testing: • Only a few tests have been researched thoroughly; the rest remain an unknown quantity. • Many studies are carried out by the test’s publisher, which tends to carry less weight than work by independent researchers. • Most studies use definitions of dishonesty that could be questioned. Many define theft by self-reports of past theft, which supposes that thieves will be honest about stealing. Other infer theft from stock ‘shrinkage’ or lower than expected takings. Only seven studies, with a total of 2500 persons, used actual theft as the outcome predicted, and these achieved much poorer results. (Honesty testing raises some interesting questions. Is detected theft really the ideal criterion? One could argue that only incompetent thieves are found out, and that what employers really want is a test that will identify the clever thieves who do not get caught.)
0470861630_05_cha04.fm Page 83 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
83
Problems with Screening False Positives Honesty testing generates a high rate of false positives: people identified as dishonest by the test, who do not commit any dishonest acts (Figure 4.9). Screening’s other failure is the false negative—dishonest people whom the test fails to identify. Some critics think false positives present a strong argument for not using honesty tests, because a false positive means an honest person is wrongly accused of dishonesty. False positives will always happen because no test will ever be perfectly accurate. Other experts argue that the false positive issue is irrelevant; if the test works at all, then using it will benefit the organisation by excluding some dishonest persons. Not screening implies accepting more dishonest persons, and so excluding some honest individuals who might otherwise have got the job. On this argument, screening by and large benefits honest applicants. Other critics argue that honesty tests may be unfair to certain groups of people. A reformed criminal who admits to past dishonesty is likely to fail an honesty test and be deprived of employment. Honesty tests that reject people who look ‘too good to be true’ may unfairly reject people with very high moral and/or religious standards.
Recommendation: Think carefully before introducing honesty tests. Examine the evidence that the test can detect dishonest applicants. Ask who will ‘fail’ the test, and why, and what they will think of your organisation.
Person Dishonest
Honest
True positive
False positive
Honest
False negative
True negative
Test result
Dishonest
Figure 4.9 The four possible outcomes of a screening test
0470861630_05_cha04.fm Page 84 Thursday, December 2, 2004 8:31 PM
84
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
THE PROBLEM OF FAKING Personality inventories are not tests but self-reports; a test has right and wrong answers but a self-report does not. Many inventories look fairly transparent to critics, who argue no one applying for a sales job is likely to say true to I don’t much like talking to strangers. Personality inventories are fakable, in the sense that people directed to maximise their chances of getting a job do generate much ‘better’ profiles. These are laboratory experiments using students: is faking good a problem in real-life selection? Early research—widely quoted—thought not: only a minority of real applicants fake good, and even they faked less than people in laboratory directed faking research (Dunnette et al., 1962). These comforting conclusions were based on very few studies, with very small samples, which identifies this as an under-researched issue. There are many lines of defence against faking. Some try to modify the way the test is administered: • Rapport. The tester tries to persuade applicants it is not in their interests to fake, because getting a job for which one’s personality is not really suited may ultimately result in unhappiness, failure etc. This argument may not have much impact on applicants who are desperate to get any job, ideal or otherwise. • Faking verboten. At the other extreme, military testers have sometimes warned people that faking will be severely punished. • Faking will be found out. Faking can be reduced by warning people that it will be detected, and may prevent them getting the job. This approach could create problems if used in routine selection. • Faking will be challenged. When applicants were told they would be required to defend their answers in a subsequent interview, faking was dramatically reduced. If the selection really includes such an interview, no unethical deception is involved. • Administer by computer. People are more honest about their health when giving information to a computer, not a person. However computer administration of personality tests does not reduce faking good. Other approaches alter the way the test is written.
Subtle Questions ‘Do you have confidence in your own abilities?’ clearly seems to suggest a ‘right’ answer when selecting managers. But what is assessed by question 2 in Figure 4.10, about conserving energy supplies? And what is the ‘right’ answer to give? Some test authors argue that you cannot work out what their items are assessing, and that even when you are told you cannot work out what answer gets a ‘better’ score. The energy conservation question is part of a dominance scale, but what is the ‘dominant’ answer? Critics say inventory questions can be divided into the unsubtle that work, and the subtle that do not.
0470861630_05_cha04.fm Page 85 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
85
Forced Choice Applicants must choose between pairs of equally flattering or unflattering statements (questions 3 and 4 in Figure 4.10). Some recent research suggests forced choice may solve the faking good problem (Martin et al., 2002). For example students can fake a manager profile on the rating form of SHL’s OPQ, but not on the forcedchoice OPQ. However, forced choice often is not suitable for selection tests. Within Subtle questions
1. I sometimes have trouble remembering where I put things. 2. We should conserve energy supplies so there will enough left for future generations.
Forced-choice questions
3. Which are you better at doing: meeting new clients OR generating new ideas? 4. Which would you rather do: tell someone they are being terminated OR fill in your income tax return?
Lie scale questions
5. I have never taken anything, however small or unimportant, that did not belong to me. 6. I never discuss other people behind their backs.
Defensiveness scale questions
7. I sometimes have strange or peculiar thoughts. 8. There are days when I just can’t get started.
Right/wrong answer questions
9. Which of the following is not a type of firearm: Gatling
Sterling
Gresley
Enfield
FN.
10. What is the likelihood of catching rabies in Britain? 1 in 10000000
1 in 100000
1 in 1000
1 in 100
Figure 4.10 Types of questions used to try to deal with faking good in personality questionnaires
0470861630_05_cha04.fm Page 86 Thursday, December 2, 2004 8:31 PM
86
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
each forced choice, one answer, for example, scores for extraversion, while the other scores for anxiety. This way of scoring means it is impossible to get all very high scores or all very low scores (whereas this is entirely possible on the 16PF, for example). With forced choice, the person who wants to express a strong preference for one trait must express less interest in another. Forced-choice measures cannot usually conclude that person A is more interested in dominating others than is person B—but this is the comparison selectors usually want to make. The typical forced-choice measure can conclude that a person is more interested in dominating others than in helping others, which is useful in career or development counselling, but not in selection.
Recommendation: Look carefully at forced-choice questionnaires and check whether the way they are scored allows you to compare one person with another.
Control keys Many inventories contain lie scales or faking good scales, more politely known as social desirability scales (questions 5 and 6 in Figure 4.10). Lie scales are lists of questions that allow people to deny common faults or claim uncommon virtues. A high score alerts test users to the possibility of less than total candour, but then presents them with a difficult decision: to discard the applicant altogether or to rely on other evidence. Discarding applicants is not always possible in practice. Coaching in how to complete personality tests, which is widely sold in the USA, helps people avoid being detected by faking good scales.
Recommendation: Check whether the questionnaire has a social desirability or lie scale. Do not use a questionnaire for selection unless it has one.
Correction for Defensiveness Some questionnaires use special scales to try to correct the rest of the person’s profile. The MMPI’s K key assesses defensiveness, then adds varying proportions of the K score to other scores to estimate the shortcomings people would have admitted if they had been more frank. A critic noted that ‘if the subject lies to the tester there is no way to convert the lies into truth’.
Change the Questionnaire into a Real Test Cattell’s Motivation Analysis Test uses several novel question formats, which are probably quite hard to fake. The Information subtest (question 9 in Figure 4.10)
0470861630_05_cha04.fm Page 87 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
87
consists of factual questions, with right and wrong answers, on the principle that people know more about things that matter to them; an aggressive person may know more about firearms. The Estimates subtest (question 10 in Figure 4.10) assumes that people’s needs and motives shape their perceptions, so a fearful person may overestimate the risk of disease.
Does Faking Affect Validity? It seems intuitively obvious that untrue answers will not predict work performance, but it has been claimed that faking does not reduce validity. However, recent studies cast doubt on this comforting conclusion (Ellingson et al., 1999; Hough, 1998). • Some applicants fake a little, some fake a lot, and some not at all, so faking good is not a simple constant error that can be subtracted to reveal the true profile. • Where only a small proportion of applicants are appointed, they tend to be the ones who faked good most. So knowing that most applicants tell the truth is not very comforting.
Can Correction Keys Restore Validity? One researcher obtained both faked and unfaked personality questionnaires from the same people, so could answer the question: do correction keys succeed in turning lies back into truth? At one level the answer was ‘yes’. Faked tests were ‘better’ on average than unfaked, but the corrected faked ones were no ‘better’ on average than the unfaked. At the individual level, however, the answer was ‘no’. Faking changed the rank order of candidates (compared with a rank order based on unfaked completion), and so changed who got the job. Correcting for faking did not change the rank order back to the unfaked order, so did not ensure the ‘right’ people were appointed after all.
Questionnaire Scores as Hypothesis The test user should regard the scores the questionnaire generates as nothing more than hypotheses, that need further information to confirm them: • In counselling or development, the hypothesis that Smith would make a good line manager should be tested against Smith’s own view, and against Smith’s own account of his/her success at managing people or projects in the past. • In selection, the risk of faking good makes it especially important to seek out confirmation of, for example, whether the person is original and flexible, from past achievements, from what referees say, or from what Smith does in the group discussion or the written exercise of an assessment centre.
0470861630_05_cha04.fm Page 88 Thursday, December 2, 2004 8:31 PM
88
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Recommendation: Never rely on inventory scores alone to make important personnel decisions.
INVENTORIES AND THE LAW Personality inventories have encountered surprisingly little trouble with the law, at least compared with ability tests.
Gender Differences Early versions of the Strong Interest Inventory had separate male and female question books, appropriately printed on pink or blue paper. Two American Civil Rights Acts later, gender differences still create potential problems for the occupational tester. There are gender differences in many inventory scores. Men score higher on scales assessing forcefulness, while women score higher on scales assessing responsibility. (Note that we say differences in inventory scores. Whether men ‘really’ are more forceful and women ‘really’ more responsible is another question, well beyond the scope of this book. Inventories may reflect differences in how men and women see themselves. For example, it is commonly argued that men are more reluctant to admit weaknesses, so try to appear tougher.) Because gender differences are found on many scales, most questionnaires used to provide separate normative data for male and female. Today most questionnaires also provide pooled gender norms, which do not distinguish male from female. Gender differences create a dilemma for the occupational tester. Using separate norms for men and women is illegal in the USA, and thought inadvisable in Britain, because calculating scores differently for men and women could be classed as direct discrimination. On the other hand, using pooled gender norms may create adverse impact. For example, using a cut-off score of 60 on CPI Dominance to select managers would exclude more women than men. On this analysis, there is no safe way to use personality tests in selection. In practice there have not been many complaints about gender and personality tests, because the male–female differences are relatively small, and because personality tests are not usually used with mechanistic cut-offs. It is in any case not good practice to base selection decisions on test data alone.
Ethnicity Analyses of differences between ethnic groups in the USA suggest personality tests will not create the major adverse impact problems found with tests of mental ability. There do not seem to be large consistent differences between white and other Americans in personality questionnaire scores. Evidence for Europe is less definite.
0470861630_05_cha04.fm Page 89 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
89
Quite large differences in neuroticism and extraversion are found between some immigrant groups in the Netherlands and the native Dutch. British research finds few differences between white and Afro-Caribbean, while Asian and Chinese differences generally ‘favoured’ the minorities, showing them to score as more conscientious than white British (and so more likely to be selected). However, this research used college students, who may not be typical of people in general.
Recommendation: Be prepared for the possibility of gender and ethnicity differences. Check your data for gender and ethnicity differences as soon as you have enough.
The Potential Danger of Multi-Score Inventories Most questionnaires come as fixed packages. The 16PF assesses all 16 factors every time it is used; the test user cannot decide to use only dominance, ego strength and shrewdness, or to leave out suspicion and guilt proneness. This creates a big danger for selectors. Suppose your job analysis has identified dominance, ego strength and shrewdness as the (only) personality characteristics needed for the job. The other 13 factors on 16PF are not job related, and should not be used to make the decision. But having the other 13 scores in front of them, the selection team may well use them anyway, which could create serious problems for the employer. Someone complains about the selection, and finds that scores were used which were not job related; the employer cannot justify their use. Inventories that describe people’s weakness and poor adjustment can be particularly dangerous: someone applying for a clerical job will be justifiably annoyed to find their level of fantasy aggression or self-doubt has been assessed.
Recommendation: Do not assess aspects of personality that you do not need to (and rely on a good job analysis to tell you if you need to).
OTHER PROBLEMS WITH PERSONALITY INVENTORIES Lack of Insight The person completing the questionnaire may describe him/herself as very dominant, when by any other evidence he/she is rather ineffectual, not because he/she is consciously faking good, but because he/she really sees him/herself that way. We have no systematic data on how many people lack self-insight in this way.
0470861630_05_cha04.fm Page 90 Thursday, December 2, 2004 8:31 PM
90
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Providing a Work Context? Inventories usually assess personality in general, which may be broader than is needed in selection. We do not need to know what someone is like at home, and it might be considered an intrusion to ask. An experimental version of the NEO inventory provides a context by inserting ‘at work’ into every item: ‘I take great care with detail at work’. This should make the test more accurate for selection. A pilot study on airline customer service supervisors showed that the ‘contextualised’ NEO did give a better prediction of work performance (Robie et al., 2000). However, setting NEO in a work context also resulted in higher scores—perhaps because people really are more careful at work—so new normative data will be needed or we risk overestimating conscientiousness.
Invasion of Privacy By their very nature, personality tests tend to invade privacy; it is difficult to assess personality without asking personal questions. In the Soroka v. Dayton-Hudson case the applicant claimed the combined MMPI and CPI inventory asked intrusive questions about politics and religion, contrary to the Constitution of the State of California; the case was settled out of court so no definitive ruling emerged. Another Californian case, Staples et al. v. Rent-a-Center, also complained of intrusive questions in the MMPI, and about computer generated reports that made ‘gross and unfounded generalisations’, e.g. ‘[the candidate] tends to be restless and impatient, should reduce caffeine and nicotine consumption and drink more water’. Employers need to read thoroughly any questionnaire they plan to use, and ask themselves ‘Would I be unhappy about answering any of these questions?’.
Recommendation: Before you use any personality test, read it all the way through, and check for items that might offend anyone.
The ‘Method Variance’ Problem Table 4.5 shows the results of assessing people with the 16PF, by conventional questionnaire and by ratings by others who know them well. The results are fairly worrying: • Warmth assessed by questionnaire and by other’s opinion should agree, but correlate only 0.11—hardly at all. The correlations in bold in Table 4.5 are for the same trait measured two ways; some are quite low. • Liveliness and vigilance assessed by self-report should correlate less well because they are different traits, but Table 4.5 shows they correlate very well— 0.79. The correlations in italic in Table 4.5 are for different traits measured by questionnaire; some are quite high.
0470861630_05_cha04.fm Page 91 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
91
Table 4.5 Data for four personality factors, assessed by questionnaire and rating Questionnaire A
F
H
Questionnaire A—warmth F—liveliness H—social boldness L—vigilance
0.44 0.43 0.40
0.78 0.79
0.71
Rating A—warmth F—liveliness H—social boldness L—vigilance
0.11 0.18 0.16 0.27
0.26 0.49 0.60 0.30
0.28 0.53 0.59 0.19
Source:
Rating L
A
F
H
0.29 0.29 0.52 0.25
0.14 0.40 0.25
0.49 0.03
0.29
L
Data from Becker (1960).
Questionnaire measurements of different traits often correlate very highly; similarly ratings of different traits often correlate very highly. This effect—called method variance—means that assessors should ideally measure every trait by two different types of measure: multi-trait multi-method measurement. This is not always easy in practice, and does not always seem to work, as research on assessment centres (Chapter 8) has found.
SURVEY OF (SOME) INVENTORIES Inventories are listed and reviewed in the US Mental Measurements Yearbooks (MMY). MMY reviews discuss size and adequacy of standardisation sample, reliability, validity and whether the manual is adequate. MMY reviews are very critical even of major tests. The housing sector has recently published a guide to tests suitable for selection in the UK (Howard, 2003). • 16 PF. A well-known test that assesses 16 personality factors, the 16PF does not fit within the ‘big five’ framework. The current edition of the 16PF—16PF5— has good internal and re-test reliability, as well as good US and UK normative data. 16PF5 is not all that similar to earlier versions of 16PF, and should not be regarded as interchangeable with them. • California Psychological Inventory (CPI). The CPI assesses 20 main personality traits. The CPI is the only major inventory in current use that relies primarily on empirical keying rather than factor analysis. The CPI does not fit within the ‘big five’ framework. The CPI has recent British normative data. • Occupational Personality Questionnaires (OPQ). A family of inventories, varying in length and format. The longest versions (OPQ32) measure 32 aspects of personality; the shorter Factor, Octagon and Pentagon versions measure 14, 8 and 5 aspects, respectively. OPQ uses three formats: endorsement, forced choice and five-point rating. The OPQ has good British normative data.
0470861630_05_cha04.fm Page 92 Thursday, December 2, 2004 8:31 PM
92
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
• NEO. The short NEO-FFI measures the ‘big five’, using 12 items per factor with a five-point rating format. The longer NEO-PI also measures 6 facets within each main factor, and contains 240 items. The NEO lacks any social desirability scale, and works on the assumption that people will tell the truth about themselves; they may if they are seeking self-enlightenment, but may not if they are seeking employment. • Hogan Personality Inventory. The HPI has 206 endorsement format items. The HPI uses an extended big-five model that divides extraversion and openness into two factors each. • Global Personality Inventory. The GPI measures the big five and 32 facets. GPI was simultaneously written in 14 countries—the USA, the UK, France, Belgium, Sweden, Germany, Spain, Norway, China, Japan, Singapore, Korea, Argentina and Colombia—to ensure item content was truly pan-cultural. The detailed development was performed in parallel in English, Spanish and Chinese. • Eysenck Personality Questionnaire. Hans Eysenck developed the Eysenck Personality Scales over 40 years. The research started in 1952 when he worked at the Maudsley Hospital and developed the Maudsley Medical Questionnaire. This was a 40-item measure of N (neuroticism or emotionality). Other scales were added over the 40 years, E (extraversion–introversion), L (the ‘lie’ scale to measure dissimulation, which we have called social desirability, in line with other instruments), P (psychoticism, more suitably renamed tough-mindedness), empathy, impulsiveness and venturesome. We have designed an interpretive proforma for occupational application, which covers all seven scales and includes the ‘lie’ scale as a measure of social desirability. • Goldberg’s item bank. Lewis Goldberg offers a bank of personality questionnaire items that can be combined in different sets to assess the big five or to provide analogues to many popular inventories. Goldberg’s item bank has an unusual feature: it is in the public domain, and can be used without restriction or payment by anyone. An item bank enables the selector to assess only those attributes identified as relevant by job analysis. Goldberg’s item bank is accessible through his IPIP website, the address of which is given on p. 95. As Goldberg notes, people should not use personality questionnaires unless they are competent to do so.
ALTERNATIVES TO THE INVENTORY The personality inventory is quick and (usually) fairly cheap, but has problems with faking, and with method variance. What other methods of assessing personality can the selector use? Other formal personality tests include projective tests, laboratory tests and archive data.
Projective Tests What are X’s motives, complexes and defences? Projective tests assume that people project their personality into what they see in drawings or inkblots, or how they complete stories. Projective tests are supposed to get around people’s defences, and
0470861630_05_cha04.fm Page 93 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
93
Figure 4.11 Drawing similar to those used in the Thematic Apperception Test
so prevent faking. The Thematic Apperception Test (TAT) is a set of pictures carefully chosen both for their suggestive content and their vagueness (Figure 4.11). The applicant describes ‘what led up to the event shown in the picture, what is happening, what the characters are thinking and feeling, and what the outcome will be’, and projects into the story his/her own ‘dominant drives, emotions, sentiments, complexes and conflicts’ (McClelland, 1971). The TAT is intended to assess motives, such as the need to achieve, or the need for other peoples’ approval and presence. The Rorschach test uses selected inkblots (Figure 4.12), which people can
Figure 4.12 Inkblot similar to those used in the Rorschach test
0470861630_05_cha04.fm Page 94 Thursday, December 2, 2004 8:31 PM
94
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
interpret in psychologically meaningful ways. Projective tests require considerable skill and experience to use and score. Neither TAT nor Rorschach is very successful as a selection test. The Defence Mechanism Test (DMT), devised by Swedish psychologists, also uses ambiguous drawings containing threatening images, and makes them more ambiguous still by showing them for only a split second. The DMT is scored for a whole range of defence mechanisms, such as denial (failing to see the threatening image) or repression (seeing the threat as something non-threatening). This test has proved quite successful in selecting military pilots, perhaps because people whose minds are full of conflict and defence cannot think fast enough to fly military aircraft.
Laboratory Tests How does X react physically to threat? ‘I shall count to ten. When I get to ten you will receive a painful electric shock.’ Most people start getting physically anxious by about number four; psychopaths do not react much, if at all (Figure 4.13). These reactions are easy to measure and hard to fake, so employers could use the threatened shock test to screen out psychopaths, who are likely to steal, attack people verbally or physically, break rules etc. This test clearly is not a practical proposition because threatening applicants with electric shocks is completely unacceptable. A less obvious problem is a high false positive rate. The shock test picks out psychopaths quite well, but also wrongly identifies as psychopaths many people who are not psychopathic.
Archive Data What do the files say about X? When deciding whether it is safe to release violent offenders from prison, information from school records about disruptive behaviour
Psychopaths
Non-psychopaths
16 14 12 10 8 6 4 2 0
1
2
3
4
5
6
7
8
9
10
Figure 4.13 Physical reactions of psychopaths and non-psychopaths to the threat of an electric shock
0470861630_05_cha04.fm Page 95 Thursday, December 2, 2004 8:31 PM
PERSONALITY TESTS
95
in childhood helps make accurate predictions, and is quite unfakable. Most prospective employers would probably find some useful information in previous employers’ files on candidates: attendance, sickness, performance appraisals etc. However, concerns over privacy make it unlikely this sort of information could be used in selection decisions. Personality can also be assessed by selection methods reviewed in other chapters: • • • •
Interview: how does X perform in interview? (Chapters 9 and 10). Peer ratings and references: what do other people think of X? (Chapter 6). Biodata: is X the sort of person who can sell life insurance? (Chapter 5). Situational/behavioural tests: can X exert influence in a group discussion? Assessment centres use group discussions and role plays to assess how people relate to each other, perform in groups etc. (Chapter 8).
USEFUL WEBSITES http://ipip.ori.org/ipip Goldberg’s personality item bank. www.reidlondonhouse.com Leading US publisher of honesty tests. www.parinc.com Psychological Assessment Resources Inc., US publisher of personality tests.
CASE STUDY This case study considers an application of psychological testing in selection. The results of a set of personality scales are used, together with other information about the applicant from ability tests, application form, CV, interview, references and a brief presentation. The ability test data were reviewed in the case study in Chapter 3.
Background to the Selection Exercise Newpharm, the pharmaceutical company based in the south east, needs to recruit a Learning Resources Officer, to promote and deliver learning and development among the 750 employees working at the production site, which also doubles as administrative and distribution headquarters. The person specification prepared by HR Manager John White (see Chapter 3 case study) is now extended as follows.
0470861630_05_cha04.fm Page 96 Thursday, December 2, 2004 8:31 PM
96
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Person specification
Assessment method
Relations with others Self-awareness Empathy
Bar-On Emotional Quotient Inventory—intrapersonal Bar-On Emotional Quotient Inventory—interpersonal
Interests Outside interests involving people interview personality Extravert Sociable Calm and self-controlled Confident, free of anxiety
Eysenck PQ—extraversion Eysenck PQ—sociability Eysenck PQ—impulsiveness (low) Eysenck PQ—emotionality (low)
Circumstances Able to live close (20 miles or so) Interview Travel in UK and abroad Interview
The results are as follows:
Newpharm Selection Centre for position of Learning Resources Officer Candidate: Alec Newman Assessment results Person specification
Assessment method
Result
Relations with others Self-awareness Empathy
Bar-On EQI—intrapersonal Bar-On EQI—interpersonal
50th percentile 60th percentile
Interests People
Interview
Wide range of social activities
Personality Extravert Sociable Calm and self-controlled Free of anxiety
Eysenck PQ—extraversion Eysenck PQ—sociability Eysenck PQ—impulsiveness Eysenck PQ—emotionality
75th percentile 60th percentile 15th percentile 20th percentile
Circumstances Able to live close Travel in UK and abroad
Interview Interview
Satisfactory Satisfactory
Sociable and caring for people in the organisation Fits readily into the team, not a troublesome person Likes to be liked and accepted by others
The typical introvert is a quiet, retiring sort of person, not standing out in the workplace May prefer to work alone Tends to withdraw from stimulation
Not anxious Optimistic Doesn't worry unduly Emotionally stable Controlled
Will look ahead at the consequences of actions Often will not take a risk Is aware of the consequences of risk taking and the nature of risk
Cautious and unlikely to take a risk Often unaware of the nature of risk Not sensation seeking
Low in awareness and understanding of the emotions and feelings of others An inability to share in the emotions of others An inability to take on the perspective of another person
Has not attempted to distort the results in anyway Little attempt to reflect socially desirable responeses
Figure 4.14 Alec Newman’s personality assessment
• •
Low Social Desirability
• •
•
Low Empathy
• • •
Low Venturesome
• • •
Low Impulsive
• • • • •
Stable
• •
•
Introvert
• • •
Tender Minded
Name ................................................................... More like this.......
Alec Newman
Answers form colours: PINK and BROWN
1 •
2 •
5 •
5–6
5 •
5–6
5 •
4–5
5 •
6
5 •
2
5 •
4 • 2
0–1
3
5 •
10–11 12–13
4 •
3–4
4 •
3–4
4 •
3
4 •
4–5
3 •
9
0–6 7–8
1–2
0
3 •
3 •
1–2
0
2 •
3 •
1–2
0
2 •
3 •
2 •
1–3
0
4 •
1
0
3 •
4 •
3 •
2 •
2 •
2 •
1 •
1 •
1 •
1 •
1 •
1 •
X
7 •
7–8
7 •
9–10
7 •
4
7 •
8 •
9–10
8 •
11
8 •
5
8 •
9 •
11
9 •
12
9 •
6
9 •
10 •
12
10 •
10 •
7–12
10 •
7 •
8 •
9 •
4–5
6 •
14
6 •
6
7 •
15–16
7 •
7
8 •
17
8 •
8
9 •
18
9 •
7–9 10–11 12–13 14–15
6 •
9–12
10 •
19
10 •
16
10 •
7–9 10–11 12–13 14–15 16–19
6 •
6
6 •
7–8
6 •
3
6 •
Short Scale + IVE Profile Chart
DR BARRY CRIPPS & PARTNER
•
An awareness and understanding of the emotions and feelings of another person An ability to share in the emotions of others An ability to take on the perspective of another person
•
•
Lie
Emp
Ven
Imp
Neu
Ext
Psy
May have disorted results in order to present a positive image May have responded unconsciously in a socially desirable way
Social Desirability
• •
•
Exhibits risk taking behaviour Knows fully that there is a risk involved Sensational seeking
Empathetic
• • •
A complete lack of looking ahead at the consequences of actions Takes a risk readily Unaware of the nature of risk
Venturesome
• •
•
Anxious and worrying Pessimistic A worrier Exhibits emotional swings Sometimes loses control
Impulsive
• • • • •
The extravert tends to make themselves noticed in the work place Does not enjoy working alone Seeks stimulation
Emotional
• •
•
Extravert
•
•
Often alone in the organisational and not overly concerned with people. Can be trouble some in a team, may be a cause of friction in the team Unconcerned about how others view them may like to upset others
Tough Minded
01 06 04 Date ............................................ More like this.....
OCCUPATIONAL APPLICATIONS
0470861630_05_cha04.fm Page 97 Thursday, December 2, 2004 8:31 PM
0470861630_05_cha04.fm Page 98 Thursday, December 2, 2004 8:31 PM
98
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
On the EPQ Alec Newman reports himself to be a stable-extravert. Sociable, caring for people, he should fit into a team. As an extravert, Alec will make himself known, seek challenge and enjoy change. He is not particularly impulsive but may take a risk at times. He can empathise with others, not overly so. His social desirability score indicates an honest and open approach to this questionnaire (Figure 4.14). The assessment data for Alec Newman indicate that he is generally a good match for the post of Learning Resources Officer at Newpharm, although his numerical ability, assessed to meet the budgeting and cost part of the person specification, is rather low.
0470861630_06_cha05.fm Page 99 Thursday, December 2, 2004 8:32 PM
CHAPTER 5
Sifting and Screening
OVERVIEW • Sifting means sorting applications into proceed or reject; sifting is often done inefficiently or unfairly. • Conventional paper application methods can be improved. • Computers and the Internet may greatly change the application process. • There are two biographical approaches: weighted application blanks, which are scored from the application form, and biodata, which are stand-alone questionnaires. • We describe the main stages in generating a biographical assessment. • We compare different approaches to devising biographical questions. • Biographical measures can predict work performance, and also tenure, training success, promotion, absence and creativity. • Biodata can be purely empirical, or can be guided by either a theory of e.g. eliteness motivation, or a relevant construct such as stress tolerance. • Biodata may be compromised by impression management or outright faking. • Biodata rarely seem to attract litigation, but can look unfair or arbitrary to candidates. • Biodata do create some adverse impact on American minorities.
INTRODUCTION The focus of this chapter is getting information about the applicant, mostly from the applicant, via paperwork the applicant completes, rather than by interview, tests etc. This chapter covers in particular: • traditional application forms, CVs/résumés; • online application systems; • biodata.
0470861630_06_cha05.fm Page 100 Thursday, December 2, 2004 8:32 PM
100
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
These approaches have one feature in common—they act as a first sift, and are used to decide who to assess in more depth. As the employer may need to process thousands of applications, sifting needs to be quick—but also to have some accuracy. There are three main issues: • what information to collect; • how to collect it, on paper or online; • what to do with it, i.e. how to make the decision.
WHAT INFORMATION TO GET Employers generally use one or more of five approaches at the first sift stage: 1 2 3 4 5
traditional application form (AF); CV/résumé; competence-based AFs; short tests of personality or ability; biodata.
The first four of these approaches and discussed in the following subsections; biodata is discussed at length later in the chapter.
Traditional Application Form Most large employers have their own printed AF, designed to collect ‘standard’ information, such as address and qualifications, as well as any particular organisationspecific requirements. Anyone who responds to the advertisement might complete and return a form and enter the first sift. Except in times of full employment, employers usually get hundreds of applications so the first stage is to select a handful for more thorough assessment. The traditional application form therefore serves several purposes: 1 to sift applicants; 2 to help interviewers prepare their interview; 3 to form the basis of personnel records. AFs often contain questions that have been there for many years, so that no one presently knows why they are included. Some probably are not used or needed. Some may no longer be considered correct questions to ask.
Recommendation: Review your application form and ask why each question is there, and whether it needs to be included.
0470861630_06_cha05.fm Page 101 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
101
Equal Opportunities and the Application Form The advance of equal opportunities thinking has tended to restrict the information that employers can request at AF stage. In the USA, the Equal Employment Opportunities Commission’s (EEOC) Guide to Pre-Employment Inquiries lists a wide range of AF questions that are suspect, because they may not be job related, and may lead to illegal discrimination: marital status, children, child care; hair or eye colour; gender; military service or discharge; age; availability over holidays or weekends (which may discourage some religious minorities); height and weight, arrest records etc.
Recommendation: Review your application form and ask if any question is likely to screen out protected groups: women, ethnic minorities, disabled people etc.
CV/Résumé For some jobs, people apply by CV/résumé, rather than by AF. (CV stands for curriculum vitae, which is Latin for short summary of life.) The main difference is applicants create their own CV/résumé and can choose what information to give and how to present it. Otherwise, both CVs and AFs have the same function, to enter a first sift. The CV/résumé offers more scope for applicants to try to stand out in first sift, which may be important if the selection ratio is low.
Competence-Based Application Form Recently many employers have sought to improve their AFs by using their competence framework. Applicants are asked to describe things they have done which relate to key competences for the job (Figure 5.1). For example the competence ‘ability to
Innovation. Part of the role of the funding manager is to find and develop new ways of obtaining funding, and operating the funding enterprise efficiently. Think of some occasions when you were able to find a new way of dealing with a problem that had arisen. These can be from any area of your life, not just paid employment. Explain what the problem was, what your solution was and why it was successful Give several examples if possible. Describe each in about 150 words. Dealing with pressure. The funding manager’s job is quite pressured at times. Think of some occasions in the past when you were placed under some pressure. These can be from any area of your life, not just paid employment. Explain what the pressure was, how it affected you and what you did to cope effectively. Give several examples if possible. Describe each in about 150 words
Figure 5.1 Competence-based application form
0470861630_06_cha05.fm Page 102 Thursday, December 2, 2004 8:32 PM
102
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
influence others’ is assessed by ‘describe an occasion where you had to persuade others to accept an unpopular course of action’. The applicant is expected to write 200 words or so, which makes completing the application form much more work for applicants. This method might well improve the AF as a selection assessment, but we do not yet have any research on whether it does. We need to know whether: • assessors agree how much resilience or drive an account indicates (reliability); • screening by behavioural competency application form is more accurate, i.e. selects better applicants (validity); • the method creates any gender, ethnicity or disability differences (adverse impact).
Tests Some employers find short tests of job knowledge useful at the application stage. The US Office of Personnel Management finds 50–60% of applicants for IT work actually know little or nothing about subjects, e.g. C++ programming, they claim expertise in. Some employers are replacing their conventional paper application forms by short questionnaires completed over the Internet. This represents quite a radical change in the sifting process. Formerly HR staff inferred e.g. leadership potential from what applicants said they did in previous employment or at school or university. The new systems assess it more directly by a set of standard questions. This saves time spent reading AFs, and could ensure more standardised assessment of core competences. In effect the conventional AF or CV has been replaced by a short personality questionnaire, moved forward from its usual position at the shortlist stage. This approach might be very effective, but no research has been published on how such systems work. We need to ask the same basic vital questions (yet again!): • Do online screening questionnaires give consistent accounts of applicants (reliability)? • Are they accurate, i.e. do they select better applicants (validity)? • Do they create any gender, ethnicity or disability differences (adverse impact)?
Recommendation: If you introduce a new application system, such as competency-based questions, or tests, have plans to assess its reliability, accuracy and adverse impact.
Honesty A continuing concern is whether applicants tell the truth on AFs and CVs, although there is not much good-quality research on the issue. A recent UK survey (Keenan,
0470861630_06_cha05.fm Page 103 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
103
1997) asked British graduates which answers on their AFs they had ‘made up . . . to please the recruiter’. Hardly any graduates admit to giving false information about their degree, but most (73%) admit they are not honest about their reasons for choosing the company they are applying to. More worrying, 40% feel no obligation to be honest about their hobbies and interests. Most students get advice from university careers offices on how best to present themselves and their achievements at this stage. One of the authors recalls a candidate who claimed to have sailed 60 miles in an afternoon across major shipping lanes, in waters noted for dangerously strong tides and currents, in a very small dinghy. He did not know that one of the panel was an experienced sailor familiar with the area who said the journey he described was quite impossible!
HOW TO GET INFORMATION Traditional selection started with paper applications. Today many employers use online application systems, which work differently. Advertising, making applications, sifting applications and even assessment can all be carried out over the Internet, which can make the whole process far quicker. People talk of making ‘same-day offers’, whereas traditional approaches took weeks or even months to fill vacancies. • More and more jobs are advertised on the Internet, through the employer’s own website or through numerous recruitment sites. • People seeking jobs can post their details on websites for potential employers to evaluate. This gives the job seeker an opportunity that did not exist before (people could make speculative applications to possible employers, but could not advertise themselves on a global scale.) • Many employers now use electronic application systems, eliminating the conventional paper AF. This makes it far easier to store, sort or search applications, and to generate shortlists. Paper applications can be scanned into the database, or else applicants can be required to apply online in the first place. Apparently President Clinton used such a system to select his aides from thousands of applicants. • Internet recruitment can greatly increase the number of candidates, which is good for the employer if it broadens the field of high-calibre applicants, but it does also make the employer’s sifting problem greater. • Aptitude tests or assessments of personality can be completed over the Internet by the applicant. This saves time and travel costs. Internet recruitment and selection has a number of potential problems, however: • Not everyone has access to the Internet. Surveys suggest gender, ethnicity and age differences, which will have possible legal implications; employers cannot afford to find they are sifting out, for example, ethnic minorities because they are less likely to have access to the Internet. Access to the Internet may also be linked to differences in income and education, so Internet recruitment may tend to
0470861630_06_cha05.fm Page 104 Thursday, December 2, 2004 8:32 PM
104
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
exclude the less fortunate. Social exclusion is a major concern of some European governments, and the EC. • The electronic medium does not bypass problems that arise with paper. It is just as easy to lie through a keyboard as it is on paper or in person, and just as easy to give the answer you think the employer wants to hear. • Is someone who logs on to an employer’s recruitment site or sends in an unsolicited CV a job applicant? If they are an applicant, the employer may be obliged to record their gender, ethnicity, disability etc. to be able to prove protected groups are not being sifted out at this preliminary stage. This creates two problems for employers: the sheer volume of enquiries or unsolicited applications some receive, and the absence of the information needed for equal opportunities monitoring.
Resumix There are a number of commercial Internet recruitment systems in general use. These systems can be very useful for filing and sorting applications, and for generating letters to applicants. One of the best known is Resumix, which started operations as long ago as 1988 and boasts many major employers as customers, including the American armed services. Resumix is currently called Hiring Gateway, and costs ‘from’ $200 000. In fact Resumix does more than just scan and file applications; it is also a job analysis system. Resumix has a list of 25 000 KSAs (Knowledge, Skill, Ability). Employers use this list to specify the essential and desirable skills for their particular vacancy, and Resumix searches applications for the best match. Resumix’s list of skills is protected by copyright, i.e. is not released. An article by Jean MacFarland (2000) lists some of the competences Resumix uses, and which it identified in her own CV; they included leadership, budget planning and forecasting, performance assessment, staff education, performance management, performance evaluation etc. Resumix seems to save employers time and money, but may not make life all that easy for job applicants, judging from the number of consultancies in the US offering ‘how to make Resumix applications’ services.
HOW TO USE THE INFORMATION The HR manager who has 1000 applications for a single vacancy needs some method of selecting between 5 and 50 to assess in greater depth, a process called sifting or screening. Sifting can take up a lot of time in HR departments so any way of speeding up the process will be very valuable, so long as it is accurate and fair. The higher the unemployment rate, the bigger the employer’s sifting problem. When every vacancy attracts sacks full of applications, or the electronic equivalent, the question is how to select 20 or so to assess in greater depth. Sifting is not always done very effectively. Northcote Parkinson mentions a standard sifting rule used in the 1950s: first exclude anyone over 40 or Irish (Parkinson, 1958). This sounds very crude, and would of course be illegal today. Another method sometimes used is to recruit via an agency or job centre, and to ration application
0470861630_06_cha05.fm Page 105 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
105
forms, so when 500 forms have all been distributed, no one else can apply. This is also very crude. Application sifting is a danger area for the employer because all sorts of biases may creep in. Some are merely ‘noise’ that probably reduce the accuracy of selection: preferring rugby players or rejecting trainspotters. Other biases are illegal, and potentially very damaging to the employer: gender, ethnicity and disability. Any sifting decisions—explicit or implicit, conscious or unconscious—that create adverse impact on these protected groups can lead to fair employment claims. In the UK, the Commission for Racial Equality recommends that application sifting should be done by two persons (which will discourage obvious bias, but may not detect subtle or shared ones).
How Managers Sift Some research has used policy capturing analyses to reconstruct how personnel managers sift applications. This approach works back from the decisions the manager makes about a set of applications, to identify how the manager makes them. Research in Germany finds that what the managers do, according to the policy capturing analysis, often differs from what they say, when asked to describe how they sift (Machwirth et al., 1996). Managers say they sift on the basis of proven ability and previously achieved position, but in practice reject applicants because the application looks untidy or badly written. A recent review of 49 studies of gender bias in simulated employment decisions finds bias against female applicants, by female as well as male sifters (Davison and Burke, 2000). The less job information is given, the greater the discrimination against females. Another recent piece of research looked at how campus recruiters use grade point average (GPA) (course marks) to select for interview. Some chose students with high marks, which is the logical use of the information, given that GPA does predict job performance to some extent, and that it is linked to mental ability, which also predicts job performance. A second large (42%) group ignored GPA altogether. A third group selected for lower GPA, screening out any applicant with high grades. This does not seem a good way to sift, given the link between work performance and mental ability. The choice of strategy seems essentially idiosyncratic, and could not be linked to type of job or employer.
Recommendation: Review your application sifting process carefully. If possible, analyse how it actually operates, as well as how it is intended to operate, or said to operate.
Training and Experience (T&E) Ratings In the USA application sifting is assisted by training and experience (T&E) ratings, which seek to quantify applicants’ training and experience, instead of relying on
0470861630_06_cha05.fm Page 106 Thursday, December 2, 2004 8:32 PM
106
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
possibly arbitrary judgements by the sifter. T&E ratings have been widely used in the public sector in the USA, especially for jobs requiring specialised backgrounds such as engineering, scientific or research positions. T&E ratings are also useful for trade jobs, where written tests may unfairly weight verbal skills. Several systems are used; applications are assigned points for years of training and education, or else applicants are asked for self-ratings of the amount and quality of their experience. Research indicates T&E ratings contribute increased accuracy to the selection process (a modest validity overall of 0.13). T&E ratings are labour intensive as they require an expert to read carefully every single application. Some methods would be fairly easy to computerise. The behavioural consistency method of T&E ratings gives the best results. Areas that reveal the biggest differences between good and poor workers are identified by the critical incident technique, which collects examples of good and poor performance. Applicants describe their major achievements in these areas, and their accounts are rated by the selectors using behaviourally anchored rating scales (described in Chapter 14). Behavioural consistency T&E ratings are twice as accurate as other T&E systems, achieving a validity of 0.25. They might prove less easy to computerise.
Sifting by Software Software can scan applications and CVs to check whether they match the job’s requirements. This is very much quicker than conventional sifting of paper applications by HR staff; the Restrac system is said to be able to search 300 000 CVs in 10 seconds. Automated sifting systems can eliminate bias directly based on ethnicity, disability or gender, because they are programmed to ignore these factors. They will not necessarily ignore factors linked to ethnicity, disability or gender, such as sports and pastimes. Sifting software will do the job thoroughly, whereas the human sifter may get tired or bored and not read every application carefully. Software should achieve 100% consistency in sifting, whereas fallible human sifters are likely to be inconsistent (i.e. unreliable) in their decisions. Sifting electronically has many advantages, but is not necessarily any more accurate. Accuracy depends on the decision rules used in sifting, which in turn depends on the quality of the research the employer has done. Reports suggest that some scanning systems do nothing more sophisticated than search for key words; once applicants realise this, they will take care to include as many as possible. Resumix says that its software does not use simple word counting, nor is there a list of ‘buzzwords’ that applicants can include to improve their chances of being selected. The system is described as ‘intelligent’ and as able to recognise the contextual meaning of words. The software is copyright and no details are released. Computerised sifting seems likely to work well for fairly specific requirements, such as CIPD membership, or upper second class degree in psychology. It is less clear how well it could sift applications for broader attributes such as drive, resilience, or flexibility. There is an urgent need to know what application sifting programs actually do. The people who sell them are not always very forthcoming, which tends to make
0470861630_06_cha05.fm Page 107 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
107
one wonder if they are doing anything very complicated. Personnel managers should be very wary. Application sifting is a ‘selection test’, both in practice and in law. It selects some people and rejects others. Accordingly, the usual questions should be asked: • Does it select the right people, i.e. people who will do the job well? Accuracy or validity of electronic application sifting appears completely unresearched. • Does it reject protected groups—women, minorities, the disabled—more often? In the case of Hightower et al. v. Disney (1997), non-white applicants claimed that Resumix looked for keywords which were more likely to be used by white applicants. Psychologists tend to be rather sceptical of application processing software, for one fairly simple reason. If these systems are doing something tremendously subtle and complex, where did the people who wrote them acquire this wisdom? • There is no evidence that human application sifters are doing anything highly complex that software can model. • There is no body of research on application sifting that has described any complex subtle relationships to put into your software. Research on human decision-making in HR and related areas (e.g. medical diagnosis) consistently finds that people make their decisions in a fairly straightforward way, which can be characterised as additive. Additive means that the human expert reads the application form, and notes good points and bad points. The expert then reaches a decision by adding the good points and subtracting the bad points, the resulting total being their decision. What human experts do not seem to do is search for, or use, more complex patterns, such as saying exam marks are important in school leavers or graduates under the age of 22 but not in people over 25. (This conclusion derives from policy capturing research: many sifters will tell you they use complex decision rules, but analysis of the decisions they actually make does not confirm this.) We should perhaps remember one of the things we are told about computers but tend to forget: they are very good at doing mindlessly simple tasks extremely quickly, but cannot think for themselves, and cannot do anything unless given very precise and detailed instructions. If we know how to sift applications, the computer might be able to do it very quickly, but if we do not, the computer will not find a magic way of doing it.
Recommendation: If you are thinking of buying an application sifting package, ask what it actually does, and what evidence the supplier has on validity and adverse impact. Do not expect computers to do your thinking for you.
0470861630_06_cha05.fm Page 108 Thursday, December 2, 2004 8:32 PM
108
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
BIODATA Biodata represents the most ambitious transformation of the conventional application form, using statistical analysis of applications to identify details that predict success. Biodata is not new. Over 80 years ago, Dorothy Goldsmith devised an ingenious new solution to an old problem: identifying the small minority of people who could succeed at selling life insurance. She took 50 good, 50 poor and 50 middling salesmen from a larger sample of 502, and analysed their application forms. She identified factors that collectively distinguished good from average and average from bad: age, marital status, education, (current) occupation, previous experience (of selling insurance), belonging to clubs, whether the candidate was applying for full- or part-time selling and whether the candidate himself had life insurance. Some items—married/single—were scored +1/−1. Other attributes like age or education were more complicated: the best age range for selling insurance was 30–40, with both younger and older age groups scoring less well. The system proved effective. Low scorers in Goldsmith’s sample almost all failed as insurance salesmen; the small minority of high scorers formed half of a slightly larger minority who succeeded at selling life insurance (Figure 5.2). Goldsmith had turned the conventional application form into a weighted application blank (WAB). The principle is familiar to anyone with motor insurance. The insurance company analyses its records to find what sort of person has more accidents: people who drive sports cars, people who live in big cities, people who run bars etc. Insurers do not rely on common sense, which might well suggest that younger drivers, with faster reflexes, will be safer; they rely on their records, which
30
150 ×
20
100
×
50
Percentage successful
Number of subjects
200
10
× Low Med. High Low Med. High Low Med. High Low
Medium
High
Sales WAB score
Figure 5.2 Results from the first published weighted application blank (WAB) Source:
Data from Goldsmith (1922).
0470861630_06_cha05.fm Page 109 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
109
show that young drivers are a poorer risk. If insurers can calculate premiums from occupation, age and address, perhaps personnel managers can use application forms in the same way, as a quick but very powerful way of selecting employees. Early biographical systems collected information through the conventional application form, whereas nearly all current systems use a separate questionnaire. A separate questionnaire makes it clear to the candidate that he/she is being assessed, so loses the invisibility of AF-based formats. Traditional biodata screening was done on paper and scored by hand, but could equally be administered over the Internet, and processed by computer. Biodata has the advantage of established technology, and a body of research on how to use the information Biodata has a number of features: • It starts by analysing past applicants to identify facts or items that are linked to the outcome you are trying to predict. • It needs very large numbers to work properly. • It must be cross-validated before being used, i.e. items must be shown to work in both of two separate samples. As a consequence biodata is expensive to set up, but cheap to use once in place. • It is backward looking; it will find ‘more of the same’, e.g. more managers like present successful managers. This may be a problem in rapidly changing industries, where ‘cloning’ can be dangerous. Even so ‘finding more of the same’ is better than ‘selecting’ at random, which is all some selection methods can achieve. Figure 5.3 gives some typical biodata questions.
Constructing Biographical Assessments To devise a biodata, the employer will need: 1 Two or more groups of people who differ in some key respect, e.g. good at selling/ poor at selling, regarded as thieves/regarded as honest. Biographical measures need large numbers at this stage. Most experts recommend a minimum of 200 in each group, i.e. 200 good salespersons and 200 poor. 2 A large number of possible biographical questions. Biodata questions can be drawn from ready-made item banks, such as Glennon’s Catalog of Life History Items, which lists 484 biographical items. However the trend is increasingly toward generating questions that are conceptually related to what you are trying to predict. 3 Both groups (e.g. good and poor sales staff) answer the entire set of biographical questions, and the answers are analysed to look for differences that can be made into a biodata. There is a risk of failing to find any differences and so failing to generate a test at all. An attempt to write a biodata for selecting ministers of
0470861630_06_cha05.fm Page 110 Thursday, December 2, 2004 8:32 PM
110
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
1. How old was your father when you were born? 1. about 20 2. about 25 3. about 30
4. about 35
5. I don’t know
2. How many hours in a typical week do you engage in physical exercise? 1. none 2. up to 1 hour 3. 2–3 hours 4. 4–5 hours 5. over 5 hours 3. In your last year at school, how many hours in a typical week did you study outside class hours? 1. none
2. up to 2 hours
3. 2–4 hours
4. 5–8 hours
5. over 8 hours
4. How old were you when you first kissed someone romantically? 1. 12 or under 2. 13 or 14 3. 15 or 16 4. over 16
5. never kissed anyone romantically
5. How interested in current affairs are you? 1. not at all
2. slightly
3. fairly
4. very
5. extremely
6. Which best describes your present height/weight ratio? 1. definitely overweight
2. somewhat overweight
3. slightly overweight
7. How often did you play truant from school? 1. never 2. once or twice 3. 3 to 10 times
4. just right
4. once a month
8. What do you think of children who play truant from school? 1. very strongly 2. strongly 3. disapprove 4. unconcerned disapprove disapprove
5. underweight
5. once a week or more
5. can sympathise
9. My superiors at work would describe me as 1. very lazy
2. fairly lazy
3. average
4. quite hard working
10. How many times did you have to take your driving test? 1. once 2. twice 3. three 4. four or more
5. very hard working
5. never taken it
Figure 5.3 Some typical biodata questions. It is instructive to ask yourself which of these could be verified by the employer. How? At what cost?
religion failed at this first stage: no biographical differences between ‘successful’ and less successful minsters were found. 4 The essential last step in writing a biodata is cross-validation, which means doing stage 3 a second time. The original samples are used to construct the biographical measure. The biodata’s accuracy can only be calculated from a second set of samples (known as cross-validation or hold-out samples). This means the employer will need a second set of groups of 100 each. A biodata that has not been cross-validated should not be used for selection. Some biographical details will appear to distinguish the groups but have in fact got in by chance; crossvalidation checks that every question consistently distinguishes good from poor
0470861630_06_cha05.fm Page 111 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
111
salespeople, thieves from honest employees etc. The second set of samples is used to calculate the biodata’s accuracy. To summarise, the steps involved in writing a biodata are: • identify two or more groups of people who differ in whatever you are trying to predict; • generate a pool of biodata questions that relate to whatever you are trying to predict; • analyse the data to devise a first-draft biodata; • cross-validate on two or more further groups of people who differ in whatever you are trying to predict; • analyse the data to devise a final draft of the biodata. This is a substantial undertaking, and therefore costly. Once in place, however, biodata can improve the sifting stage of selection considerably for some years to come (but not forever).
SOME TECHNICALITIES OF BIODATA CONSTRUCTION Types of Biodata Question Biodata questions can be divided into hard, which are verifiable but also often intrusive, like question 6 in Figure 5.3, and soft, which cause less offence but are easier to fake, like question 5. Some questions ask about controllable aspects, while others do not; we choose our exercise pattern (question 2) but cannot choose our parents’ ages (question 1). Some biodata systems adopt a policy of avoiding non-controllable questions, because they make biodata harder to defend against criticism. Questions asking about present behaviour prove as successful as those asking about past behaviour, and may be more acceptable, because they avoid the suggestion of rejecting people because of past behaviour they cannot now change. Many biodata contain questions about attitudes, such as question 8. Some biodata questions ask what other people think of you, e.g. question 9, which is verifiable, at least in theory. Including this type of question makes the biodata resemble a personality test.
Empirical Keying Traditional biographical methods were purely empirical. If employees who stole were underweight, or had no middle initial, or lived in Hackney, those facts entered the scoring key. Purely empirical measures worry some psychologists, who like to feel they have a theory. They are not happy knowing people with no
0470861630_06_cha05.fm Page 112 Thursday, December 2, 2004 8:32 PM
112
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
middle initial are more likely to steal; they want to know why. Ideally, they would like to have predicted from their theory of work behaviour that people with no middle initial will make dishonest employees. Critics of pure empiricism also argue that a measure with a foundation of theory is more likely to hold up over time and across different employers, and may be easier to defend if challenged. Trying to explain in court the connection between no middle initial and theft could be difficult!
Factorial Keying The first attempt to give biodata a more theoretical basis relied on factor analysis, a statistical technique for identifying themes in biographical information. If no middle initial proved to be linked to half a dozen other biographical facts, all to do with, for example, a sense of belonging, we begin to get some idea why it relates to honesty in the workplace, we can explain to critics why it is included, and we can perhaps search for better questions to reflect the underlying theme.
Rational Keying Modern approaches to biographical assessment generally use a more targeted approach. John Miner (1971) wanted to assess ‘eliteness motivation’, so he stated specific hypotheses about eliteness, for example that status-conscious Americans will serve (as officers, of course) in the navy or air force, rather than the army. Recent biodata sometimes use a behavioural consistency approach. If job analysis indicates that the job needs good organisational skills, questions are written that reflect this, either on the job—‘How often do you complete projects on time?’—or off the job— ‘To what extent do you prepare when going off on holiday?’ These questions are still tested, with 200 well-organised and 200 poorly organised employees, to check that they work (and still double-checked or cross-validated with two further groups). This approach has two advantages. It has a logic to it so it is easier to defend, and it is more versatile; a biodata for organisational skills can be used for any job that requires such skills.
DO BIODATA WORK? There is an extensive body of research in the USA and the UK on the effectiveness of biodata in selection. Two recent reviews offer very similar overall estimates of biodata validity, placing it at 0.30 (Bliesener, 1996; Bobko and Roth, 1999). This indicates biodata can make a very effective way of screening applicants. Recall that most other first-sift systems either offer little or no evidence of their effectiveness, or have been shown to lack much power to identify good applicants.
0470861630_06_cha05.fm Page 113 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
113
Absence Creativity Tenure Training success Objective performance* Proficiency rating 0
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Validity
Figure 5.4 Validity of biodata for six aspects of work Source: Data from Bliesener (1996). * Covers both production and sales.
Different Outcomes Biodata have been used more extensively than other selection measures to predict a variety of work-related outcomes (as well as some non-work outcomes such as credit rating). Figure 5.4 shows: • biodata predict general proficiency at work, assessed by supervisor ratings; • biodata are good at predicting objective performance indictors such as sales and production; • biodata can predict absence and tenure, but less accurately. These are, however, notoriously hard to predict, possibly because they depend more on how the organisation is run than what sort of people it selects.
Different Types of Work Figure 5.5 shows biodata achieve good accuracy in selecting for a range of occupations: managerial, sales, clerical and science/engineering. They seem to work less well—but still better than chance—for military selection. Biodata have also proved useful for more specialised work, such as research achievement in science and technology, and success in pilot training.
Incremental Validity Biodata are often used in conjunction with other methods, which raises the question of incremental validity. Does the combination of, for example, biodata and mental ability
0470861630_06_cha05.fm Page 114 Thursday, December 2, 2004 8:32 PM
114
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
Science/engineering Military Clerical Sales Managers 0
0.1
0.2
0.3
0.4
0.5
Validity
Figure 5.5 Validity of biodata for five areas of work Source:
Data from Bliesener (1996).
test improve much on the accuracy achievable by either used alone? Several studies suggest that biodata can improve on the predictions made by psychological tests, for attrition in army recruits, proficiency in clerical work and organisational citizenship (Mael and Ashworth, 1995; McManus and Kelly, 1999; Mount et al., 2000). However, it has been suggested that biodata and mental ability tests sometimes correlate quite highly, which implies that biodata may not always achieve much incremental validity.
Consortium Measures Biographical measures need large samples, which appears to rule them out for all except very large employers. However, organisations that do not employ vast numbers can join a consortium to devise a biodata. The Aptitude Index Battery (AIB) is a consortium biodata used by the North American insurance industry since the 1930s. AIB is presently called Career Profile and is part of the Exsel system. AIB data have been analysed for 12 000 insurance sales staff from 12 large insurance companies. AIB works in all 12 insurance companies, but is more successful in larger, better run companies that recruit through press adverts and agencies, than in smaller companies that rely more on personal contacts. AIB has been rewritten and re-scored many times, but has retained some continuity. Another consortium biodata is the (US) Supervisory Profile Record, which was originally written using staff from 39 separate organisations, and which works well across all 39, regardless of gender, race, supervisory experience, social class or education.
Recommendation: Consider whether you could benefit from introducing biodata to sift applicants.
0470861630_06_cha05.fm Page 115 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
115
PROBLEMS WITH BIODATA Fakability ‘Hard’ biographical questions, e.g. father’s age, can be checked, at a cost. However, many biodata questions cannot be checked, e.g. age of first kiss (item 4 in Figure 5.3) or interest in current affairs. There is no sensible way of questioning the information the applicant gives you about such issues. Biodata inventories may be prone to faking, because they are visible to the applicant, and because many biodata items are not objective or verifiable. The problem of biodata faking has a series of parallels with faking personality inventories, discussed in Chapter 4: • people directed to fake good can usually improve their biodata scores; • people directed to fake good distort their answers far more than job applicants, so directed faking research may be misleading; • the extent of faking by real job applicants is uncertain. One study found that only 3 items of 25 were answered differently by job applicants; these items were less historical, objective and verifiable (Becker and Colquitt, 1992). On the other hand, another study compared applicants—motivated to fake—with people who already have the job—who have less need to fake—and found applicants’ answers much ‘better’ in areas like preferred working climate, work style or personal and social adjustment (Stokes et al., 1993). As with personality tests, there seems to be a faking problem, but we are not sure how big it is. There are several possible ways of dealing with faking in biodata, again tending to repeat approaches tried with personality tests: • More objective and verifiable items create fewer differences between applicants and people who already have the job, so may be less fakable. • Warning people that the biodata includes a lie-detection scale (even if it does not) reduces faking. • A trick question about experience with the ‘Sonntag connector’, a non-existent piece of electrical equipment, caught out one-third of applicants, who claimed to have used one. This third got better overall scores on the biodata, showing they were faking, but their biodata score correlated less well with a written job knowledge test, suggesting they were not better applicants. However, trick questions are unethical, and might not work for long in practice. • Biodata measures can be provided with faking good scales, modelled on faking good scales in personality measures, e.g. ‘I have never violated the law while driving a car’. These are intended to identify applicants who ‘deny common faults’ and ‘claim uncommon virtues’. Applicants with high ‘faking good’ scores are discarded.
Fairness and Equal Opportunities If biographical questions are linked to race, sex, disability or age, these protected minorities may get lower scores on biodata measures, so giving rise to problems of
0470861630_06_cha05.fm Page 116 Thursday, December 2, 2004 8:32 PM
116
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
adverse impact. Most research has focused on ethnicity in the USA. An early review concluded that biodata did not by and large create adverse impact on ethnic minorities applying for work as bus drivers, clerical staff, army recruits or supervisors (Reilly and Chao, 1982). Another analysis found only fairly small ethnicity differences in biodata scores. The most recent analysis, however, is less optimistic, and concludes that differences between Whites and African Americans when completing biodata measures are large enough to create a problem. Although biodata may have the potential to cause equal opportunities problems, to date they have not figured in many court cases in the USA. Table 12.2 summarises US adverse impact evidence for various selection methods. It shows that biodata create far less adverse impact than tests of mental ability. Individual biodata questions might discriminate against protected groups in subtle ways. For example, questions about participation in sport could discriminate against disabled people. American experts say it is unclear whether biodata could be legally challenged question by question, so a few questions creating adverse impact may not matter. In fact, it is extremely unlikely that no gender, age, ethnicity or disability differences would be found in any of 50–100 biographical questions. However, if many questions show, for example, ethnicity differences, it tends to follow that total scores will also differ. It is important, therefore, to review all biodata measures thoroughly, question by question, and ask: might any of these questions generate differences between groups? Then when you have used the biodata, ask: do any questions generate differences between men and women, majority and minority? Recommendation: Before using a biodata, review the items very carefully; ask equal opportunities experts if any items are likely to find differences.
Privacy As Figure 5.3 shows, some biodata questions can be intrusive. In the Soroka v. DaytonHudson case, a department store was sued by an applicant who claimed he had been asked intrusive questions about politics and religion, contrary to the Constitution of the State of California. The case was settled out of court, so no definitive ruling emerged. Use of biodata in the USA is complicated by the fact that the 52 states all have their own, differing laws about privacy, so questions about, for example, credit rating are legal in some states but not in others.
Public Relations Some psychologists have proposed the Washington Post test for biodata plausibility. Imagine two headlines in the Washington Post. One headline reads ‘Psychologists reject people for management training because they don’t like the colour blue’, which sounds arbitrary and unfair. Another headline reads ‘Psychologists reject people for management training because they held no positions of responsibility at school’, which sounds much more reasonable.
0470861630_06_cha05.fm Page 117 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
117
Recommendation: Before using a biodata, review the items very carefully and ask yourself if you would feel uneasy about answering any of the questions. After using a biodata, check the results for adverse impact.
The Need for Secrecy? Early studies published their biodata in full, confident that cannery workers or shop assistants did not read the Journal of Applied Psychology and could not discover the right answers to give. If the scoring system becomes known, a biodata measure can lose its effectiveness. A new form of the insurance industry’s AIB worked well while it was still experimental, but lost its validity as soon as it was used for actual hiring (Figure 5.6). Field managers scored the forms and were supposed to use them to reject unsuitable applicants; instead they guided applicants they favoured into giving the right answers. When scoring was moved back to head office, the AIB regained its validity. It is doubtful whether any selection method can be kept completely secret these days because all are likely to come under intense legal scrutiny. Biodata have proven effectiveness in screening applicants for a variety of HR purposes. They are easy to include at the screening stage in paper form, and would
60
Percentage successful
50
× Grade A
× ×
40
×
30
×
× ×
× ×
× Grade C
× 20
× ×
×
1
2
× Grade E
10 3 Year
4
5
Figure 5.6 Success rates for three groups of insurance sales staff, identified by biodata as likely to be good (grade A), average (grade C) or ineffective (grade E), for 5 successive years. In year 3 scoring was devolved to branch office. In year 5 it was moved back to head office Source:
Data from Hughes et al. (1956).
0470861630_06_cha05.fm Page 118 Thursday, December 2, 2004 8:32 PM
118
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
be equally easy to include in online recruitment systems. They are specifically designed to be easy to administer and easy to score, requiring no human judgement. They do, however, require detailed analysis of data from large, carefully selected groups of people. The same, however, is likely to be true of any other applicant screening system, whether computerised or paper in form.
CASE STUDY Scale 101 We have mentioned several times the problem of screening people who work with children, to keep out those who have the wrong motives. There have been a number of cases where people have got work in child care facilities or schools, and used their position to abuse children. We need a way of detecting such people and excluding them from working with or near children. Several approaches have been suggested, and some implemented: • People applying to work with children cannot withhold information about criminal convictions, even when they are ‘spent’ under the Rehabilitation of Offenders Act. • The Criminal Records Bureau has been set up to allow employers to check criminal records (not always successfully, as the ‘Background Checks’ case study in Chapter 11 shows). • The Waterhouse and Walner Reports have suggested personality testing. • Some paedophiles are convinced their behaviour is acceptable, and will try to argue that sexual contact does not harm children, that children want it, that it is good for their development etc. It would be unwise to rely on paedophiles stating such ‘self-justifying beliefs’ when applying for jobs. • Only employ women in child care work, on the basis that 99% of people who abuse children sexually are male. This would, of course, be illegal, and also unfair on men who want to work with children for the right reasons. The problem with screening is that no one is going to admit to paedophile tendencies, if they are trying to get a job working with children. Most methods of assessing applicants rely on what the applicant tells you, on the application form, at the interview and in self-report tests. We can expect paedophiles to fake good on personality tests, lie in interviews and conceal past behaviour, including convictions. One possible way forward might be biographical assessment. If we could find biographical pointers that distinguish paedophiles from other people, we could use a biographical screen to identify at least some dubious applicants. The traditional approach would be to assemble a long list of questions, taken from perhaps the Glennon Catalog of biographical questions, find a group of 200 paedophiles and a group of 200 people matched for age, social background etc., and hope to find enough biographical differences to constitute a workable screening test. We would then need a second pair of groups, 100 of each, to weed out biographical
0470861630_06_cha05.fm Page 119 Thursday, December 2, 2004 8:32 PM
SIFTING AND SCREENING
119
details that have got in by chance, and do not really distinguish paedophiles from non-paedophiles. Scale 101 takes the more focused approach favoured in recent biodata construction. There is an extensive research literature on the nature and origin of paedophile tendencies, and on differences between paedophiles and non-paedophiles, in areas such as their early life, and their (non-sexual) interests and preferences. We combed through this literature to try to pick out anything that could be used as a biographical item. Our first draft contains 101 items, hence the title Scale 101. We now need to go through the traditional biodata generation stages described above. We need to: 1 2 3 4 5
get Scale 101 completed by 200 paedophiles; get Scale 101 completed by 200 control persons; produce a first-draft biodata; repeat steps 1 to 3 with two more groups; produce a working draft biodata.
Stage 1 is quite difficult. There are over 110 000 convicted paedophiles in England and Wales. But finding some when you want them is not easy. We have addressed the problem of faking in two ways. The first is to try to devise subtle items that will not suggest a ‘right’ answer. The second is to incorporate a faking good scale, to pick out people who are faking good. Further stages of the research programme will assess the success of these approaches: 6 get Scale 101 completed twice by the same group, first honestly, then under directed faking good instructions; 7 get Scale 101 completed twice by the same group, first as part of the selection process for child care work; second, later on, when they are in post. People who have the job have less need to fake than people trying to get the job. The final stage will be a follow-up study, where people applying for child care work complete. Scale 101 as part of the selection procedure, and we subsequently compare those who have sexually abused children with those who have not. This will take a very long time, because there will not be many abusers. It is, however, the ultimate test of whether Scale 101 works. We have included two sample items from Scale 101. We cannot include any more, partly on grounds of space, but principally so as not to compromise the measure by letting the people it is intended to detect find out what answers to give. 1. Which period of your life do you consider to have been the happiest? when I was aged less than 10 when I was aged 10 to 14
0470861630_06_cha05.fm Page 120 Thursday, December 2, 2004 8:32 PM
120
PSYCHOLOGICAL ASSESSMENT IN THE WORKPLACE
when I was aged 15 to 19 when I was aged 20 to 29 when I was aged over 30 none of these cannot say (This item is based on the assumption that many paedophiles were happiest as children, and have not achieved a satisfactory adjustment to adult life.) 2. Which statement best describes your relations with your parents? I got on very well with my parents I got on fairly well with my parents I got on well with my parents some of the time my relations with my parents were fairly poor none of these cannot say (This item is based on the assumption that many paedophiles had poor relations with their parents. This question alone would not of course distinguish paedophiles from other people. Biodata items are intended to build a cumulative picture.)
0470861630_07_cha06.fm Page 121 Saturday, December 4, 2004 3:22 PM
CHAPTER 6
References and Ratings
OVERVIEW • The opinions of others about the applicant can be useful. • Others’ opinions are collected through references and peer ratings. • References are used to check facts, to check for problem behaviour and to assess quality of work. • Reference requests take various forms, including unstructured, rating and competence-based formats. • References may not be very reliable. • References do not appear to predict work performance very well. • References are inaccurate through leniency and idiosyncrasy. • The legal position of the reference has changed, making many employers reluctant to provide them. • We review possible ways of making the reference more useful. • Peer ratings can be collected in various forms. • Peer ratings are fairly reliable. • Peer ratings can predict work performance quite well. • There are major limitations on the possible use of peer ratings.
INTRODUCTION Some people know your applicants very well, far better than you, the prospective employer. Former employers, school teachers, colleagues or fellow trainees have seen the applicant all day, every day, perhaps for years. They can report how he/she usually behaves, and perhaps what he/she is like on ‘off days’. They know about applicants’ typical behaviour as well as their best behaviour, so are potentially very useful assessment