PSYCHOLOGICAL ASSESSMENT SEMINAR
Rubik’s cubes of colour,
shape and size:
Ethical psychological testing
in a multicultural work
environment
Department of Industrial & Organisational
Psychology
18 August 2010
AGENDA
Morning session: 08h30 - 13h00
• Welcome by Prof Dirk Geldenhuys
• Introduction: Welcome to the Rubik’s Cube
Adventure
• A short history of psychological assessment
• The areas of psychological assessment
• Psychological assessment myth busters
Tea break: 11h00
• Legislation and ethics
• Ethical freeze role play
Lunch: 13h00 - 13h45
• Current topics in psychological assessment
• Psychological assessment Crosswords
• The future of psychological assessment
Conclusion: 15h00
A short history of Psychological Assessment
Why look at the past?
 History influences current practice
 Relevance
 Evolved and progressed
 History is important
 Helps explain current practice
 Strengths and weaknesses
 Prevent repetitions of the ‘wrongs of the past’
History of testing: An International Perspective
 Over the years many authors, philosophers and scientist have explored
various avenues in an attempt to assess human attributes. To mention a
few;
 Astrology (positions of planets to describe the personality of individuals
and predict what may happen in their lives)
 Physiognomy (judging the character of a person from external features of
the body, especially the face.)
 Graphology (the systematic study of handwriting and the notion that it
can be used as an expression of personality characteristics)
 Humorology
 Phrenology
(Foxcroft, 2009)
Testing in South Africa: The Early Years
• Psychology emerged as an academic discipline in South Africa after
World War 1.
• Many of the influential figures in the development of intellectual tests
in South Africa where trained at American institutes.
• Therefore tests which where typically used in the early phases
where also American such as; Goddards 1911 revision of the
Binet-Simons test, Termans 1916 Stanford revision, the Army Beta,
the Porteus Maze Test etc.
• In the period between the two world wars, social and human
sciences in South Africa became important contributors to debates
on crucial social issues including ‘the Native question”
(Louw, 1997; Foxcroft, 2009; Classen, 1997)
Testing in South Africa: The Early Years
 Psychological knowledge was used as a tool legitimise a social
order based on race as the perception was that psychological
testing produced empirical data which supported certain
explanations of this order.
 There was an increase in the use of science as a factor in regulating
aspects of social economic life.
 Differential performance of black and white reinforced the idea of
hierarchy of human societies and consequently differential treatment
in terms of education, employment etc.
(Louw, 1997)
Key notes
Psychometric Testing
Psychometric testing is applied worldwide and used in various
industries for recruitment, selection and counselling purposes
(Gregory, 2000).
For the purpose of this seminar four important areas for
assessment have been identified.




Cognitive assessment
Personality assessment
Behavioral assessment
Interest
Range
oftests
tests
distributors
Range of
andand
distributors
availableavailable
- Cognitive Cognitive
MEASUREMENT
CONSTRUCT
Cognitive
CONCEPT
TEST
DISTRIBUTORS
APPLICATION IN
INDUSTRIAL/
PSYCHOLOGICAL
FIELDS
IQ,
General
cognitive,
Multiple aptitude,
Learning
potential
WAIS, SSAIS, ASAT,
RPM,
DAT Forms R, S, K & L
Mindmuzik Media
Personnel screening and
selection,
Career guidance and
counseling,
CPP, CTA
Magellan Consulting
Personnel development
APIL-B,
TRAM 1 & 2,
AproLab
M&M Initiatives
LPCAT
M&M Initiatives
TST
Thomas
International
COPAS
Integrity
International
Range
oftests
tests
distributors
Range of
andand
distributors
availableavailable
- Cognitive Cognitive
MEASUREMENT
CONSTRUCT
Cognitive
CONCEPT
TEST
DISTRIBUTORS
APPLICATION IN
INDUSTRIAL/
PSYCHOLOGICAL
FIELDS
IQ,
General
cognitive,
Multiple aptitude,
Learning
potential
WAIS, SSAIS, ASAT,
RPM,
DAT Forms R, S, K & L
Mindmuzik Media
Personnel screening and
selection,
Career guidance and
counseling,
CPP, CTA
Magellan Consulting
Personnel development
APIL-B,
TRAM 1 & 2,
AproLab
M&M Initiatives
LPCAT
M&M Initiatives
TST
Thomas
International
COPAS
Integrity
International
Cognitive Assessment
• Some consensus
– IOPs agree that cognitive ability test are valid and fair
– Cognitive ability tests provide good but inadequate
measures of intelligence
– Other attributes are necessary to account for multi
dimensional nature of performance (90% of
respondents)
(Murphy, Cronin & Tam)
Cognitive Assessment
• Some controversy
- Societal concerns
- Cultural bias
- Education
- Socio-economic status
- Education levels of parents
- Effects of negative stereotypes
- The g-Ocentric vs MI
(Murphy, Cronin & Tam)
(McKay, Doverspike, Bowen-Hilton & McKay)
Cognitive Assessment
• Unifying theoretical approach
– 3 approaches
• Structural approach (factor analytic)
• Information-processing
• Dynamic (based on learning theory)
“Learning as the critical factor underlying cognitive competence
and the mastery of problems and challenges”
(Taylor)
Structural - Spearman (g – one general factor) and Thurstone 7&9
Primary mental abilities
•static
- fluid & crystallised (Cattell)
- mostly power tests
Info processing – e.g. Ravens progressive matrices
receiving, processing & retrieval speeds
computer admin essential
Learning / dynamic approach
• Vygotsky – Zone of proximal development (ZPD)
•less susceptible to cultural bias
•3 phases of learning – conceptual understanding, compilation of
execution procedures & automatization of processing.
•Test – teach – test or pre-test – mediation – post-test
Emphasis on potential and not achievement
Range of tests available: Personality
MEASUREMENT
CONSTRUCT
Personality
CONCEPT
Personality
preferences,
Management
styles,
Communication
styles,
Interpersonal
styles,
Coping styles
TEST
DISTRIBUTORS
MMPI, TAT, SORT, VS ,
WLQ
Mindmuzik Media
OPQ 32
SHL
15 FQ, OPP, Jung
Psytech
MBTI, Big Five, WPI, LSP
MSP, CPI
Jopie van Rooyen
PAW
Integrity International
APPLICATION IN
INDUSTRIAL/
PSYCHOLOGICAL
FIELDS
Personnel selection,
Career counselling,
Academic counselling,
Personal development,
Psychological growth,
Candidate - job match,
Management
development,
Stress management
Range of tests available: Personality
MEASUREMENT
CONSTRUCT
Personality
CONCEPT
TEST
DISTRIBUTORS
Locus of
control,
motivation
and autonomy
LCI
AMP
Jopie van Rooyen
Emotional
Intelligence
MP
SHL, Magellan Consulting
EQI
Jopie van Rooyen
15 FQ EI
Psytech
TEI
Thomas International
Giotto
Mindmuzik Media
PSI
Jopie van Rooyen
IP-200
IMI
Integrity International
Integrity
APPLICATION IN
INDUSTRIAL/
PSYCHOLOGICAL
FIELDS
Personnel selection
Career counselling
Academic counselling,
Personal development
Psychological growth,
Candidate - job match,
Management
development,
Stress management
Range of tests available: Behavioral
MEASUREMENT
CONSTRUCT
Behavioural
CONCEPT
Underlying
emotional
issues,
Stress,
Anxiety,
Depression,
Conflict
TEST
DISTRIBUTORS
BESS
Jopie van Rooyen
BAI
BDI-II
BDI Fast screen
Mindmuzik Media
Jopie van Rooyen
SWSI
SRP
MBI
Jopie van Rooyen
SP
Mindmuzik Media
TKI
Jopie van Rooyen
APPLICATION IN
INDUSTRIAL/
PSYCHOLOGICAL
FIELDS
Screening and selection
Training and development
Promotion
Management potential
Group-role analysis
Interpersonal relationships
Career psychology
Personnel psychology
Organizational psychology
Conflict management
Personal development
Range of tests available: Interest
MEASUREMENT
CONSTRUCT
Interest
CONCEPT
Career
TEST
DISTRIBUTORS
SDS
CVS
CIP
Jopie van Rooyen
19 FII
CDQ
SAVII
Mindmuzik Media
IDEAS workbook
COP
??
APPLICATION IN
INDUSTRIAL/
PSYCHOLOGICAL
FIELDS
Career guidance and
counselling
Personal development and
growth
Cost Pricing Analysis
• Option 1: Purchasing a “report”
– Advised for non-accredited users and for individual testing on
once-off basis
– Some test can be expensive (up to R3,000 per report)
– Suppliers include SHL, Jopie van Rooyen, and Thomas
International
– Benefit: initial costs is low
Computer generated reports is time efficient
– Downfall: ongoing high costs
Potential misuse of reports due to lack of control
Cost Pricing Analysis
• Option 2: Purchasing Material: Software or Paper-and-Pencil version
– Advised for accredited users and group or large-scale testing on
ongoing basis
– Initial purchasing of material and software can be expensive
(need manuals, booklets, answer sheets, scoring stencils, etc –
between R2,000 – R25,000)
– Suppliers include Minimize, M&M Initiatives, PsyTech
– Benefit: ongoing assessments at minimal fee
– Downfall: initial start-up fees can be high
Software need ongoing purchasing of credits for reporting
Manual interpretation, scoring and reporting is time consuming
Key notes
Legal aspects of Psychological Assessment
The Constitution of the Republic of South
Africa:
–
Chapter 2: Bill of Rights “is a cornerstone of democracy in South
Africa - democratic values of human dignity, equality and freedom”.
–
Section 9(3): “The state may not unfairly discriminate directly or
indirectly against anyone on one or more grounds, including race
gender, sex, and pregnancy, marital status, ethical or social origin,
colour, sexual orientation, age, disability, religion, conscience,
belief, culture, language and birth.
–
Section 9 (4): “No person may unfairly discriminate directly or
undirectly against anyone on one or more grounds in terms of
subsection (3).
Employment Equity Act, No.55
(1998)
• Section 8: “Psychological testing and other
similar assessments of an employee are
prohibited unless the test or assessment being
used –
– Has been scientifically shown to be valid
and reliable;
– Can be applied fairly to all employees;
– Is not biased against any employee or
group.
EEA (continued)
•
Section 11: Burden of proof.
•
Section 15: Affirmative action.
•
Subsection 20(3): “capacity to acquire, within a
reasonable time, the ability to do the job”
•
Code of Good Practice on the Integration of
Employment Equity into Human Resources
Policies and Practices: Section 10.
The Labour Relations Act,
No. 66 (1995)
• Chapter VIII: Unfair discrimination and unfair
labour practices – same grounds as the
Constitution.
• Chapter VIII: Unfair discrimination and unfair
labour practices – same grounds as the
Constitution.
Legal Cases
•
Association of Test Publishers of South Africa Saville &
Holdsworth (SA) and the Chairperson of the Professional
Board for Psychology
•
Dispute: ATP: Any individual (unregistered) should be able to
conduct psychometric assessments? HPCSA: Only registered
professionals !
•
Ruling (19 February 2010): In favour of ATP SA. “The court order
declares the notice published on the 10th November 2008, which
stated that “it is not permissible to use unregistered persons to
render psychological services including the administration of tests,
instruments or techniques”, to be void and of no force and
effect.
•
Current status: This ruling will stand until further notice. WATCH
THIS SPACE!!!!
Legal Cases (continued)
• Buthelezi v Amalgamated Beverage
Industries
[1999] JOL 5086 (LC)
• Hendricks v SA Airways
[2002] JOL 10382 (LC)
• NUM obo Moeng / Douglas Colliery
[2007] 7 BALR 647 (CCMA)
What will the long arm of the law do
to you?
• Fines: Employment Equity Act
• Costs: CCMA, Labour Court,
Constitutional Court
Closing statement
• Assessments play an important role in the measuring of
performance, selection, recruitment, etc. in the
workplace, schools and various other institutions.
• Assessment Practitioners should today more than ever
be aware of the responsibilities that the legal use of
these powerful instruments require.
• Our history has not been without bumps, hurdles and
mountains, but it is our responsibility to promote a fair
and unbiased future and set an example for the rest of
the world. The current limitations and challenges that we
face can only be achieved through co-operation,
adhering to legislation and high ethical standards.
We rest our case…
The Regulation of Psychological Assessment
Regulation by
Legislation
Regulation by
Professional
Bodies
SelfRegulation
• LEGISLATION
 Employment Equity Act (1998) – different to all other countries, in
that we are asked to proactively defend our position with regards to
psychological assessment
• PROFESSIONAL BODIES
• (Responsibilities: Training, Accreditation, Test classification)
 HPCSA -- might not be in a position to efficiently regulate access to
and use of tests
 Psychometric Committee of the board of Psychology – falls under
HPCSA
 International Test Commission (2001)
 Society for Industrial and Organisational Psychology of South Africa
 People Assessment in Industry
 Human Sciences Research Council
• PROFESSIONAL BODIES (cont.)




South African Board for People Practices
Association of test publishers
Test Developers/Publishers
Any others? What about universities?
• SELF-REGULATION
 Strategy of self-regulation (Paterson & Uys, 2005) – practitioners
empowered to make informed decisions through access to
information and adequate training
 Training = better decision making with regards to selection of
(registered) tests
 Continuing professional development
 Personal moral compass
• ADDITIONAL ISSUES TO CONSIDER
 Conflicting policies (e.g. Employment Equity Act (1998) –
assessments must not discriminate against groups, but
can be used to assist with addressing previously
disadvantaged groups)
 Personal morals vs Ethical policies
 Canadian Psychological Association – ethical principle
ranking to assist decision making
Competency Based Assessment -
Competency Based Assessment
Development
• What is competency-based testing?
– Competencies
• Knowledge, Skills, Abilities
– Job Analysis
– Effective in Application
– Movement from intelligence
testing
Defining Competency Based Assessment
• A way to monitor and assess the competences a person
has, no matter where or how these competences were
gained.
• Three major principles differentiating competence-based
assessment from other methods used to assess are:
– Assessment of evidence
– Current abilities
– Standards or competence
Benefits of Competency Based Assessment
• An assessment of past behaviour can assist you in
predicting future behaviour
• Multiple assessment techniques are
used to verify the results obtained
• The competencies measured are related to successful
job execution
• Objectivity of evaluation is ensured through the utilisation
of trained assessors
Recognition of Prior Learning
“Recognition of prior learning is a process whereby, through
assessment, credit is given to learning which has already been
acquired in different ways”. – SAQA
• Discuss practical example of the RPL process
(real life case study)
Benefits of RPL
RPL is seen to have the capacity:
• to contribute to redress equity by opening up more ways for
people to attain qualified status;
• enable more people to reach higher levels of qualification and
expertise by beginning with an acknowledgement of existing
skills and knowledge;
• contribute to enhancing international economic
competitiveness by building on often invisible and
unacknowledged workforce skills;
• and offer the first step in attaining the goal of developing a
multiskilled and flexible workforce by acting as an auditing tool
to quantify existing competence.
Competency Based Assessment Series (CAS)
• Designed by JvR professionals and associates
• Purpose is to provide their clients with a range of uniquely South
African assessment centre exercises
• the exercises are designed to evaluate performance against certain
job relevant competencies and can contribute to an informed
selection decision.
• The exercises can be categorised into four groups, namely:
1.
2.
3.
4.
In-basket Exercises
Group Exercises
Strategy and/or Presentation Exercises
Role Play Exercises
• All the exercises are job relevant and assess current as well as
future potential.
Examples of Competency Based Assessment
COMPETENCY ASSESSMENT SERIES (CAS)
The aims of the CAS exercises include:
• To identify behavioural competencies needed for a specific job.
• To assist the assessor in gaining a comprehensive understanding of
the candidates’ behavioural competencies.
• Provide additional, job-relevant, “tangible” information to supplement
that gathered with traditional psychometric tools.
• To determine the participants’ strengths and development areas.
• To use the information in combination with collateral information for
selection and development purposes in the workplace.
Examples of Competency Based Assessment
•
•
•
•
•
•
•
Case studies
In-basket exercises
Job sample tests/Skills Tests
Direct Observation
Oral/Written Tests
Portfolios
Simulation
Designing a Competency Based Battery
• Step 1: Planning
– Purpose
• Step 2: Establish Competencies
– Job Analysis
• Step 3: Develop assessment battery
– Structure
• Step 4: Implement and evaluate battery
Uses of Competency Based Assessment
• Within in the organisation
• In the South African context
– Advantages
– Disadvantages
• Fairness
– Unbiased, ethical
Key notes
Computerised Assessment
Before we start
•
•
•
•
•
•
CAPA
CBA
CBT
CAT
CBTI
IDT
Computer-based Assessment
Assessment
Generation
Assessment
Delivery
Storage,
Retrieval and
Transmission
Assessment
Scoring and
Interpretation
Taking a step back in History
• International:
1962
1970’s
1980’s
1990’s
2000’s
Taking a step back in History
• South-Africa
1970
Late 1970’s and early 1980’s
1979
1980’s
1986
1990
1993-1994
Advantages and Disadvantages
• Advantages:
–
–
–
–
–
–
More enjoyable
Ultimate levels of standardization
Biasing effect is eliminated
Reduction of time
More information about test-takers
Spatial and perceptual measured to greater extend
Advantages and Disadvantages
• Advantages (continued)
– Voice activated and touch screen responses are
possible
– Assessments can be individually tailored
– Greater element of control
– Number of assessment practitioners and assistants
needed
– Scoring by a computer
– Increased test security
Advantages and Disadvantages
• Disadvantages
– Eish Eskom
– Hardware / software
compatibility & capability
– Screen size
– Machine availability
– Socio-cultural and linguistic
factors
– System failure
Advantages and Disadvantages
• Disadvantages (continued)
–
–
–
–
–
–
–
Copy right violation
Lack of security
Problems with confidentiality
Skilled clinical judgement is overlooked
Difficult to detect certain problems in software
High costs in item development
Costly
Advantages and Disadvantages
• Disadvantages (continued)
– Vital qualitative information
not assessed
– Human-computer interface
issues
– Lack of computer literacy
Computer-based Interpretation
• Four approaches to CBTI are recognized:
– Scoring reports
– Descriptive reports
– Actuarial reports
– Computer-assisted clinical reports
CAT
• What is it
• Purpose
• Characteristics
• Measurement Advantages
Dynamic Assessment
• What is it?
• Increased attention
• What does it involve?
• South Africa
Assessing Potential
• Effective management tool
• Time and cost effective
• Usage
Assessing Potential
• Why will we use it?
• Present and potential
• Opportunity to be measured fairly.
• Is suitable at all levels
• Scientifically proven valid and reliable
• Learning takes place during the assessment
• Is extremely time-saving
• Uses computerised adaptive testing
Assessing Potential
• Advantages:
– Fair
– Suitable for use at all levels of ability;
– Valid and reliable indication of potential to
cope with training;
– Time saving;
– Uses CAT which allows for all ability levels.
Item Response Theory
• Definition
• Higher an individual’s ability level, the
greater the chance of getting an item
correct.
• Each item is referenced
Item Response Theory
10
9
8
7
6
5
4
3
2
1
0
Item 2
Item 1
Low Ability
Average
High Ability
Good practice and Ethical considerations
• The International Test Commission (ITC)
– Give due regard to technological
– Attend to quality issues in CBT and internet
testing;
– Provide appropriate levels of control
– Make appropriate provision for security
Good practice and Ethical considerations
• Minimum professional and ethical
standards:
– Competence
– Potential utility
– Choose a technically (psychometrically)
sound computer-based test
– Equivalence of paper and computer-based
version
– Consider the human factors and issues of
fairness
Good practice and Ethical considerations
– Prepare test takers
– Verify the identity of test-takers
– Closely supervise the administration
– Unsupervised psychological test
– Contingency
– Securely stored.
– Computer scoring system
– Interpretation of results
– Debrief test-takers
The future – high definition video and
virtual reality
• Multimedia
• A five-dimension framework
– Innovations in item format
– Innovations in response
– Innovations in media inclusion
– Innovations in the extent of interactivity
– Innovations in scoring
– Authenticity
The Future of Psychological Assessments
Research building on existing knowledge
•Enhancing consumer psychology
•Reduced costs
•System generated reports
•Dynamic assessment of intelligence
•Construction of new adaptive tests
The Future Role of the HPCSA
•Need for an all encompassing body to:
Monitor test use
Advise practitioners
Research and review tests with information centrally
available
To monitor and coordinate test development,
adaptation, and updating (Foxcroft, 2004)
• Control and regulation
•Increased availability of information via internet
Artificial Intelligence
•Virtual Reality – a new tool for psychological assessment
(Fernandez-Ballesteros, 2009)
•Computer assisted assessment
•Ethical considerations
•Control and access to electronic assessments
•Closer examination of the tester-testee relationship
The Future of Assessments in South Africa
• Locally developed tests appropriate for all cultures within
SA
• Leadership tests appropriate for SA context
• Explore other intelligences – cosmic, spiritual, moral,
emotional, transcendental contributions
• Stimulating use of assessments by smaller organisations
and NGOs
• Industrial psychologists making
significant contributions locally and
internationally
The Future of Assessments in South Africa
• Psychometric tests – most effective of predicting
behaviour (Mittner, 1998)
• Tests 4 X more effective than screening interviews (Van
der Walt, 1998)
• Employment Equity Act – definite move towards making
assessment techniques more scientific (Eckstein, 1998)
• To be used as additional aid in decision making i.e. not in
isolation
• Retain professional level for conducting psychometric
assessment.
The Future of Assessments in South Africa
• (Bartram & Coyne, 1998) in a recent world-wide survey
found lack of psychologists involvement in testing and
use of tests
– Found 60% of test users were not specifically trained
in use of psychological tests.
– Therefore recommend training and increasing public
understanding.
– Gregoire, (1999), recommend academic education,
continued education, and more publications on tests
and assessment methods.
The Future of Assessments in South Africa
• Cook (1997), trend towards assessment centres and
competency based assessment in changing sociopolitical and economic context. [some psychological
factors that competence measures cannot provide, i.e.
learning and potential]
• Consideration should be given to development of new
tests with urgent special emphasis on culture-fairness.
• Validation of tests in different organisations that they are
used in. Cook (1997),
Conclusion: History
 In the past the same tests were
used for everyone.
 Psychologists used international
tests.
 Testing was for a long time viewed
as discriminatory and unjust.
 Practitioners are becoming aware
of the advantages of fair
assessment.
 The implementation of crossculturally fair tests.
Conclusion: Assessment Areas
COGNITIVE
PERSONALITY
Assessment
Areas
BEHAVIOURAL
INTEREST
Conclusion: Legislation
 Changes in legislation.
 The EEA has been established in
1998 to ensure more equitable
and fair Psychological testing.
 Tests need to be scientifically valid
and reliable;
 tests should be able to be applied
fairly to all employees; and
 tests should not be biased against
any employee or group.
Conclusion: Ethics
 Psychological tests can act as
a disabling factor.
 Practitioners therefore have to
be more cautious and informed
and should take responsibility
to ensure that a test can be
fairly applied.
 Practitioners should act in an
ethical way before, during and
after psychological tests.
Conclusion: Competency Based Assessment
 Focus on the essential
behaviours required to perform a
specific job.
 Focussing on individual
differences in terms of workrelated constructs relevant to job
performance.
 This contributes to keep tests fair
and relevant as required by the
law.
Conclusion: Computerised Assessment
 In our modern society there is new
technology on the market every
day and everything is being
computerised in one way or
another as well as psychological
testing.
 There is a lot of advantages to
computerized testing we just need
to make sure that it remains
unbiased, ethical, valid and
reliable for our rainbow nation.
Conclusion: Challenges
 According to research the most
frequently cited hindrance to the
administration of psychological
tests is language.
 When administering individual
intelligence tests, psychologists
often argue that it is justifiable to
administer the measure in
English, irrespective of whether
English is the first or second
language of the test-taker.
Continue...
Conclusion: Challenges
 It is important to determine
whether the performance on the
test reflects the testee’s actual
ability and not the testee’s
competence in the test language.
 Translation of a test is not a quickfix.
 Changing some wording in an
item/question can draw a question
mark over the construct, score and
predictive comparability.
Continue...
Conclusion: Challenges
 Other factors that need to be taken
into account: cultural and
environmental factors, age and
gender.
 Test anxiety has an effect on test
resuls.
 Anxiety can hamper a person’s test
performance.
 Employees may also start to view tests
as unfair and invalid.
 Explain everything in simple terms.
“If you talk to a man in a language he understands,
that goes to his head. If you talk to him in his
language, that goes to his heart.” Nelson Mandela
“ The most important single ingredient in the
formula of success is knowing how to get
along with people” Theodore Roosevelt.
Crossword Competition