Nadia Brits
Supervisor: Prof. Deon Meiring
ACSG Conference
16 March 2011
EVOLUTION OF THE CONSTRUCT-VALIDITY DEBATE
Department of Industrial Psychology
University of Stellenbosch
Department of Industrial Psychology

Faculty of Economic and Management Sciences
TODAY’S JOURNEY
SACKETT &
DREHER (1982)
CONSEQUENCES
OF INVALID
CONSTRUCTS
DON’T TAKE
AWAY MY
DIMENSIONS
ABANDON
DIMENSIONS
DESIGN FIXES:
TRIED &
TESTED
EPIC OF THE CV
DEBATE
CRITICS OF
LANCE (2008)
ACs AT A
CROSSROADS
DESIGN FIXES
BACKGROUND
•
Sufficient evidence for CV exists (Arthur et al., 2000; Thornton &
Gibbons, 2008)
•
The validity of ACs is questioned
•
CONSTRUCT VALIDITY PUZZLE (Lievens, Chasteen, Day &
Christiansen, 2006)
•
CONSTRUCT-RELATED VALIDITY PARADOX (Arthur, Day & Woehr,
2008)
•
SO-CALLED CONSTRUCT VALIDITY PROBLEM (Howard, 1997)
ORIGINS OF THE CONSTRUCT-VALIDITY DEBATE
• Sackett & Dreher (1982)
• Expectations:
• CONVERGENT VALIDITY
• DISCRIMINANT VALIDITY
• ratings cluster according to DIMENSIONS, not exercises (which
became known as EXERCISE EFFECT)
• Similar research followed
CONSEQUENCES OF INVALID CONSTRUCTS
• SELECTION of applicants based on AC performance ratings
• FEEDBACK based on AC results
• WASTE OF TIME AND MONEY
• Practitioners provide MISLEADING services to companies who appoint
them to design and run ACs
DESIGN FIXES
• DEFINITION of dimensions
• NUMBER of dimensions
• TRANSPARENCY of dimensions to candidates
• Behavioural CHECKLISTS
• Type of SCORING METHOD
• ASSESSOR training
• TYPE of assessor (Lievens & Klimoski, 2001; Gaugler et al.,
1987)
• EXPERIENCED assessors (Kolk et al. 2002; Thornton &
Rupp, 2005).
DESIGN FIXES: TRIED AND TESTED
• Design fixes show LITTLE, INSIGNIFICANT IMPROVEMENT
• Controlling assessor variance has only a MARGINAL EFFECTS on
construct-validity
• EXERCISE EFFECTS STILL DOMINATE
• What else can we do?
ABANDONING DIMENSIONS
• Move from dimensions-based ACs to task-based AC
• Why?
• Recurring exercise effects
• Exercise factors show positive correlations with
external performance criteria
DON’T TAKE AWAY MY DIMENSIONS
• useless to learn a task that the participant might never
encounter again
• Human performance is multidimensional
• cannot capture the full complexity of a real job (novel, non-
repetitive tasks)
• Only dimensions will allow generalisation of AC results
• Research supports dimensions (Connelly et al., 2008; Melchers
& Konig, 2008; Bowler & Woehr, 2006)
• knowledge about exercise-based ACs still lacking (Lievens,
2008).
RESULT : 3 LINES OF RESEARCH
• EXERCISES and DIMENSIONS explained same amount of
variance – 34% (Lievens & Conway, 2001)
• EXERCISE EFFECT were larger than dimension effects – 52%
(Lance et al., 2004)
• EXERCISES explained most variance – 33%, DIMENSIONS
ALSO explained substantial amount - 22% (Bowler & Woehr,
2006)
EPIC OF THE HEATED DEBATE
• LANCE (2008):
“ACs do not work they way they are
supposed to”
• NOT SUFFICIENT EVIDENCE for 3 requirements of CV
• Supports the EXERCISE-BASED MODEL
• DISCARD DESIGN FIXES
• Exercise effects represent cross-situational specificity in
candidate performance, not method bias...TRUE
VARIANCE
COMMENTS/CRITICS TOWARDS LANCE (2008)
• ignores evidence of both dimension and exercises account for
variance in AC performance (Howard, 2008)
• design fixes should continue to be investigated (Schleicher et
al., 2008; Howard, 2008; Arthur, Day & Woehr, 2008; Melchers
& König, 2008)
• Lance received support for candidates’ inconsistent
performance....true variance
• individuals use set of stable skills and can adjust and adapt
KSA’s according to the situation
• Some people perform better than others in a specific
exercise
ACs AT A CROSSROADS
• Persistent EXERCISE EFFECTS
• performance variability: more situation-specific (57%)
than situation-consistent (43%) (Hoeft and Schuler’s (2001)
• Lack of consensus on solutions and future directions
• Walter Mischel (1968)
• Consistency in behaviour ONLY when situational
factors are acknowledged and taken into account
• TRAIT-ACTIVATION THEORY (TAT)
TRAIT-ACTIVATION THEORY (TAT)
• person-situation interaction to explain behaviour on the
basis of responses to trait-relevant cues found in situations.
• SITUATION STRENGTH: strong vs weak situations
• SITUATION RELEVENCE: A situation is considered relevant
to a trait if it provides cues for the expression of traitrelevant behaviour
APPROACHES TO IMPLEMENT TAT
• Adapting the CONTENT of the exercise
• Influencing the INSTRUCTIONS of each exercise to guide
participants what type of behaviour to show
• training ROLEPLAYERS on how to interact in order to elicit
certain behaviours from participants
• LARGE NUMBER OF SHORTER EXERCISES to obtains
samples of performance on a number of independent tasks
•
Halaand and Christiansen (2002) found stronger
convergence of AC ratings
• GAP IN RESEARCH about effectiveness of TAT
TO SUMMARISE
SACKETT &
DREHER (1982)
CONSEQUENCES
OF INVALID
CONSTRUCTS
DON’T TAKE
AWAY MY
DIMENSIONS
ABANDON
DIMENSIONS
DESIGN FIXES:
TRIED &
TESTED
EPIC OF THE CV
DEBATE
CRITICS OF
LANCE (2008)
ACs AT A
CROSSROADS
DESIGN FIXES