To Use the Teamwork Test -- Or
Not? A Psychometric Evaluation
Janet L. Kottke
California State University, San Bernardino
Kimberly A. French
University of South Florida
Rhiannon J. Kirchner
California State University, San Bernardino
Presented to PTC of Southern California, July 23, 2013
TKSA Development
• Stevens and Campion (1994; 1999) develop
the Teamwork KSA Test
– 35 items, multiple choice format (4 options,
dichotomous scoring)
• Reviewed teamwork literature
– Identified 14 KSAs identified with effective
teamwork
Kottke, French, & Kirchner
PTC July 23, 2013
2
14 KSAs → 5 facets
• These 14 KSAs represent
– Five facets
•
•
•
•
•
Conflict resolution
Collaborative problem solving
Communication
Goal setting and performance management
Planning and task coordination
Kottke, French, & Kirchner
PTC July 23, 2013
3
5 facets → 2 dimensions
• Two higher-order dimensions:
– Interpersonal KSAs
• Conflict resolution
• Collaborative problem solving
• Communication
– Self-management KSAs
• Goal setting and performance management
• Planning and task coordination
Kottke, French, & Kirchner
PTC July 23, 2013
4
Teamwork KSA Structure
O’Neill, Goffin, & Gellatly, 2012, p. 37
Kottke, French, & Kirchner
PTC July 23, 2013
5
Knowledge, Skill, and Ability (KSA) Requirements for Teamwork
I.
INTERPERSONAL KSAs
A. Conflict Resolution KSAs
1. The KSA to recognize and encourage desirable, but discourage undesirable, team conflict.
2. The KSA to recognize the type and source of conflict confronting the team and to implement an
appropriate conflict resolution strategy.
3. The KSA to employ an integrative (win-win) negotiation strategy rather than the traditional
distributive (win-lose) strategy.
B. Collaborative Problem Solving KSAs
4. The KSA to identify situations requiring participative group problem solving and to utilize the
proper degree and type of participation.
5. The KSA to recognize the obstacles to collaborative group problem solving and implement
appropriate corrective actions.
C. Communication KSAs
6. The KSA to understand communication networks, and to utilize decentralized networks to
enhance communication where possible
7. The KSA to communicate openly and supportively, that is, to send messages which are: (1)
behavior- or event-oriented; (2) congruent; (3) validating; (4) conjunctive; and (5) owned.
8. The KSA to listen nonevaluatively and to appropriately use active listening techniques.
9. The KSA to maximize consonance between nonverbal and verbal messages, and to recognize
and interpret the nonverbal messages of others.
10. The KSA to engage in ritual greetings and small talk, and a recognition of their importance.
II. SELF-MANAGEMENT KSAs
D. Goal Setting and Performance Management KSAs
11. The KSA to help establish specific, challenging, and accepted team goals.
12. The KSA to monitor, evaluate, and provide feedback on both overall team performance and
individual team member performance.
E. Planning and Task Coordination KSAs
13. The KSA to coordinate and synchronize activities, information, and task interdependencies
between team members.
14. The KSA to help establish task and role expectations of individual team members, and to
ensure proper balancing of workload in the team.
(Adapted from Steven & Campion, 1994, Table One, p. 505)
6
Sample Question
• When you set work goals for yourself or your
work team, what are the best goals to set?
A. Set goals to "do your best."
B. Set general and broad goals.
C. Set specific and detailed goals.
D. Set easy and simple goals.
Kottke, French, & Kirchner
PTC July 23, 2013
7
Criterion validity
• Several studies have found predictability of
supervisor and observer ratings of team
effectiveness (rs range .20 to .56)
– Ellis, Bell, Ployhart, Hollenbeck, & Ilgen, 2005
– Leach, Wall, Rogelberg, & Jackson, 2005
– McClough & Rogelberg, 2003
– Morgeson, Reider, & Campion, 2005
• Mixed results in predicting team member peer
ratings
Kottke, French, & Kirchner
PTC July 23, 2013
8
Questions about predictive validity
• Miller, 2001
– Issues raised about Stevens & Campion validation
studies
• Team performance was predicted less well (r = .44) than task
work (r = .56)
• TKSA predicted team performance only slightly better than
did an aptitude test
– But: TKSA added incremental variance beyond GMA
• Aptitude data suggest individual aptitude, not teamwork
aptitude measured
– Miller’s own study finds no significant relationship
between team performance and TKSA scores
Kottke, French, & Kirchner
PTC July 23, 2013
9
Questions about psychometrics
• O’Neill, Goffin, & Gellatly (2012) examine the
TKSA
– Comprehensive analysis
•
•
•
•
•
•
Reviewed all known studies using TKSA
Classic test theory analysis
Confirmatory factor analysis
Exploratory factor analysis
Convergent, discriminant validation
Criterion validity analysis
– Disappointing results all around
Kottke, French, & Kirchner
PTC July 23, 2013
10
Our plan
• Replicate O’Neill CTT & CFA analyses with fresh
samples
• Data sources
– Had used the TKSA Test for a group project in several
classes (sample 1)
– Collected new data to address unexpected finding
from sample 1 that aptitude was negatively related to
teamwork interest (sample 2)
– Collected data for a thesis project on teamwork and
task interdependence (sample 3)
Kottke, French, & Kirchner
PTC July 23, 2013
11
The 3 samples
• Students from a mid-size southwestern
University.
– Sample 1 = 251 college students
• Cohort groups, upper level undergraduate-level psychology
course, term-length group project.
– Sample 2 = 279 college students
• Recruited from psychology and business courses
– Sample 3 = 404 college students
• From upper-level undergraduate courses in both psychology
and business departments, either a long-term (i.e., several
weeks) or term-length group project
Kottke, French, & Kirchner
PTC July 23, 2013
12
Work experience of samples
• Although students, vast majority were
employed
– 96% in sample 1
– 82.4% in sample 2
– 88.4% in sample 3
• Work experience ranged from means of 5.5 to
7.3 years
– SDs ranged from 5.6 to 6.3 years
Kottke, French, & Kirchner
PTC July 23, 2013
13
Table 1.
Demographic Descriptive Statistics
Sample 1
M
SD
25.1
6.6
7.3
5.9
-
Sample 2
M
SD
23.1
7.0
5.5
6.3
1.8
2.0
4.4
11.9
Sample 3
M
SD
23.9
6.1
5.9
5.6
-
Gender (% Female)
Ethnicity
African American
Asian American
Hispanic
82.1%
74.9%
65.1%
9.2%
4.4%
45.4%
11.5%
4.7 %
47.3%
8.2%
4.5%
45.0%
White
Other
Education
Freshman
Sophomore
Junior
Senior
Graduate
Other
Respondents with work experience
Respondents with supervisor experience
35.1%
6.0%
26.9%
8.2%
29.7%
11.9%
0.4%
2.8%
43.4%
51.0%
2.0%
16.5%
14.3%
33.3%
34.1%
0.4%
0.0%
4.7%
50.2%
42.6%
0.2%
0.4%
96.0%
40.2%
0.4%
82.4%
24.7%
0.5%%
88.4%
-
Demographic Characteristic
Age
Work experience (years)
Number of term-length group projects
Number of work group projects
Kottke, French, & Kirchner
PTC July 23, 2013
14
Analyses conducted
• Analytic Approach
– In each sample we evaluated
• Classical test theory
– Item, subscale, and total measure properties,
» Means, standard deviations, and item-total correlations
– Subscale and total measure means, standard deviations, and
internal reliability
• Structure using categorical confirmatory factor analysis
(Mplus)
– One factor model, two factor model, and a five factor model
» For multi-factor models, both a covaried model (all factors
were allowed to correlate), as well as a higher-order model
(includes higher-order general teamwork KSA factor)
Kottke, French, & Kirchner
PTC July 23, 2013
15
Supplemental
• Supplemental analysis
– To ensure psychometric issues are not a result of
student samples, ran classical test analyses on the
subset of working respondents from each sample.
– Results from working samples were not substantially
different than full samples
• For example, average absolute difference in factor and total
means was .11 for sample 1 [SD = .07], .21 for sample 2 [SD
= .19], and .14 for sample 3 [SD = .21].
• The average difference in inter-item correlations was .02 [SD
= .01])
• Thus, only the results from the full samples are presented
here.
Kottke, French, & Kirchner
PTC July 23, 2013
16
RESULTS: Item means, SDs, item total
correlations
Table 2.
Teamwork KSA Test item-level means, standard deviations, and item-total correlations for all samples.
Sample 1
Subscale Item Statistics
M(p)
Conflict resolution mean
Conflict resolution SD
Communication mean
Communication SD
Collaborative problem solving mean
Collaborative problem solving SD
Goal setting & performance management mean
Goal setting & performance management SD
Planning and task coordination mean
Planning and task coordination SD
Grand mean
Grand SD
0.69
0.16
0.54
0.13
0.42
0.14
0.60
0.12
0.64
0.10
0.55
0.16
Sample 2
SD
ri-f
ri-h
ri-t
0.43
0.07
0.48
0.02
0.47
0.02
0.47
0.04
0.47
0.03
0.47
0.04
0.14
0.09
0.13
0.06
0.13
0.09
0.20
0.07
0.15
0.02
0.14
0.08
0.22
0.14
0.17
0.06
0.18
0.16
0.22
0.04
0.18
0.02
0.19
0.10
0.26
0.15
0.17
0.07
0.20
0.17
0.23
0.05
0.24
0.07
0.21
0.12
M(p)
0.64
0.19
0.54
0.13
0.42
0.13
0.58
0.12
0.63
0.12
0.54
0.15
Sample 3
SD
ri-f
ri-h
ri-t
0.45
0.06
0.48
0.02
0.48
0.02
0.48
0.01
0.47
0.03
0.48
0.03
0.08
0.05
0.08
0.09
0.14
0.09
0.07
0.07
0.13
0.05
0.10
0.08
0.22
0.11
0.13
0.11
0.16
0.10
0.13
0.07
0.17
0.09
0.15
0.10
0.17
0.14
0.15
0.11
0.18
0.10
0.19
0.05
0.23
0.09
0.17
0.10
M(p)
0.67
0.20
0.53
0.15
0.40
0.13
0.59
0.11
0.64
0.13
0.54
0.16
SD
ri-f
ri-h
ri-t
0.44
0.07
0.48
0.02
0.47
0.02
0.48
0.01
0.47
0.04
0.47
0.03
0.05
0.05
0.11
0.07
0.11
0.09
0.19
0.06
0.15
0.03
0.12
0.07
0.16
0.11
0.14
0.08
0.14
0.08
0.25
0.02
0.22
0.02
0.17
0.08
0.17
0.12
0.16
0.07
0.16
0.09
0.28
0.02
0.24
0.05
0.19
0.08
Notes. M(p) = item means, SD = item standard deviations, ri-f = corrected item-factor correlation,, ri-h = corrected item-higher order factor correlation, ri-t =
corrected item-total correlation.
Kottke, French, & Kirchner
PTC July 23, 2013
17
More Results: Scale means, SDs, inter
item correlation means, KR 20s
Table 3.
Teamwork KSA Test facet and total means, standard deviations, reliability, and item correlation summary for all samples
Sample 1
Sample 2
Inter-item
correlation
Sample 3
Inter-item
correlation
Inter-item
correlation
Teamwork KSA Test
Score
No. of
Items
M
SD
K-R 20
M
Range
M
SD
K-R 20
M
Range
M
SD
K-R 20
M
Range
Interpersonal KSAs
Conflict Resolution
Communication
Collaborative Problem
Solving
Self-Management KSAs
Goal Setting &
Performance
Management
25
4
12
13.05
2.75
6.49
3.55
.97
2.04
.58
.25
.36
.06
.09
.05
-.12, .32
-.06, .26
-.11, .26
12.84
2.55
6.53
3.36
.95
1.89
.52
.15
.24
.05
.05
.03
-.17, .23
-.01, .14
-.10, .22
12.72
2.67
6.41
3.29
.91
1.99
.51
.10
.33
.04
.03
.04
-.15, .19
-.11, .16
-.11, .19
9
10
3.82
6.23
1.71
1.99
.34
.49
.05
.09
-.12, .23
.00, .27
3.76
6.06
1.72
1.87
.35
.39
.06
.06
-.08, .22
-.08, .22
3.63
6.11
1.64
2.10
.28
.54
.04
.11
-.07, .18
-.01, .26
5
3.01
1.29
.40
.12
.04, .24
2.90
1.16
.17
.03
-.06, .14
2.93
1.29
.38
.11
.03, .23
5
35
3.22
19.29
1.20
4.84
.30
.69
.08
.06
.00, .18
-.14, .32
3.17
18.81
1.19
4.69
.27
.65
.07
.05
-.05, .22
-.17, .25
3.19
18.82
1.21
4.70
.32
.67
.09
.05
-.01, .26
-.15, .26
Planning & Task
Coordination
Teamwork KSA Overall
Kottke, French, & Kirchner
PTC July 23, 2013
18
Confirmatory Factor Analysis
Results on following slide
Kottke, French, & Kirchner
PTC July 23, 2013
19
Table 4.
Summary of confirmatory factor analysis results
Sample
Sample 1
Model
Five factor, covariedb
Revised five factor, covarieda
2
198.12*
df
158
CFI
.85
RMSEA
.032
Absolute
Loading
Mean
.34
309.23**
152
.41
.064
.41
Five factor, higher order
Revised five factor, higher
359.39**
154
.24
.072
.44
-.03 to 1.00
202.701*
159
.84
.033
.34
-.51 to .82
202.95*
156
.36
.066
.36
-.09 to 1.00
202.701*
159
.84
.033
.34
-.10 to .82
211.12
183
.88
.022
.29
-.08 to .58
308.61**
182
.47
.048
.38
-.09 to .71
Two factor, higher order
Did not converge
Revised two factor, higher ordera
Single factor
Five factor,
covariedb
Revised five factor,
covarieda
Five factor, higher order
Revised five factor, higher
Did not converge
ordera
Two factor, covaried
368.40**
182
.22
.058
.38
-.09 to 1.00
212.79
184
.88
.023
.30
-.49 to.62
Two factor, higher order
Revised two factor, higher
Did not converge
ordera
Single factor
Sample 3
Five factor,
covariedb
Revised five factor,
covarieda
363.73**
182
.24
.057
.32
-.08 to 1.00
212.54
184
.88
.022
.30
-.08 to .62
327.42**
213
.78
.036
.33
-.01 to .61
471.13**
211
.51
.055
.40
.01 to .71
Five factor, higher order
Did not converge
Revised five factor, higher ordera
Two factor, covaried
565.28**
209
.32
.065
.40
.01 to 1.00
345.70**
214
.75
.039
.32
-.55 to .52
Two factor, higher order
Revised two factor, higher
Single factor
-.05 to .92
Did not converge
ordera
Two factor, covaried
Sample 2
Loading Range
-.10 to .83
Did not converge
ordera
527.38**
211
.40
.061
.32
.03 to 1.00
353.40**
214
.74
.040
.30
-.01 to .52
*p < .01, **p < .001
Kottke, French, & Kirchner
PTC July 23, 2013
20
Discussion
• To be fair to the test authors
– They recommend the subscales be used for training and
developmental purposes only
– In all of their publications, they use only total scores
– So, whether the test authors intended a hierarchical
structure is not a certainty
• Yet, with their careful development that categorized the
14 KSAs into specific facets, one would expect
meaningful structure
• Criterion validity has been found, but issues here as well
– TKSA test correlates highly with general mental ability
– Some incremental predictability
Kottke, French, & Kirchner
PTC July 23, 2013
21
Conclusion
• Very limited evidence for adequate
psychometric properties
• Use with caution
– Recognize the test’s limitations
Kottke, French, & Kirchner
PTC July 23, 2013
22
Thank you
Kottke, French, & Kirchner
PTC July 23, 2013
23
Questions?
Kottke, French, & Kirchner
PTC July 23, 2013
24
References
Ellis, A. J., Bell, B. S., Ployhart, R. E., Hollenbeck, J. R., & Ilgen, D. R. (2005). An evaluation
of generic teamwork skills training with action teams: Effects on cognitive and skillbased outcomes. Personnel Psychology, 58, 641-672.
Leach, D. J., Wall, T. D., Rogelberg, S. G., & Jackson, P. R. (2005). Team autonomy,
performance, and member job strain: Uncovering the teamwork KSA link. Applied
Psychology: An International Review, 54, 1-24.
McClough, A. C., & Rogelberg, S. G. (2003). Selection in teams: An exploration of the
Teamwork Knowledge, Skills, and Ability test. International Journal of Selection and
Assessment, 11, 56-66.
Miller, D. L. (2001). Reexamining teamwork KSAs and team performance. Small Group
Research, 32(6), 745.
Morgeson, F. P., Reider, M. H. & Campion, M. A. (2005). Selecting individuals in team settings:
The importance of social skills, personality characteristics, and teamwork knowledge.
Personnel Psychology, 58, 583-611.
O'Neill, T. A., Goffin, R. D., & Gellatly, I. R. (2012). The knowledge, skill, and ability
requirements for teamwork: Revisiting the teamwork‐KSA test's validity. International
Journal of Selection And Assessment, 20, 36-52.
Stevens M. J. & Campion M. A. (1999). Staffing work teams: Development and validation of a
selection test for teamwork settings. Journal of Management, 25, 207-228.
Stevens, M. J. & Campion, M. A. (1994). The knowledge, skill, and ability requirements for
teamwork: Implications for human resource management. Journal of Management, 20,
503-530.
Kottke, French, & Kirchner
PTC July 23, 2013
25