Computer Applications
in Testing and Assessment
James P. Sampson, Jr.
Florida State University
Copyright 2002 by James P. Sampson, Jr., All Rights Reserved
Overview of Topics
• Computer-Assisted vs. Computerized
• Rationales for Computer-Assisted
Testing
• Potential Benefits
• Potential Problems
• Future Use
2
Computer-Assisted vs.
Computerized
• Both describe use of computer
technology in assessment process
• Used with nearly the same frequency
• Using the term computerized may
cause problems and foster
misunderstanding
3
Computerized
• Implies that computers replace humans
previously needed to complete a task
• May lead to practitioner under use or
misuse
• An inaccurate depiction of computer
technology in testing
4
Computerized
• Under Use
– Uninformed rejection of computer
technology
– View computers as performing inappropriate
functions
– Believe computers lead to dehumanization
5
Computerized
• Misuse
– View computers as effective in performing
functions not intended by developer
– Use of computers to compensate for
a lack of professional skill
6
Computer-Assisted
• Computer fulfills a support role
• Improves, not replaces the human
element
• More likely to avoid under use or misuse
7
Rationales for CAT
1. Focus on assigning humans and
computers to tasks best suited to the
capabilities of each
2. Develop previously unavailable
techniques to improve the quality of
tests and assessments
8
Potential Benefits of CAT
• Overview of topics:
– Enhanced validity and reliability
– Enhanced client integration of assessment
data in counseling
– Enhanced staff efficiency and
cost-effectiveness
9
Enhanced Validity and Reliability
• Overview of topics:
– Test Administration and Scoring
– Administration of Interviews and Checklists
– Computer-Based Test Interpretation
10
Test Administration and Scoring
• Greater degree of standardization
• More accurate results for people with
disabilities
• Availability of adaptive testing
• Collection of ancillary data (e.g.,
response manner and specific response
recording)
11
Enhanced Validity and Reliability
• Overview of topics:
– Test Administration and Scoring
– Administration of Interviews and Checklists
– Computer-Based Test Interpretation
12
Administration of Interviews
and Checklists
• More comprehensive topic coverage
• More consistent topic coverage
• Less evaluative tension from repeated
questioning by the practitioner
• Results less affected by time of day
• More honest client response
13
Enhanced Validity and Reliability
• Overview of topics:
– Test Administration and Scoring
– Administration of Interviews and Checklists
– Computer-Based Test Interpretation
14
Computer-Based Test
Interpretation (CBTI)
• Assists in the interpretation of test data
• Provides expanded & consistent
knowledge base
• Includes research data and
the clinical experience of other clinicians
• Standardizes computer functioning
• Provides consistency
15
Types of CBTI
• Descriptive interpretations
• Clinician-modeled interpretations
(renowned clinician type)
• Clinician-modeled interpretations
(statistical model type)
• Clinical actuarial interpretations
16
Descriptive CBTI
• Descriptive sentences related to specific
scales
• Scales interpreted independent of each
other
• Typically cryptic in nature
17
Clinician-Modeled CBTI
(Renowned Clinician Type)
• Replicates the interpretive judgment of a
single clinician
• Based on the clinician with the most
experience with the test
• Provides consistent results
• Mistakes also consistent
18
Clinician-Modeled CBTI
(Statistical Model Type)
• Replicates interpretive judgments
of a group of practitioners
• Includes a group comprised of
experienced clinicians
• Statistically valid model
• Incorporates the best consensus possible
among experts
19
Clinical Actuarial CBTI
• Provides narrative descriptions and
clinical hypotheses
• Based on clinical research findings for
particular score patterns
20
Potential Benefits of CBTI
• More comprehensive and objective
• Not subject to interpreter bias or
extraneous circumstances
• Provides rapid data processing
• Ready for use in early counseling sessions
• Helps practitioners organize and systematically
access extensive data bases
• Easily updated when new research is available
21
Potential Benefits of CBTI
• CBTI serves in a consulting role
• Practitioner integrates data from variety
of sources to create assessment reports
• Computerized interpretive reports only
used as adjuncts, not as the final word
22
Potential Benefits of CAT
• Overview of topics:
– Enhanced validity and reliability
– Enhanced client integration of assessment
data in counseling
– Enhanced staff efficiency and
cost-effectiveness
23
Enhanced Client Integration of
Assessment Data in Counseling
• Computer applications used as one
aspect of the therapeutic process
• CBTI reports in clinical settings are
typically not given to most clients
• Interpretive reports are typically provided
directly to clients in career counseling
– provides basic understanding of results
– outlines the decision-making process
– suggests follow-up resources
24
Enhanced Client Integration of
Assessment Data in Counseling
• Generalized test interpretations
– video-based technology
25
Potential Benefits of CAT
• Overview of topics:
– Enhanced validity and reliability
– Enhanced client integration of assessment
data in counseling
– Enhanced staff efficiency and
cost-effectiveness
26
Enhanced Staff Efficiency and
Cost-Effectiveness
• Computer-assisted testing is more costeffective
• Saves testing time
• Improves time utilization of support staff
• Gives practitioners more time to prepare
test interpretations, write test reports, and
work directly with clients on more
complex issues
27
Potential Problems Associated with
Computer Applications
• Overview of Topics
– Threats to Validity and Reliability
•
•
•
•
administration and scoring
test interpretation
ethical issues
ethical standards
– Ineffective Implementation
28
Threats to
Validity and Reliability
• Administration and Scoring
– Norms used from paper-and-pencil tests
– Blanket assumptions of equivalency
– Scoring errors
– Perceived inherent credibility - infallibility
– Client misunderstandings of instructions
29
Threats to
Validity and Reliability
• Test Interpretation
– Different reports from different software
developers
– Differences in CBTI reports and conclusion
reached by clinical interviewer
– Computer may be perceived adding
a level of scientific precision that does not
exist
30
Threats to
Validity and Reliability
• Misuse of CBTI
– Qualified practitioners may misuse CBTI as
a result of being overworked
– Unqualified practitioners may use to offset
their lack of training and experience
– Practitioner dependence on CBTI
31
Threats to
Validity and Reliability
• Appropriate Use of CBTI
– Computer serves as a consulting function
– Practitioner has final responsibility for
communicating test results
– Qualifications necessary for appropriate use
– Prescreening
– Not all clients can provide valid item
responses
– Follow ethical standards
32
Potential Problems Associated with
Computer Applications
• Overview of Topics
– Threats to Validity and Reliability
•
•
•
•
administration and scoring
test interpretation
ethical issues
ethical standards
– Ineffective Implementation
33
Ineffective Implementation
• Staff Resistance
– Lack of staff participation in decision-making
– Lack of experience with computers
– Poor quality program documentation
– time and effort required
• Poor staff support results in reduced
acceptance
34
Effective Implementation
• Improves the likelihood of success
– Careful planning
– Group input in decision making
– Reasonable rate of change
• Effective implementation involves
interpersonal dynamics more than
technology
35
The Future
• Hardware Development
– Human voice
– Video-based stimuli
– Virtual reality
• Possible Problems
– High cost
– Extraneous auditory and visual data may
distract test taker
– Stereotypes
36
The Future
• Software Development
– Expert and decision support systems
– Expert systems will be added to existing
CBTI systems
– Expert systems can be used as a powerful
training resource
37
Conclusion
• Do computer applications actually
improve testing and assessment in
counseling?
– Depends more on practitioner attitudes and
skills than future advances in technology
– Without effective use, new technology is
meaningless
38