Outline
1.
2.
Ethics vs. Law
Ethical use of tests – issues:






Competence
Informed Consent
Knowledge of Results
Confidentiality
Test security
Divided Loyalties
1
Outline
4.
Professional Issues



5.
Theoretical issues
Test adequacy
Actuarial vs. clinical prediction
Social Issues


Dehumanization
Access to benefits of psychological testing
2
Outline

6. Two examples of bad tests



Handwriting analysis
TOEFL
7. The CPA Code of Ethics (synopsis)




Respect for the dignity of persons
Responsible caring
Integrity in relationships
Responsibility to society
3
4
Ethics vs. Law

Ethics: what one
should or should not
do, according to
principles or norms of
conduct


Ethics codes are not
produced by
democratically-elected
legislatures
Enforcement
mechanisms are
usually informal, may
be complex, even
unconscious
5
Ethics vs. Law

Law: what one must or
must not do, according
to legal dictates


Laws are created by
democratically-elected
legislatures in
democracies such as
Canada
Laws come with
explicit penalties for
infractions and a
formal enforcement
system
Ethical use of tests – issues:
1.
2.
3.
4.
5.
6.
Competence
Informed Consent
Knowledge of Results
Confidentiality
Test security
Divided Loyalties
6
7
1. Competence


develop competence in
assessment concepts
and methods
recognize boundaries
of your competence

competence is very
important if test result
affects someone’s life
2. Informed Consent

Generally, the
client/subject must
voluntarily consent

psychologist must
inform client about
nature and purpose of
assessment in
understandable
language
8
Informed consent - exceptions

Sometimes, it is
acceptable to test
without getting consent
– e.g., you have given
implied consent to be
tested by registering in
this course, but I have
never sought your
explicit consent

Sometimes it is
necessary to test without
getting consent, even
when consent is
explicitly refused by
person to be tested –
e.g., when mandated
by law
9
3. Knowledge of Results

must fully disclose test
results in
understandable
language

but do not do so in a
way that reveals the
content of test items,
because that content is
proprietary
information (e.g.,
somebody owns it – it’s
not yours to reveal)
10
Constraints on client access to results

PIPEDA Principal 4.9:
 Upon request, a
person must be
informed of the
existence, use, and
disclosure of his or
her personal
information and must
be given access if
requested.

Section 9.3b says that
access can be denied
if it would mean
revealing confidential
commercial information
(unless latter is
severable).
11
Constraints on client access to results

Ontario Personal Health
Information Act: access
rights do not apply to a
record that contains raw
data from standardized
psychological tests or
assessments, unless those
data are reasonably
severable.

CPA and leading Canadian
test publishers say that Test
Materials are not “personal
information” so should not
be released.
12
13
4. Confidentiality

Test results are
confidential
information

Release of results
should only be made
to another qualified
professional after
client’s consent
14
5. Test Security

Test materials must be
kept secure


Test items are not
revealed except in
training programs and
when mandated by
law, to protect test
integrity
Test items are private
property
6. Automated Scoring/Interpretation Systems

Psychologist is still
responsible for proper
interpretation of test
results

Psychologists are
professionals – they
take personal
responsibility for their
work
15
16
7. Divided loyalties


Who is the client?
The person being
tested, or the institution
you work for?


What if these parties
have conflicting
interests?
How do you maintain
test security but also
explain an adverse
decision?
17
Divided loyalties


Suppose you are
asked to test a child
who might have
disorder X or Y
Suppose X is very
common and so the
government provides
resources to help
schools deal with it


Suppose Y is not
common and no
resources are
available in school to
deal with it
If your testing suggests
Y, will you report Y –
or X, so the child gets
some sort of help?
Professional Issues
1.
2.
3.
Theoretical issues
Test adequacy
Actuarial vs. clinical prediction
18
19
1. Theoretical Issues

Are you measuring a stable
characteristic of the person
being tested?

If so, differences in scores
over time reflect
measurement error or
subject variables such as
fatigue
20
1. Theoretical Issues

Are you measuring a stable
characteristic of the person
being tested?


In fact, we have little
evidence that this is true, so
what is the value of your
test result – will it still be
true next year?
Especially problematic for
personality tests
21
1. Theoretical issues

Are you measuring a stable
characteristic of the person
being tested?


Suppose you test someone
as part of a hiring process.
If the test result says that
person does not have the
characteristic you’re looking
for, does that mean they
could never acquire that
characteristic?
2. Adequacy of tests


How do we measure
test adequacy?
(We’re supposed to be
good at measuring
things!)

What should go into
an assessment of test
adequacy?
22
2. Adequacy of tests

So far, in considering
the relative merits of
various tests, we have
asked whether they
are psychometrically
adequate



We haven’t asked: is
the best test available
good enough?
Society uses that
standard when testing
becomes a legal issue
But the answer is not
psychometric
23
3. Actuarial vs. clinical judgment

U.S. Supreme Court
Justice Potter Stewart
said in a 1964 case
that he didn’t know
how to define
pornography, but he
knew it when he saw it.

Is it possible for us to
make good judgments
on a question when we
cannot articulate the
basis for the
judgment?
24
3. Actuarial vs. clinical judgment

Actuarial judgment
occurs when we feed
test scores into
statistical formulas to
diagnose a
psychological condition
or predict future
performance.

Clinical judgment
occurs when we have a
trained psychologist
interpret test scores to
diagnose a
psychological condition
or predict future
performance.
25
3. Actuarial vs. clinical judgment


In actuarial judgment,
we cannot make
accurate predictions
tailored to individuals
Instead, our conclusion
will be the same for
every person with a
given set of test scores


In clinical judgment,
the claim is that you
can determine “what
caused what” in an
individual’s person’s
life (Dawes, 1994)
But clinical judgment
does not improve with
experience
26
3. Actuarial vs. clinical prediction

Meehl’s work in the
1950s: feed test
scores into statistical
formula to predict
future performance –
or have trained
psychologist interpret
scores


statistical formula does
a better job of
prediction
still, someone has to be
legally responsible –
that has to be a
trained professional
27
Social issues
1.
2.
Dehumanization
Access to benefits of psychological testing
28
29
1. Dehumanization

Does computerized
testing and analysis of
test results create a
danger of minimizing
human uniqueness?


humans are very
complex – which
allows us to be
individuals, different
from each other
but testing and
interpretation
generalize
30
1. Dehumanization


Is generalization a
problem?
Computer-scored and
administered
psychological tests
might determine
important aspects of
your life


If you think this is a
problem, why do you
think so? What
evidence do you have
that this hurts people?
What alternative
approach would you
use?
2. Access to psychological testing

We’ve considered
possibility of being
hurt by tests – but
what about possible
benefits of tests? Who
gets those benefits?

WAIS-III kit costs $775
per person – for tester
to buy the kit. Cost
must be passed on to
someone. Who should
that be?
31
2. Access to psychological testing


Psychological services
are not covered by
OHIP; Psychiatric
services are
Psychiatrists have
opposed extending
OHIP coverage to
psychological testing


Is that just turf
protection?
There is rivalry
between the
professions – but we
could make a stronger
case for billing OHIP if
we used only actuarial
judgment
32
Two examples of bad tests
1.
2.
Graphology (handwriting analysis)
TOEFL
33
Graphology (handwriting analysis)


Used in hiring decisions
in business
Used to make
assessments of
personality type and
to predict on-the-job
performance


Simner & Goffin
(2003): very little
evidence of validity
for graphology
studies using nonautobiographical
writing are the most
useful and show the
least evidence of
validity.
34
Graphology (handwriting analysis)

Dean (1992) meta-analysis
of handwriting studies
showed an average
correlation of .16 between
graphological judgment
and job performance, using
autobiographical writing.

Correlation drops to .09
when using nonautobiographical writing.
35
Graphology (handwriting analysis)

Simner & Goffin: both
general mental ability
testing and structured
employment interviews
correlate around .50
with job performance.
When used together,
the two correlate .63
with job performance.

Thus, GMA testing and
interviewing are much
more likely to lead to
sensible hiring
decisions.
36
Test of English as a Foreign Language

Widely used to select
among students who
are not native
speakers of English for
admission to Canadian
universities.


TOEFL manual presents
a distorted picture of
validity studies
Simner (1998)
presents CPA’s
recommendation –
don’t use TOEFL for
university admission
decisions
37
TOEFL – Criterion Validity

TOEFL manual leaves
out studies that show
low to moderate
correlations between
TOEFL scores and
teachers’ ratings of
English language
proficiency (.34 to .55)

TOEFL manual leaves
out studies that show
low to moderate
correlations between
TOEFL scores and oral
interviews (.30 to .60)
38
TOEFL – Criterion validity

Generally, these
studies assess
concurrent validity, not
more relevant
predictive validity

Simner: some criterion
validity tests for TOEFL
involve the Michigan
TELP, very similar to
TOEFL – so this is
really parallel forms
reliability, not validity
evidence
39
TOEFL – Construct validity


ETS: Native speakers
of English do much
better than non-natives
on TOEFL.
Simner: in these
studies, native
speakers are a more
highly-selected group.



Non-natives were
unselected examinees
from a range of
backgrounds.
Natives were selected
from among entering
freshmen
Test anxiety
differences
40
TOEFL – Construct validity

Unpublished ETS study:
5000 English speakers
had an average score
of 590, below the 600
cutoff used for nonnatives at many
Canadian universities.

If native English
speakers can succeed
with scores < 600 why
can’t non-natives also
succeed with scores
below 600?
41
Test of English as a Foreign Language

Simner: Since 1995, 10
Ontario universities have
raised their TOEFL cutoff
from around 550 to 600.
This means that only the top
10% of examinees are
accepted, instead of top
30%.


Taylor-Russell tables: r =
.20, now 10% accepted,
before 30%
Net gain in percentage
likely to be successful is
4%, at a cost of rejecting
an additional 20%.
42
TOEFL: Simner’s recommendations


make admissions
decisions on the basis
of all relevant
information, not just
TOEFL scores
do not use rigid cutoff
scores


consider the kinds and
levels of English
proficiency required in
different fields and
levels of study
consider resources
available for improving
English language skills
43
CPA Code of Ethics
The following slides show the four basic principles
advanced in the CPA Code of Ethics. All of the material
on the next 15 slides is quoted directly from that Code.

There is significantly more information on each
principle at this web address:
http://www.cpa.ca/cpasite/userfiles/Documents/Cana
dian%20Code%20of%20Ethics%20for%20Psycho.pdf


(Note: scroll down if the page looks blank)
44
CPA Principle I: Respect for the Dignity of Persons.

In the course of their work as scientists, practitioners,
or scientist-practitioners, psychologists come into
contact with many different individuals and groups,
including: research participants; clients seeking help
with individual, family, organizational, industrial, or
community issues; students; trainees; supervisees;
employees; business partners; business competitors;
colleagues; employers; third party payers; and, the
general public…
45
CPA Principle I: Respect for the Dignity of Persons.
In these contacts, psychologists accept as
fundamental the principle of respect for the dignity
of persons; that is, the belief that each person should
be treated primarily as a person or an end in
him/herself, not as an object or a means to an end…

46
CPA Principle II: Responsible Caring.
A basic ethical expectation of any discipline is that its
activities will benefit members of society or, at least, do
no harm. Therefore, psychologists demonstrate an active
concern for the welfare of any individual, family, group,
or community with whom they relate in their role as
psychologists. This concern includes both those directly
involved and those indirectly involved in their activities.

47
CPA Principle II: Responsible Caring.
However, as with Principle I, psychologists’ greatest
responsibility is to protect the welfare of those in the
most vulnerable position. Normally, persons directly
involved in their activities (e.g., research participants,
clients, students) are in such a position. Psychologists’
responsibility to those indirectly involved (e.g.,
employers, third party payers, the general public)
normally is secondary.

48
CPA Principle III: Integrity in Relationships.
Psychologists are expected to demonstrate the highest
integrity in all of their relationships. However, in rare
circumstances, values such as openness and
straightforwardness might need to be subordinated to
the values contained in the Principles of Respect for the
Dignity of Persons and Responsible Caring.

49
CPA Principle III: Integrity in Relationships.
The relationships formed by psychologists in the course of their
work embody explicit and implicit mutual expectations of
integrity that are vital to the advancement of scientific knowledge
and to the maintenance of public confidence in the discipline of
psychology. These expectations include: accuracy and honesty;
straightforwardness and openness; the maximization of
objectivity and minimization of bias; and, avoidance of conflicts
of interest. Psychologists have a responsibility to meet these
expectations and to encourage reciprocity.

50
CPA Principle III: Integrity in Relationships.
In addition to accuracy, honesty, and the obvious
prohibitions of fraud or misrepresentation, meeting
expectations of integrity is enhanced by self-knowledge
and the use of critical analysis.

51
CPA Principle IV: Responsibility to Society.
Psychology functions as a discipline within the context
of human society. Psychologists, both in their work and as
private citizens, have responsibilities to the societies in
which they live and work, such as the neighborhood or
city, and to the welfare of all human beings in those
societies.

52
CPA Principle IV: Responsibility to Society.
Two of the legitimate expectations of psychology as a
science and a profession are that it will increase
knowledge and that it will conduct its affairs in such
ways that it will promote the welfare of all human
beings.

53
CPA Principle IV: Responsibility to Society.
Freedom of enquiry and debate (including scientific
and academic freedom) is a foundation of psychological
education, science, and practice. In the context of society,
the above expectations imply that psychologists will
exercise this freedom through the use of activities and
methods that are consistent with ethical requirements.

54
CPA Principle IV: Responsibility to Society.
The above expectations also imply that psychologists
will do whatever they can to ensure that psychological
knowledge, when used in the development of social
structures and policies, will be used for beneficial
purposes, and that the discipline’s own structures and
policies will support those beneficial purposes. Within
the context of this document, social structures and
policies that have beneficial purposes are defined as
those that more readily support and reflect respect for
the dignity of persons, responsible caring, integrity in
relationships, and responsibility to society.

55
CPA Principle IV: Responsibility to Society.

If psychological knowledge or structures are used against
these purposes, psychologists have an ethical responsibility to
try to draw attention to and correct the misuse. Although this is
a collective responsibility, those psychologists having direct
involvement in the structures of the discipline, in social
development, or in the theoretical or research data base that
is being used (e.g., through research, expert testimony, or
policy advice) have the greatest responsibility to act. Other
psychologists must decide for themselves the most appropriate
and beneficial use of their time and talents to help meet this
collective responsibility.
56
CPA Principle IV: Responsibility to Society.
When a person’s welfare appears to conflict with
benefits to society, it is often possible to find ways of
working for the benefit of society that do not violate
respect and responsible caring for the person. However,
if this is not possible, the dignity and well-being of a
person should not be sacrificed to a vision of the greater
good of society, and greater weight must be given to
respect and responsible caring for the person.

57
CPA’s Caveat…
Even with the above ordering of the principles,
psychologists will be faced with ethical dilemmas that
are difficult to resolve. In these circumstances,
psychologists are expected to engage in an ethical
decision-making process that is explicit enough to
bear public scrutiny. In some cases, resolution might be
a matter of personal conscience.

58
CPA’s Caveat…
However, decisions of personal conscience are also
expected to be the result of a decision-making
process that is based on a reasonably coherent set of
ethical principles and that can bear public scrutiny. If
the psychologist can demonstrate that every
reasonable effort was made to apply the ethical
principles of this Code and resolution of the conflict
has had to depend on the personal conscience of the
psychologist, such a psychologist would be deemed
to have followed this Code.

59