Portfolio Assessment:
A literature review
Philip Smyth
English Centre
The University of Hong Kong
Overview
Historical perspectives
 Defining the portfolio
 Purposes of portfolios
 Issues in portfolio assessment
 Research in portfolio assessment
 My own possible research avenues

Historical perspective
Portfolios widely used for many years
 Late 80s interest in portfolios for
assessment (Belanoff and Dickson
1991)
 90s saw advent of eportfolios
 A shift in emphasis away from
assessment to learning?

Definitions
“collection of student work that
demonstrates achievement or
improvement” (Stiggins 1994)
 “a portfolio is a collection of evidence
that is gathered together to show a
person’s learning journey over time
and to demonstrate their abilities” (Butler

2006)
Definitions

“…student writing over time, which
contains exhibits showing the stages
in the writing processes a text has
gone through and the stages of the
writer’s growth as a writer, and
evidence of the writer’s self-reflection
on her/his identity and progress as a
writer” (Hamp-Lyons 1996)
Definitions

portfolios are “…prepared with a
particular audience in mind”, “…are
selective” and “call for judgments” (Calfee
and Freedman 1996)
Definitions

“…a purposeful collection of student
work that illustrates efforts, progress,
and achievement in one or more
areas [over time]. The collection must
include: student participation in
selecting contents, the criteria for
selection, the criteria for judging merit,
and evidence of self-reflection” (The
Northwest Evaluation Association cited in Barret 2005)
Definitions – main
characteristics



They are collections of work, different from
a single timed impromptu essay or a class
essay carried out over a semester.
They are purposeful in that they
“demonstrate”, “exhibit” or provide
“evidence” of “achievement”, “improvement”,
“the writer’s self reflection”, “the writing
process” and “the writer’s growth”.
The degree to which these characteristics
are evidenced in portfolios largely depends
on their purpose.
Types of portfolio






a process portfolio
a showcase portfolio
an assessment
portfolio
A dossier portfolio
A reflective portfolio
A classroom portfolio






A positivist portfolio
A constructivist
portfolio
A personal portfolio
A structured portfolio
An employment
portfolio
A working portfolio
Where are portfolios used?


Primary and secondary classrooms
In tertiary settings:







Teacher education
Medicine
Nursing
Engineering
Dentistry
Psychology
ESP/EAP writing classes
How are portfolios used?
In a class
 Across more than one class
 Statewide
 Across a university curriculum

Purposes
Accountability; evaluating program or
curriculum effectiveness
 Evaluating individual student
progress; grading, certifying student
accomplishment
 Diagnosing students’ needs; informing
classroom instructional planning

Purposes
Encouraging teacher efficacy;
encouraging reflective practice at the
school and classroom levels;
supporting teachers’ professional
development
 Encouraging student efficacy;
promoting student self-assessment;
motivating student performance

(Herman, Gearhart and Acshbacher, 1996)
Issues
Different audiences (who is the
portfolio for?)
 Grading – who grades? (fairness)
 Learning and reflection get lost in
drive to measure competency
(Herman and Winter 1994)

Issues
Gap between psychometrics and
collaborative nature of the revision
process (Song and August
2002,Hamp-Lyons and Condon 2000 )
 Time needed for both teachers and
students (Callahan 1995, Herman and
Winters 1994)

Common problems

Purposes – clear to teacher and
student? (Callahan 1995)
Mismatch between assessment
criteria and goals of programme of
study
 Student anxiety and confusion (Butler
2006)

How do portfolios function best?
Research in portfolio
assessment
A slim collection?
Research in portfolio
assessment
Validity and reliability
 Fairness
 Impact

Validity

Vermont program
Correlation ranging from .47 to .58
between writing portfolio scores and
direct writing assessments
 Similar correlation between portfolio
scores and multiple-choice maths test
scores

(Koretz 1993 cited in Herman and Winter 1994)
Validity

Gearhart and others (1993) cited in Herman
and Winter (1994) found:


No relationship when comparing writing
portfolios with standard writing assessments
Two thirds of students classified as “masters”
on the portfolio assessment would not have
been so classified on the standard
assessment.
Validity

Gearhart and others (1993) cited in Herman
and Winter (1994) also found:


When portfolios were scored in two different
ways (holistic and individual pieces scored)
correlations were in the .6 range
Half the students who would have been
classified as masters on the single portfolio
score would not have been so classified
when individual pieces were averaged
Which assessment best represents an
enduring capability?
Validity

CUNY (Song and August 2002)



2 groups, 1 assessed by portfolio and writing
test, the other only a writing test
Students twice as likely to move to the next
course when evaluated by portfolio
At the end of the next course the pass rate
and grade distribution for the two groups
were nearly identical
Reliability
Vermont interrater reliability of .28 to
.60
 Pittsburgh portfolio system ranged
from .6 to .7
 Herman et al. (1993)found
correlations of .82 in an elementary
school portfolio containing final drafts
of writing

Reliability

Little work on other sources of
portfolio reliability
Score stability over time
 Stability across different rater groups
 The portfolio set in which a particular
portfolio is rated

(Herman and Winter 1994, pg. 51)
Reliability
Heller, Sheingold and Myford (1998)
think-aloud protocol on portfolio raters
to see if they fit process model of
portfolio rating
 Found score validity was threatened
when a major process was omitted or
extraneous assessment criteria were
applied

Reliability

Nystrand, Cohen and Dowling (1993)
found reliability could be significantly
improved if:
Raters scored each task in response
to a prompt before moving to the next
task and
 Raters read several examples
together to decide how they were to
be rated

Impact
Herman and Winter (1994) based on
self-reports from teachers and others
implementing portfolios appears to
have positive effects on instruction
 Vermont principals affirmed that the
portfolio assessment program had
beneficial effects on curriculum and
instruction

Impact

Aschbacher’s (1993) action research cited
in Herman and Winter 1994 suggests
teacher’s instructional practices and their
attitudes towards students changed.


Reported ways they thought about their own
teaching
Two-thirds of teachers expected higher level
of performance from students
Impact

Hirvela and Sweetland (2005) used 2 case
studies showing the 2 students did not
strongly endorse the portfolios as used in 2
different courses.


Seemed to need more explanations of what
portfolio approaches were meant to achieve
Even with a 5% final course grade students
saw the portfolio as essentially summative in
nature
Impact

Richardson (2000) study involved
classroom observations teacher and
student interviews and examination of
student writing and teacher response.

Found that students regard teacher
responses as directives. Were not
prepared to make independent
judgments largely because of the
threat of grades
The future?
More technical quality
 Issue of fairness needs to be
addressed
 More research on impact
 A move away from psychometric
measurement?

Potential research
Equity of portfolio assessment
 Validity v reliability
 Nature of feedback
 Portfolios for employers

References





Abrami, P. C., & Barrett, H. (2005). Directions for Research and
Development on Electronic Portfolios. Canadian Journal of
Learning and Technology, 31(3).
Belanoff, P., & Dickson, M. (Eds.). (1991). Portfolios : process
and product. Portsmouth, N.H.: Boynton/Cook Publishers.
Calfee, R. C., & Freedman, S. W. (1996). Classroom Writing
portfolios:Old, New, Borrowed, Blue. In R. C. Calfee & P.
Perfumo (Eds.), Writing Portfolios in the Classroom. MahWah,
N. J.: L. Erlbaum Associates.
Callahan, S. (1995). Portfolio expectations: Possibilities and
limits. Assessing writing, 2(2), 117-151.
Case, S. H. (1994). Will mandating portfolios undermine their
value? Educational Leadership, 52(2), 46-47.
References




Desmet, C., & Cummings, R. (2004). Negotiating the TeachingAssessment Cycle in Writing Programs with XML. Paper
presented at the World Conference on E-Learning in Corporate,
Government, Healthcare, and Higher Education 2004
Washington DC.
Frederiksen, J. R., Sipusic, M., Sherin, M., & Wolfe, E. W.
(1998). Video Portfolio Assessment: Creating a Framework for
Viewing the Functions of Teaching. Educational Assessment,
5(4), 225-297.
Hamilton, S. J. (2006). A Principle-Based ePort Goes Public
(and Almost Loses its Principles). In A. Jafari & C. Kaufman
(Eds.), Handbook of Research on ePortfolios (pp. 434-446).
Hershey: Idea Group Reference.
Hamp-Lyons, L. (1990). Second language writing: assessment
issues. In B. Kroll (Ed.), Second language writing : research
insights for the classroom. Cambridge: Cambridge University
Press.
References





Hamp-Lyons, L., & Condon, W. (2000). Assessing the Portfolio:
Principles for Practice Theory and Research. Cresskill, NJ:
Hampton Press.
Heller, J. I., Sheingold, K., & Myford, C. M. (1998). Reasoning
about Evidence in Portfolios: Cognitive Foundations for Valid
and Reliable Assessment. Educational Assessment, 5(1), 5-40.
Herman, J. L., Gearhart, M., & Aschbacher, R. (1996). Writing
portfolios in the classroom : policy and practice, promise and
peril. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in
the classroom : policy and practice, promise and peril
(pp. x, 374 p.). Mahwah, N.J.: L. Erlbaum Associates.
Herman, J. L., Gearhart, M., & Baker, E. L. (1993). Assessing
writing portfolios: Issues in the validity and meaning of scores.
Educational Assessment, 1(3), 201-224.
References





Herman, J. L., & Winters, L. (1994). Portfolio research: A slim
collection. Educational Leadership, 52(2), 48-55.
Hirvela, A., & Sweetland, Y. L. (2005). Two case studies of L2
writers' experiences across learning-directed portfolio contexts.
Assessing writing, 10, 192-213.
Holt, D., & Baker, N. W. (1991). Portfolios as a follow-up option
in a proficiency-testing program. In P. Belanoff & M. Dickson
(Eds.), Portfolios: process and product. Portsmouth NH:
Boynton/Cook.
Jafari, A., & Kaufman, C. (Eds.). (2006). Handbook of Research
on ePortfolios. Hershey: Idea Group Reference.
Murphy, S., & Camp, R. (1996). Moving towards systemic
coherence: A discussion of conflicting perspectives on portfolio
assessment. In R. C. Calfee & P. Perfumo (Eds.), Writing
portfolios in the classroom: policy and practice, promise and
peril. Mahwah, N.J.: L. Erlbaum Associates.
References





Nystrand, M., Cohen, A. S., & Dowling, N. M. (1993).
Addressing reliability problems in the portfolio assessment of
college writing. Educational Assessment, 1(1), 53-70.
O'Brien, K. (2006). ePortfolios as Learning Construction Zones.
In A. Jafari & C. Kaufman (Eds.), Handbook of research on
ePortfolios (pp. 74-82). Hershey: Idea Group Reference.
Ostheimer, M. W., & White, E. M. (2005). Portfolio assessment
in an American college. Assessing writing, 10, 61-73.
Pullman, G. (2002). Electronic Portfolios Revisited: The efolios
Project. Computers and Composition, 19, 151-169.
Richardson, S. (2000). Students' conditioned response to
teachers' response: Portfolio proponents, take note! Assessing
Writing, 7, 117-141.
References





Song, B., & August, B. (2002). Using Portfolios to Assess the
Writing of ESL Students: A Powerful Alternative? Journal of
Second Language Writing, 11, 49-72.
Spalding, E., & Cummins, G. (1998). It was the best of times. It
was a waste of time: University of Kentucky students' view of
writing under KERA. Assessing Writing, 5(2), 167-199.
Stiggins, R. J. (1994). Student-centred classroom assessment.
New York: Merrill.
Wagner, M., & Lamoureaux, E. (2006). Implementing an
Outcome-Based Assessment ePortfolio. In A. Jafari & C.
Kaufman (Eds.), Handbook of Research on ePortfolios (pp.
539-550). Hershey: Idea Group Reference.
Weigle, S. C. (2002). Assessing writing. Cambridge, UK
Cambridge University Press.
References


Yancey, K. B. (1996). Portfolio as genre, rhetoric as reflection:
Situating selves, literacies, and knowledge. WPA 19(3), 55-69.
Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S.,
& Lamson, S. (2008). Validity evidence of an electronic portfolio
for preservice teachers. Educational Measurement: Issues and
Practice, Spring 2008, 10-24.