Criterion-referenced Language Testing
Sample report
Conclusions
Criterion-referenced tests are difficult and time consuming to develop,
administer, score, analyze, and revise. Such a project involves a immense amount
of work, but a dedicated and resolute group of teachers with good leadership,
can create such tests, even in multiple forms like we did. However, the benefits
can also be enormous. For instance, the information that we gained from
administering our CRTs as diagnostic pretests and achievement posttest was
directly and immediately applicable to what we are doing on a daily basis in our
ESL classes and helped us improve all of the following components of our
curriculum: students’ needs, goals and objectives, the tests themselves, materials
development, teacher support, and program evaluation (for more on these
curriculum components, see Brown, 1995).
First, the Criterion-referenced tests helped us to examine our views of the
students’ needs. Using our CRT results, we were able to examine how the
students performed at the beginning and end of our reading courses on each
and every objective, as well as the degree to which those objectives were
defined in clear, observable, and defensible terms. Some of our initial
perceptions turned out to be completely wrong, and, as mentioned above, the
tests had to be completely overhauled. However, that outcome was better than
blithely persisting in teaching the students material that they did not need to
learn. Hence, our knowledge of the students’ needs was considerably improved
by the implementation and analysis of our CRTs.
Second, learning which goals and objectives were and were not functioning well
made it possible for us to focus on those objectives that best reflected the
students’ needs while adding new objectives designed to meet more advanced
needs. The new objectives helped us avoid wasting time and energy teaching
material the students already knew. We could instead concentrate on material
they truly needed with a great deal more efficiency. Perhaps we were falling
prey to what Tumposky (1984) cynically called “the cult of efficiency.” Frankly,
we do not see anything wrong with trying to deliver ESL instruction in a
relatively efficient way. After all, the students pay good money for our services.
In short, our objectives, newly honed by our CRT results, will simply help us to
foster as much language learning as possible during the shortest period of time.
Third, and most obvious, implementing, analyzing, and revising the Criterionreferenced helped us to improve the tests themselves. But there is much more to
it than that. The process of doing Criterion-referenced testing afforded us the
1
luxury of working together as groups of teachers to create, administer, score,
analyze, and revise classroom tests that were probably much more effective than
any single teacher could have done in isolation. In addition, the information
learned from developing Criterion-referenced tests can be used to improve other
types of tests. For example, information learned from CRT achievement test
development might be useful for revising our placement procedures in order to
overcome any mismatch that develops between placement test decisions and the
curricula of the courses into which the students are being placed (as
demonstrated in Brown, 1981). An example of how Criterion-referenced testing
practices can be used to modify placement procedures so that they are more
closely aligned with courses is explained in Brown (1989a).
Fourth, developing CRTs and changing our objectives based our diagnostic and
achievement test results naturally lead us to rethinking our materials
development. Naturally, our goal in rethinking our materials was to better
match our new perceptions of what the students needed. In some cases, the
materials have changed completely, but more often, the materials, teaching
techniques, and practice exercises have been incrementally modified. In general,
the CRT development process helped us better gauge the correct levels and
objectives for the textbooks that we ended up adopting, or for the materials
modules that we developed to fill gaps in the textbooks.
Fifth, the primary purpose of all of our curriculum development activities was
teacher support. Basically, we just wanted to free teachers of onerous tasks so
they could focus on what they do best – teaching. To that end, our
Criterion-referenced tests help teachers do rational and well-designed
achievement testing (which is required in all of UHM’s undergraduate courses).
Teachers are welcome to add their own sections to the final examination, but
the core test is essentially provided for them, by having them work together on
it. Because these CRTs are so important to the teachers and their students, the
teachers most directly involved review and revise them in groups. As such, the
CRTs serve as an important focal point for discussion, while the teachers are
working constructively together toward a common teaching / learning goal.
The sixth and final benefit gained from our CRT development involves program
evaluation. Program evaluation can be viewed as being formative or summative.
Taking the formative view, our CRTs have clearly helped us to revise and fine
tune our curriculum as it has continued to evolve. However, if called upon to
perform a summative program evaluation by some outside agency, the CRTs
would also put us in a very powerful position to defend our program because
we would have a staggering amount of information ready to furnish:
(a) norm-referenced information about the overall proficiency of our students in
terms of their TOEFL scores for admissions, (b) norm-referenced information
about their placement based on the six subtests of the ELI Placement Test,
(c) Criterion-referenced information on the students’ knowledge at the
beginning (diagnostic) and end (achievement) of each course, and
2
(d) Criterion-referenced information about what and how much the students
have learned in our courses. In short, our CRTs will clearly put us in a strong
position to write a summary report that describes our program in terms of
students’ needs, goals and objectives, materials development, teacher support,
and of course, program evaluation. More importantly, our CRTs put us in a
strong position for suggesting positive program changes as part of our on-going
curriculum development processes.
We sincerely hope that most of our students are served well by the procedures
discussed in this report. We especially hope that those students are correctly
admitted, placed, diagnosed, and promoted during their time in the ELI at UHM.
We recognize that these decisions are made by human beings and that, even
using apparently objective test scores, human beings do not always make correct
judgments. We further recognize that incorrect decisions may cost students a
great deal in terms of money (e.g., extra tuition paid) or time (e.g., extra,
unnecessary time spent in ESL classes). As a result, we take our decisions about
our students’ lives very seriously and base them on the best available
information. One important source of that information comes from our
Criterion-referenced tests.
References
[The paper references are integrated with the book references]
3