FLL ASSESSMENT PLAN 2001-2003
5-20-03
Introduction.
The Department of Foreign Languages and Literatures (FLL) has chosen to focus its initial study
for 2001-02 on 300-level courses, because we believe that facilitating student progress in thirdyear is key to a successful and coherent program. FLL concluded that it is essential to focus first
on assessing student readiness, and teacher and student expectations with respect to learning
objectives at the third-year level. Because FLL offers third-year courses in multiple (10)
languages, and because Spanish instructors are currently involved in the implementation of a
PEW grant for redesign of first year Spanish, we chose to pilot our assessment program with the
two other large third-year programs offered by French and German.
In the second year of implementation we plan to expand our focus to include instructors
and students taking courses at the fourth-year level in French and German and to gather
information on student satisfaction and participation in both literature and language courses.
THIRD YEAR ASSESSMENT: 2001-2002
The third-year foreign language courses have traditionally served a broader audience of students
than upper division fourth-year courses. Third-year is the bridge for students progressing on to
a major or concentration in a foreign language. It is also often the final year of study for students
with an interest in a language and culture but who are majoring in another subject. Completion
of third-year (or third-year proficiency) is also required for students with a major in International
Studies. The diversity of this student population and their varying goals and levels of
preparation is perceived as a source of difficulty for creating a successful learning environment.
FLL recognizes that it needs to gather information that will allow each language to assess how its
third-year courses contribute to the learning objectives of the FLL major and proficiency in a
foreign language. We will identify the ways that we can most efficiently and consistently assess
entering students’ preparation and progress.
Background.
Five years ago conversion to a four-credit standard required the Department of Foreign
Languages and Literatures to review its third and fourth language and literature offerings. At the
same time we hoped to improve support for proficiency based learning objectives for the major
and to address the interests of students who wish to continue their study of a language and
culture outside of a major. In some cases, review confirmed the logic of existing sequences.
Limited staffing and the number of years required to reach more advanced proficiency were
barriers to introducing greater variety of courses for some of the languages. Some courses taught
at the third-year level had to be dropped for budgetary reasons. Other courses had to be offered
less frequently. The Department added more course offerings in English related to the target
cultures. Three of the language majors, French, German and Spanish introduced structural
and/or thematic changes. The purpose of the changes were:
1. to introduce greater variety in the third-year curriculum, allowing some alternatives to
a traditional grammar-centered approach, or literature-based study
2. to provide greater scheduling flexibility for students, allowing them to take third-year
level courses out of the traditional 301-302-303 or 341-342-343 sequences
3. to accommodate changes (decrease) in staffing
Changes were implemented without a systematic effort to gather information for assessing
whether enrollment patterns or student success or satisfaction changed, or whether there was an
impact on the major. Program objectives and offerings may therefore need some realignment.
In addition to our own programmatic changes, there have since been changes in the standards
and expectations for proficiency in a foreign language in Oregon’s high schools (CIM and CAM)
that suggest that PSU will be admitting students with different, and potentially higher levels of
proficiency than in the past. Traditional placement practices (essentially self-placement) may
need adjustment.
Learning Objectives and the FLL Assessment Focus.
In 2000-2001 the FLL Department tentatively approved a set of learning objectives for its majors
that were developed over 1999-2000. These objectives include three major areas focusing on the
development of proficient communication skills and strategies in the written and spoken
language, intellectual skills for critical inquiry and making cultural connections in and through
the target language. These objectives are aligned with national standards approved by the
American Council for the Teaching of Foreign Language (ACTFL).
FLL has the following objectives for its majors relating to developing communication strategies:
CS1 => to develop the strategies and skills for oral communication in the target language
CS2 => to develop the strategies and skills for written communication in the target language
CS3 => to comprehend spoken forms of the target language
CS4 => to read and understand written forms and texts in the target language
CS5 => to learn to recognize and respond to culturally distinct patterns of interaction within the
target language
CS6 => to develop language skills for academic and ‘real world’ contexts
A survey of departmental course offerings completed by instructional faculty for their languages
revealed that the third-year level courses share the emphasis of first and second year language
courses on developing skills for oral and written communication in the target language (CS1 and
CS2). In addition, third-year courses, by and large, appear to place a significant secondary
emphasis on acquiring the skills to read and understand written forms and texts in the target
language (CS4). Our study will be designed to focus on assessing these three learning
objectives.
Continued progress towards proficiency in communication skills is key to successful
participation in upper division courses on literature and culture taught in the target language for
majors. It is also a crucial goal for students who do not major in the language but want to be able
to use the language for personal or career-related reasons. We need to confirm what proportion
of these groups constitute our student population in third-year and consider how well we are able
to balance their interests in the context of FLL requirements for a major. We will begin our
assessment by targeting students in the basic language sequences for third-year in French and
German. The FLL Department has stated in the PSU catalogue that enhancing spoken, written
and reading language proficiency is one of the principle objectives of the 301, 302, 303 series.
Goals for Assessment:
FLL wishes to determine if students who are enrolling for third-year course work at Portland
State are where they need to be both in terms of their level of proficiency and expectations for
the course. In the initial stage of this assessment, beginning in the fall of 2001 and continuing
through the winter and spring of 2002 the FLL Assessment committee will implement a study
focused on the following questions:
Question 1. What is the presumed level of proficiency, knowledge, and preparation of entering
third-year students with respect to oral and written communication and reading skills as
understood by the instructors, FLL department and the students themselves?
Question 2. What can quantitative assessment measures tell us about the actual level of spoken,
written and reading proficiency of students entering our third-year courses?
Focus on these two key questions can provide information on a number of related topics of
concern to FLL:
1) Do we have adequate information on the background and preparation of our students? How
many completed their initial two (or more) years of instruction at Portland State, how many
elsewhere? Do we have any articulation issues between our own second and third-year
programs? What articulation issues might be looming with respect to the CIM and CAM
standards? How do the anticipated skill or proficiency levels of entering students correspond
to the learning objectives for the course? How are these objectives communicated to the
students? Do instructors agree on what the appropriate learning objectives are?
2) Is there a match between student preparation and course expectations? Is a mismatch
perceived as an obstacle to learning by the students or the instructors? Do we need a placement
test to ensure that students new to PSU are correctly placed? Is there a correlation between the
CAPE scores or proficiency levels of those completing a PSU 203 language course and those
entering 301 or 302? Is there a correlation between measures of oral, written, and/or reading
skills and student success in the first (and final) term of third-year as measured by grades?
Strategies for implementing the assessment.
The FLL Assessment Committee is charged with designing and implementing a pilot assessment
plan for 2001-02. Its members include representatives from Asian and European languages
groups and all members of FLL are invited to participate. We will be able to build on research
and testing or first and second year students that was begun with a Faculty Vitality Grant in the
Spring of 2001. This grant allowed FLL to purchase the right to use the BYU on-line language
test Web CAPE. We can also use or adapt national proficiency guidelines and rating scales that
have been developed by ACTFL for assessing speaking and reading. Members of the
Assessment Committee and FLL have been trained in their application.
Instruments for Question 1.
1) To gather the information about student background and interests needed, we will have
students complete an on-line survey with questions about their previous study of the language,
learning objectives and transition to study at PSU. (This survey was developed for the Vitality
Grant.) The FLL Assessment Committee will design a second survey to capture student
expectations and experiences of third-year.
2) The FLL Assessment Committee will design and conduct an interview for third-year French
and German instructors, and potentially FLL department program coordinators, to identify
expectations and target proficiency levels for third-year. To facilitate analysis and comparison,
we will use the guidelines for assessing proficiency developed by ACTFL to develop a
functional skills checklist that teachers (and possibly students) can respond to.
Instruments for Question 2. At least two proficiency level checks will be implemented during
the fall and winter terms of 2001-2002.
1) Third-year German and French students will take an on-line computer assisted test developed
by Brigham Young University (Web CAPE). The results of this test can be compared to
results from students who completed second year French and German in the spring of 2002.
It may also be possible to correlate results with ACTFL proficiency levels. (This will have to
be determined.) It is believed that the Web CAPE test has some correlation with the
students’ skills in reading and written communication. Scores will be
compared to student grades at the end of the term. To assess whether or not a correlation
exists, the instructors may be asked to provide written work samples that can be rated on the
ACTFL scale.
2) A second level check will aim to capture the range of oral proficiency levels of third-year
German and French students with oral interviews of randomly selected students.
Additional data: We will also collect and analyze syllabi for each third-year course and discuss
the construction of a database that will allow FLL to track its continuing students.
Implementation Process.
During November of 2001, with the help Douglas McCartney, lab supervisor for IRN, the FLL
administered the WEB CAPE assessment to third-year French, German and Spanish students.
Students also completed an on-line FLL questionnaire about student backgrounds and
preparation. Instructors were also asked to provide a level check based on classroom
observation. Student grades for the course will be collected at the end of the term.
During the Winter Term (2002) the Assessment Committee will begin analysis of the data
gathered in the fall and develop and test a template for conducting the interviews of third-year
instructors. The committee will meet with third-year instructors at the beginning of the term to
present and discuss the goals of the assessment plan. We will explore the possibility of inserting
a question into the mid term exams of third-year French and German students during the term
that could be rated (using ACTL guidelines) and used as another cross check against the WEB
CAPE scores.
The data from Web CAPE, course grades and instructors collected during the fall term will be
compiled and analyzed with the help of individuals who are also assisting FLL complete work on
its Faculty Vitality Grant funded research using WEB CAPE data. Individual members of the
committee will analyze selected questions from the questionnaire and report on their findings.
This information should help us determine what further questions we need to ask students in
third-year about their expectations for the course, if needed.
Since the WEB CAPE test assesses primarily the students’ reading abilities along with
knowledge of the structure and vocabulary of the target language, the committee will also need
to gather information on the level of oral proficiency. Department members with experience in
administering oral proficiency tests (using ACTFL guidelines) will interview 5 to 10% of the
students in third-year French and German. The students will be selected blindly along the range
of their WEB CAPE scores. We should be able to determine whether there is any
correspondence between student performance on the on-line written test and oral communication
skills, as well as gage the range of our students’ abilities in both domains.
After analysis of the data, the Committee will determine if testing students again at the end of the
third-year with WEB CAPE, or other instrument, is warranted, or if additional testing will take
place at the beginning of fourth year.
We will be relying on assistance from the University Assessment Network to help us design and
test an interview instrument that will allow us to gather both quantitative and qualitative
information from the instructors teaching third-year courses. One feature we would like to
include is a functional skills inventory pegged to ACTFL guidelines and/or Benchmarks
developed for Oregon CIM and CAM to allow the instructors to identify more specifically what
their objectives and expectations are for third-year.
During Spring Term (2002) a “neutral interviewer” will conduct the interviews. Members of the
Assessment Committee will code and analyze the results. If time allows, we may also interview
selected other members of the FLL department, including administrators, to determine what the
prevailing global expectations of third-year are in the Department.
Outcomes.
In the latter part of the Spring term, the FLL Assessment Committee will prepare a report based
on its findings to the FLL Department Executive Committee and then for FLL as a whole for
discussion. The report will summarize what the present levels and expectations are for third-year
students and from third-year students in French and German. It will also consider whether the
information gathered suggests that there are changes that could be made in the curriculum or
admission standards and practices that might better address the needs of the students who are
currently registering for third-year French and German.
The Committee’s report should also evaluate whether the instruments it used provided data that
were meaningful in terms of the assessment goals that it defined. The Department will need to
determine whether the Committee’s efforts have produced a model for assessment that can be
adopted on a department-wide basis.
Specific Outcomes
Information gathered about entering third-year French and German students will
1) help German and French instructors determine if current pre-requisites for third-year give
students sufficient guidelines to allow them to self-place and the extent to which learning
objectives are in sync with student preparation and expectations
2) help FLL determine if Web CAPE can be used as a placement mechanism
3) guide the Assessment Committee in preparation for the second stage of third-year
assessment, i.e. measuring learning outcomes for third-year students
4) help us determine if we provide the necessary opportunities for our majors to meet learning
objectives and prepare for advanced study
5) also help us determine if we are successful in meeting the needs of continuing students who
are not majors
SECOND STAGE OF ASSESSMENT: 2002-03
We hope that the information that we gather in this initial stage of assessment will allow French
and German to establish baselines to evaluate the progress of third-year students toward
departmental learning objectives, and to weigh what kind of curricular changes they might wish
to implement to improve student learning. We foresee that these concerns will probably require
the interviewing of instructors and testing of students at the fourth year level. This follow up in
2002-03 will allow us to assess whether there is sufficient continuity between third-year level
courses and advanced courses, or if we currently have assessment measures in place to track the
progress of our majors towards FLL learning objectives.
Prospects for dissemination of results.
Faculty members participating in the work of the Assessment Committee will have the
opportunity to share some of the initial results of our research at COFLT and ACTFL (in Salt
Lake City, 2002). COFLT (Council of Oregon Foreign Language Teachers) is a particularly
important venue because it will allow us to exchange information with K-12 instructors who are
involved in preparing students to meet the Oregon benchmarks in foreign languages for
admission to the University.