LA ACADEMY COHORT COURSE ASSESSMENT: LH 111 Fall 2009 Date: 1-15-10 Department: Foreign Languages and Literatures Course: LH 111 Curriculum or Curricula: LA PART I. STUDENT LEARNING OBJECTIVES For Part I, attach the summary report (Tables 1-4) from the QCC Course Objectives Form. TABLE 1. EDUCATIONAL CONTEXT LH-111 is the first part of the introductory sequence of foreign language study (a requirement for a successful transfer to the junior year of a baccalaureate program). It is a foundation course required for (AA) degree in Liberal Arts and Sciences, Fine Arts, and Business Transfer. TABLE 2. CURRICULAR OBJECTIVES Note: Include in this table curriculum-specific objectives that meet Educational Goals 1 and 2: Curricular objectives addressed by this course: N/A TABLE 3. GENERAL EDUCATION OBJECTIVES Gen Ed objective’s ID number from list (1-10) General educational objectives addressed by this course: Select from preceding list. (1) Communicate effectively through reading, writing, listening and speaking. (2) Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. TABLE 4: COURSE OBJECTIVES AND STUDENT LEARNING OUTCOMES Course objectives Learning outcomes 1. Provide basic information in Students will be able to talk about themselves by providing their name, Hebrew about yourself, your city, address, phone number, school schedule. They will also be able to describe your classmates, your family, their family or hobbies by answering questions in prompted dialogues. friends, hobbies, and daily activities. 2. Demonstrate ability to accomplish simple communicative tasks on every day topics such as greeting people or introduce yourself to others, or describing your life in school or at work, or during your spare time. Students will be able to greet people and/or introduce themselves, or exchange personal information by participating in communicative tasks where they will play the role of a new student at QCC or a party guest trying to make friends. 2 PART II. ASSIGNMENT DESIGN: ALIGNING OUTCOMES, ACTIVITIES, AND ASSESSMENT TOOLS For the assessment project, you will be designing one course assignment, which will address at least one general educational objective, one curricular objective (if applicable), and one or more of the course objectives. Please identify these in the following table: TABLE 5: OBJECTIVES ADDRESSED IN ASSESSMENT ASSIGNMENT Course Objective(s) selected for assessment: (select from Table 4) 1. Provide basic information in Hebrew about yourself, your city, your classmates, your family, friends, hobbies, and daily activities. 2. Demonstrate ability to accomplish simple communicative tasks on every day topics such as greeting people or introducing yourself to others, or describing your life in school or at work and in your spare time. Curricular Objective(s) selected for assessment: (select from Table 2) N/A General Education Objective(s) addressed in this assessment: (select from Table 3) 1. Communicate effectively through reading, writing, listening and speaking. 2. Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. In the first row of Table 6 that follows, describe the assignment that has been selected/designed for this project. In writing the description, keep in mind the course objective(s), curricular objective(s) and the general education objective(s) identified above, The assignment should be conceived as an instructional unit to be completed in one class session (such as a lab) or over several class sessions. Since any one assignment is actually a complex activity, it is likely to require that students demonstrate several types of knowledge and/or thinking processes. Also in Table 6, please a) identify the three to four most important student learning outcomes (1-4) you expect from this assignment b) describe the types of activities (a – d) students will be involved with for the assignment, and c) list the type(s) of assessment tool(s) (A-D) you plan to use to evaluate each of the student outcomes. (Classroom assessment tools may include paper and pencil tests, performance assessments, oral questions, portfolios, and other options.) Note: Copies of the actual assignments (written as they will be presented to the students) should be gathered in an Assessment Portfolio for this course. 3 TABLE 6: ASSIGNMENT, OUTCOMES, ACTIVITIES, AND ASSESSMENT TOOLS Briefly describe the assignment that will be assessed: This project will assess students’ speaking abilities at a novice-mid proficiency level as described in ACTFL 2006 guidelines. Their conversational abilities will be assessed by engaging them in role-play activities where both they and the instructor play a specific role. Students may play the role of a new student at QCC, or a party goer trying to make new friends, or may share some personal information with a close friend (played by the instructor) about a new boy/girlfriend. Oral tasks will prompt the students in providing personal information, i.e. their name, address, phone number, or class schedule, or describing their daily activities and hobbies. Desired student learning outcomes for the assignment (Students will…) List in parentheses the Curricular Objective(s) and/or General Education Objective(s) (1-10) associated with these desired learning outcomes for the assignment. Briefly describe the range of activities student will engage in for this assignment. Dialogue between student and instructor on familiar topics, such as family, lodgings, hobbies. What assessment tools will be used to measure how well students have met each learning outcome? (Note: a single assessment tool may be used to measure multiple learning outcomes; some learning outcomes may be measured using multiple assessment tools.) Gen-Ed objective (1) Communicate effectively through reading, writing, listening and speaking. (2) Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. Students will be asked to complete a task where they need to exchange personal information with a new acquaintance (personified by the instructor) about themselves, their family, their hobbies (see Appendix I). Curricular objectives N/A 4 PART III. ASSESSMENT STANDARDS (RUBRICS) Before the assignment is given, prepare a description of the standards by which students’ performance will be measured. This could be a checklist, a descriptive holistic scale, or another form. The rubric (or a version of it) may be given to the students with the assignment so they will know what the instructor’s expectations are for this assignment. Please note that while individual student performance is being measured, the assessment project is collecting performance data ONLY for the student groups as a whole. TABLE 7: ASSESSMENT STANDARDS (RUBRICS) Brief description of assignment: (Copy from Table 6 above) This project will assess students’ speaking abilities at a novice-mid proficiency level as described in ACTFL 2006 guidelines. Their conversational abilities will be assessed by engaging them in role-play activities where both they and the instructor play a specific role. Students may play the role of a new student at QCC, or a party guest trying to make new friends, or may share some personal information with a close friend (played by the instructor) about a new boy/girlfriend. Oral tasks will prompt the students in providing personal information, i.e. their name, address, phone number, or class schedule, or describing their daily activities and hobbies. Desired student learning outcomes from the assignment: (Copy from Column 1, Table 6 above; include Curricular and /or General Education Objectives addressed) Gen-Ed objectives (1) Communicate effectively through reading, writing, listening and speaking. Assessment measures for each learning outcome: (Copy from Column 3,Table 6 above) Students will be asked to complete a task where they need to exchange personal information with a new acquaintance, or in a friend’s house, or share a new boy/girlfriend’s personal information with a close friend (played by the instructor) regarding general personal information, family and lodgings. Standards for student performance: The parameters for measuring students’ speaking abilities will be to determine whether: (i) they understand the questions being asked by the interlocutor, (ii) they are accurate with regard to their use of Hebrew syntactic structures and vocabulary, (iii) they are understood by the interlocutor by using the correct intonation and pronunciation, and (iv) they speak with some degree of fluidity. 75% of the students tested are anticipated to meet the course’s expectations as described in the attached rubric (see Appendix II). 2) Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. Curricular objectives N/A 5 PART IV. ASSESSMENT RESULTS TABLE 8: SUMMARY OF ASSESSMENT RESULTS Use the following table to report the student results on the assessment. If you prefer, you may report outcomes using the rubric(s), or other graphical representation. Include a comparison of the outcomes you expected (from Table 7, Column 3) with the actual results. NOTE: A number of the pilot assessments did not include expected success rates so there is no comparison of expected and actual outcomes in some of the examples below. However, projecting outcomes is an important part of the assessment process; comparison between expected and actual outcomes helps set benchmarks for student performance. TABLE 8: SUMMARY OF ASSESSMENT RESULTS Desired student learning outcomes: (Copy from, Column 1,Table 6 above; include Curricular and/or General Education Objectives addressed) Student achievement: Describe the group achievement of each desired outcome and the knowledge and cognitive processes demonstrated. Gen-Ed objectives See Table 9 Communicate effectively through reading, writing, listening and speaking. Curricular objectives N/A TABLE 9. EVALUATION AND RESULTING ACTION PLAN In the table below, or in a separate attachment, interpret and evaluate the assessment results, and describe the actions to be taken as a result of the assessment. In the evaluation of achievement, take into account student success in demonstrating the types of knowledge and the cognitive processes identified in the Course Objectives. A) Analysis and interpretation of results 33 students (2 classes) completed the speaking task in Hebrew, and their performance was rated according to five parameters: listening comprehension, fluidity, pronunciation, vocabulary and grammar. Student performance was scored using the scoring rubric. The average score achieved for listening comprehension was 3.09, and the average score for pronunciation was 2.08. On both these parameters, students on average met expectations. The average score for fluidity of speech was 2.84; the average score for proficient use of vocabulary was 2.84, while the average score for accuracy of grammar was 2.54. On these three parameters students on average almost met expectations (see Chart 1 below): 6 Chart 1 Chart Title 4.5 4 3.5 Axis Title 3 2.5 2 1.5 1 0.5 0 Series1 LISTENING FLUIDITY PRONUNCIATION VOCABULARY GRAMMAR OVERALL 3.09 2.84 2.08 3.84 2.54 2.87 The total points a student could achieve (adding up all five parameters, each worth a maximum of four points) was 20. The scale agreed upon beforehand by the Assessment Committee defined the following ranges: 0-5 points= student performance does not meet expectations; 6 – 10 points= student performance almost meets expectations; 11-15 points= student performance meets expectations; 16-20= student performance exceeds expectations. The average of total points achieved by students tested in Hebrew was 14.03. According to the scale, this average falls in the 11-15 range, signifying that student overall performance on the task meets expectations. The Assessment Committee also predicted that 75% of students tested would achieve a score indicating that their performance meets expectations. The results for number of students who achieved each performance level in Hebrew are the following: 0 students scored at Level 1, 0-5 points, and their performance does not meet expectations; 12 students (36%) scored at Level 2, 6-10 points, and their performance almost meets expectations; 12 students (36%) scored at Level 3, 11-15 points, and their performance meets but does not exceed expectations; finally, 15 students (45%) scored at Level 4, and the quality of their performance exceeds expectations. Adding together Levels 3 and 4 gives us a total of 27 students (88%) who meet or exceed expectations, as defined by the scoring rubric. B1) Evaluation of student results The results of the speaking task in Hebrew show that, as predicted, at least 75% of students tested meet or exceed expectations on overall performance of the task: the actual percentage of students tested in Fall 2009 who meet or exceed expectations is 88%. The largest portion of students (45%) fell into the category rated as exceeding expectations, which is achieving 16-20 points out of 20. Adding together Levels 1 and 2, the percentage of students whose performance does not meet expectations is 18%. 7 If we look at student performance according to individual parameters, the highest average score was in listening comprehension, a passive skill. The other four parameters measured active skills. The two lowest scoring parameters were vocabulary (2.84) and grammar (2.54). These parameters require active knowledge of the material learned. Proficient use of vocabulary requires memorization, and accurate application of the rules of grammar requires analytic skill. In the context of a speaking task, linguists expect the performance on parameters of vocabulary and on grammar to be weaker than in the context of a writing task. The student results achieved on each of the individual parameters coincide with the expected results for a speaking task. B2) Evaluation of the assessment A formal evaluation of the assessment tools and assessment process has not been conducted, but informal reactions were very positive overall. The great majority of students evidently took the speaking task assessment very seriously, and had studied for it and students gained a real sense of accomplishment from having participated successfully in the speaking task. C) Further actions. The overall average achieved by students on the speaking task in Hebrew shows a satisfactory result, with 81.81% of students meeting expectations as defined by the Assessment Committee (see Chart 2 below): Chart 2 18 16 14 Axis Title 12 10 8 6 4 2 0 NOT MEETING EXPECTATIONS (0‐5) ALMOST MEETING EXPECTATIONS (6‐10) MEETING EXPECTATIONS (11‐15) EXCEEDING EXPECTATIONS (16‐20) Series2 0% 18.18% 36.36% 45.45% Series1 0 6 12 15 The averages achieved on each individual parameter in Hebrew show areas of relative strength, as well as two areas of relative weakness: vocabulary and grammar. After discussion of the results across all languages taught in the department, it has been decided by the Assessment Committee to postpone any changes in curriculum emphases or instructional methodology until more complete data is gathered, including student performance on written tasks, as well as a second assessment using the speaking task. After compiling and analyzing data for all assessment completed in academic year 8 2009-2010, the Assessment Committee, in consultation with the faculty of the department, will make recommendations concerning curriculum and classroom practice for the 2010-2011 academic year. . C. Resulting action plan: Based on A and B, what changes, if any, do you anticipate making? To start “oral training” earlier in the semester and give all the support and attention to those who need more so that they can better meet expectations. All in all the results were good out of 100% 18.18% “almost” met expectations. The others either met (36.36%) or exceeded (45.45%) the expectations for a total of 81.81%. 9 APPENDICES APPENDIX I SITUATIONS FOR ORAL ASSESSMENT IN HEBREW GREETINGS A new student joined your class. You want to know him better and you ask him/her following information Possible questions: 1. 2. 3. 4. 5. 6. 7. 8. Where do you live? How do you feel today? How old are you? What is your nationality? What are your hobbies? What do you like or dislike? Did you have good friends in your last school? Could you describe them? ? איפה אתה גר.1 ? איך אתה מרגיש היום את/בן כמה אתה ? איזה אזרחות יש לך ת לעשות בזמן הפנוי/את אוהב/מה אתה ? את לא אוהבת לעשות/מה אתה י עליהם/היו לך חברים טובים בתיכון? תספר 10 DESCRIPTION OF LODGINGS Describe your room/apartment at the University or at home. Describe the pieces of furniture in it, what colors are used. Explain if you need or want other pieces of furniture. Tell why you like or do not like the room/apartment: Possible questions: 1. 2. 3. 4. 5. 6. 7. Do you live in a house or an apartment? How big is it? How many rooms? Do you like living with your parents or alone with roommates? Where does your family live? How is your room? Do you have any pets? Do you like to do your homework at home or in the school library? תאור מגורים י/ תסביר. הצבעים, תאר את הריה וט.תארי את המגורים שלך באוניברסיטה או בבית/תאר .י אם אתה אוהב או לא אובה את המגורים שלך/ תגיד.אם אתה צריך או רוצה ריהוט נוסף ? אתה גר בדירה או בבית ? כמה גדול זה? כמה חדרים ? אתה אוהב לחיות עם ההורים שלך או לבד עם חברים ? איפה המשפחה שלך גרה ? איך החדר שלך ? יש לך חיות בבית ?ספר-אתה אובה לעשות שעורי בית בבית או בספריה של הבית 11 APPENDIX II RUBRIC FOR ORAL ASSESSMENT IN HEBREW Listening Comprehension Performance exceeds expectations (4 points) Performance meets expectations (3 points) Performance almost meets expectations (2 points) Performance does not meet expectations (0 – 1 points) Fluidity Pronunciation Vocabulary Grammar Rich use of vocabulary Correct use of basic language structures (1-5 errors) Does not interfere with communication Adequate and accurate use of vocabulary for this level Adequate use of basic language structures (6-10 errors) Speech choppy and/or slow with frequent pauses. Few or incomplete thoughts Occasionally interferes with communication Somewhat inadequate and/or inaccurate use of vocabulary Emerging use of basic language structures (11-15 errors) Speech halting and uneven with long pauses or incomplete thoughts Frequently interferes with communication Inadequate and/or inaccurate use of vocabulary Inadequate and/or inaccurate use of basic language structures (more than 16 errors) Student understands the examiner’s questions and responds easily and without probing Speech continuous with few pauses or stumbling Student understands the examiner’s questions and knows how to respond but needs occasional probing Some hesitation but manages to continue and to complete her/his thoughts Student only understands the examiner’s questions after probing Student fails to understand most questions even after probing Enhances communication 12 Summary & Conclusion Generally speaking, data have portrayed a quite uniform acquisition scenario among the six language groups tested. As shown in Chart 1 below, the majority of students met and exceeded the standard set up for the course, namely Chinese (97%), French (88%), German (82%), Hebrew (81%), Italian (86%), and Spanish (88%). The percentage of students that performed below the expected level was irrelevant, and, in some languages, null. The number of students that almost reached the expected proficiency level was also uniform among the different languages ranging from the 3 percentile and the 18 percentile. Chart 1 Percentage of students in each proficiency level among the six languages tested 70 60 Chinese 50 French 40 German 30 Hebrew 20 Italian 10 Spanish 0 Below Almost Lev Level Exceed Lev The scenario does not substantially change when we take a closer look at how students performed with each parameter tested, namely “Listening Comprehension”, “Fluidity”, “Pronunciation”, “Vocabulary”, and “Grammar”. Results, in fact, do not report significant discrepancies among these parameters as well as the six language groups. As indicated below, for each parameter, students reached or slightly exceeded the expected level (3.0). Chart 2 Accuracy rates of the five parameters among the six languages 4 3.5 Chinese 3 French 2.5 2 German 1.5 Hebrew Italian 1 0.5 Spanish 0 Listening Fluidity Pronunc. Vocab. Grammar Interestingly, the typology and the complexity of the sound pattern of a language do not seem to delay the development of a particular speaking skill. For instance, the tone system that 13 characterizes languages such as Chinese does not create major acquisition delays to students whose native phonological system is quite different. In conclusion, given the overall positive outcomes obtained, we can safely assume that the instruction time, the teaching methodology practiced and the tools used appear to be effective in helping our students reach the desired oral ability. 14 LA ACADEMY COHORT Spring 2010 Date: 6-3-2010. Department: Foreign Languages and Literatures Course: LH 111 Curriculum or Curricula: LA PART I. STUDENT LEARNING OBJECTIVES For Part I, attach the summary report (Tables 1-4) from the QCC Course Objectives Form. TABLE 1. EDUCATIONAL CONTEXT LI-111 is the first part of the introductory sequence of foreign language study (a requirement for a successful transfer to the junior year of a baccalaureate program). It is a foundation course required for (AA) degree in Liberal Arts and Sciences, Fine Arts, and Business Transfer. TABLE 2. CURRICULAR OBJECTIVES Note: Include in this table curriculum-specific objectives that meet Educational Goals 1 and 2: Curricular objectives addressed by this course: N/A TABLE 3. GENERAL EDUCATION OBJECTIVES Gen Ed objective’s ID number from list (1-10) General educational objectives addressed by this course: Select from preceding list. (1) Communicate effectively through reading, writing, listening and speaking. (2) Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. 15 TABLE 4: COURSE OBJECTIVES AND STUDENT LEARNING OUTCOMES Course objectives Learning outcomes 1. Provide basic information in Students will be able to talk about themselves by providing their name, Hebrew about themselves, their city, address, phone number, school schedule. They will also be able to describe their classmates, family, friends, their family or hobbies by answering questions in prompted dialogues. hobbies, and daily activities. 2. Demonstrate ability to accomplish simple communicative tasks on every day topics such as greeting people or introduce themselves to others, or describing their life in school or at work, or during their spare time. 3. Read and understand and write simple texts on the abovementioned topics. 4. Understand and use essential vocabulary related to everyday life. Students will be able to greet people and/or introduce themselves, or exchange personal information by participating in communicative tasks where they will play the role of a new student at QCC or a party guest trying to make friends. Students will be able to understand a small text about the daily routine of small children. They will also be able to describe in writing form about themselves, their hobbies, their plans, their families. Students will be able to provide vocabulary relating to their daily activities. 16 PART II. ASSIGNMENT DESIGN: ALIGNING OUTCOMES, ACTIVITIES, AND ASSESSMENT TOOLS For the assessment project, you will be designing one course assignment, which will address at least one general educational objective, one curricular objective (if applicable), and one or more of the course objectives. Please identify these in the following table: TABLE 5: OBJECTIVES ADDRESSED IN ASSESSMENT ASSIGNMENT Course Objective(s) selected for assessment: (select from Table 4) 1. Provide basic information in Italian about yourself, your city, your classmates, your family, friends, hobbies, and daily activities. 2. Demonstrate ability to accomplish simple communicative tasks on every day topics such as greeting people or introducing yourself to others, or describing your life in school or at work and in your spare time. Curricular Objective(s) selected for assessment: (select from Table 2) N/A General Education Objective(s) addressed in this assessment: (select from Table 3) 1. Communicate effectively through reading, writing, listening and speaking. 2. Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. In the first row of Table 6 that follows, describe the assignment that has been selected/designed for this project. In writing the description, keep in mind the course objective(s), curricular objective(s) and the general education objective(s) identified above, The assignment should be conceived as an instructional unit to be completed in one class session (such as a lab) or over several class sessions. Since any one assignment is actually a complex activity, it is likely to require that students demonstrate several types of knowledge and/or thinking processes. Also in Table 6, please a) identify the three to four most important student learning outcomes (1-4) you expect from this assignment b) describe the types of activities (a – d) students will be involved with for the assignment, and c) list the type(s) of assessment tool(s) (A-D) you plan to use to evaluate each of the student outcomes. (Classroom assessment tools may include paper and pencil tests, performance assessments, oral questions, portfolios, and other options.) Note: Copies of the actual assignments (written as they will be presented to the students) should be gathered in an Assessment Portfolio for this course. 17 TABLE 6: ASSIGNMENT, OUTCOMES, ACTIVITIES, AND ASSESSMENT TOOLS Briefly describe the assignment that will be assessed: The final exam will be used as the assessment tool to evaluate students’ abilities in listening, reading, and writing as well as their knowledge in vocabulary and grammar. Listening comprehension abilities will be evaluated by asking the students to listen to three paragraphs: one about a student going to study to Israel while his family stays in N.Y.; the other one about a couple arguing about the date and their respective occupation on specific days; the third one about daily life. Students need to answer questions, then complete a true or false section and a multiple choice section. Reading abilities will be evaluated by asking the students to read two passages: one about two children discussing the importance of their parents on their way to school; the other about a student at QCC, his daily routine and his parents’ occupation. Students need to answer questions in one instance and T/F in the other. Vocabulary knowledge will be assessed by asking the students to complete two sections: they had to choose the appropriate word (out of a provided list) to complete a sentence in vocabulary. Grammar accuracy will be assessed by asking the students to complete four sections using the correct verb conjugations; to choose the correct demonstrative adjectives in sentences; adjective forms, adverb forms, and finally plural forms of entire sentences in the singular form. Writing abilities will be assessed by asking the students to write a composition about themselves: country of origin and nationality, physical appearance and personality traits (use at least three adjectives) age and civil status, studies and/or occupation, three daily activities (where, with whom), three activities you/he/she/ like to do on the weekend, plans for a vacation (where you/he/she want to go, where are you/he/she going to do, and with whom).. Desired student learning outcomes for the assignment (Students will…) List in parentheses the Curricular Objective(s) and/or General Education Objective(s) (1-10) associated with these desired learning outcomes for the assignment. Gen-Ed objective (1) Communicate effectively through reading, writing, listening and speaking. (2) Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. Curricular objectives N/A Briefly describe the range of activities student will engage in for this assignment. For the speaking: Students will be requested to complete a task where they need to exchange personal information with a new acquaintance , or in a friend’s house, or share a new boy/girl friend ‘s personal information with a close friend (personified by the instructor) regarding personal information, family and lodgings. For the writing: Students were required to listen to 3 short passages and answer to questions; they had to choose the appropriate word to complete a sentence in vocabulary; deal with verb conjugations; to choose the correct demonstrative adjectives in sentences; read two passages and answer to questions and T/F options; finally, write a What assessment tools will be used to measure how well students have met each learning outcome? (Note: a single assessment tool may be used to measure multiple learning outcomes; some learning outcomes may be measured using multiple assessment tools.) Students will be requested to complete a task where they need to exchange personal information with a new acquaintance (personified by the instructor) about themselves, their family, their lodgings and their hobbies (see Appendix I). Listening comprehension abilities will be evaluated by asking the students to listen to three paragraphs: one about a student going to study to Israel while his family stays in N.Y.; the other one about a couple arguing about the date and their respective occupation on specific days; the third one about daily life. Students 18 composition about themselves, their daily life, their hobbies, and families. need to answer questions, true or false and a multiple choice. Reading abilities will be assessed by asking the students to read three passages: one about two children discussing the importance of their parents on their way to school; the other about a student at QCC, his daily routine and his parents’ occupation. Students need to answer questions in one instance, and T/F in the other. Vocabulary knowledge will be assessed by asking the students to complete two sections: they had to choose the appropriate word (out of a provided list) to complete a sentence in vocabulary. Grammar accuracy will be assessed by asking the students to complete four sections using the correct verb conjugations; to choose the correct demonstrative adjectives in sentences; adjective forms, adverb forms, and finally plural forms of entire sentences in the singular form. 19 PART III. ASSESSMENT STANDARDS (RUBRICS) Before the assignment is given, prepare a description of the standards by which students’ performance will be measured. This could be a checklist, a descriptive holistic scale, or another form. The rubric (or a version of it) may be given to the students with the assignment so they will know what the instructor’s expectations are for this assignment. Please note that while individual student performance is being measured, the assessment project is collecting performance data ONLY for the student groups as a whole. TABLE 7: ASSESSMENT STANDARDS (RUBRICS) Brief description of assignment: (Copy from Table 6 above) This project will assess students’ speaking abilities at a novice-mid proficiency level as described in ACTFL 2006 guidelines. Their conversational abilities will be assessed by engaging them in role-play activities where both they and the instructor play a specific role. Students may play the role of a new student at QCC, or a party guest trying to make new friends, or may share some personal information with a close friend (played by the instructor) about a new boy/girlfriend. Oral tasks will prompt the students in providing personal information, i.e. their name, address, phone number, or class schedule, or describing their daily activities and hobbies. Desired student learning outcomes from the assignment: (Copy from Column 1, Table 6 above; include Curricular and /or General Education Objectives addressed) Gen-Ed objectives (1) Communicate effectively through reading, writing, listening and speaking. 3) Use analytical reasoning to identify issues or problems and evaluate evidence in order to make informed decisions. Curricular objectives N/A Assessment measures for each learning outcome: (Copy from Column 3,Table 6 above) For the speaking: Students will be requested to complete a task where they need to exchange personal information with a new acquaintance , or in a friend’s house, or share a new boy/girl friend ‘s personal information with a close friend (personified by the instructor) regarding personal information, family and lodgings. For the writing: Students were required to listen to 3 short passages and answer to questions; they had to choose the appropriate word to complete a sentence in vocabulary; deal with verb conjugations; to choose the correct demonstrative adjectives in sentences; read two passages and answer to questions and T/F options; finally, write a composition about themselves, their daily life, their hobbies, and families. Standards for student performance: The parameters for measuring students’ speaking abilities will be to determine whether: (i) they understand the questions being asked by the interlocutor, (ii) they are accurate with regard to their use of Hebrew syntactic structures and vocabulary, (iii) they are understood by the interlocutor by using the correct intonation and pronunciation, and (iv) they speak with some degree of fluidity. 75% of the students tested are anticipated to meet the course’s expectations as described in the attached rubric (see Appendix II). The parameters used to measure students’ writing abilities will be to determine whether: (i) they will provide all the information they have been asked, (ii) their writing displays a vocabulary appropriate to their proficiency level, (iii) they show an adequate control of the syntactic structures for an Italian beginning level class. (iv) Their sentences are fully developed, even though sporadically connected. 75% of the students tested are anticipated to meet the course’s expectations as described in the attached rubric (see Appendix IV). Students’ vocabulary knowledge will be determined by calculating the accuracy rates in providing the correct words or phrases in a given situation. 75 per cent of them are expected to meet the course standards (80% accuracy rate). 20 Students’ grammar knowledge will be determined by calculating their accuracy rates in providing the correct forms or structures. 75 per cent of them are expected to meet the course standards (80% accuracy rate). Students’ listening comprehension ability will be determined by calculating their accuracy rates in providing the correct answers to the questions asked. 75 per cent of them are expected to meet the course standards (80% accuracy rate). Students’ reading comprehension ability will be determined by calculating their accuracy rates in providing the correct answers to the questions asked. 75 per cent of them are expected to meet the course standards (80% accuracy rate) (see Appendix III for scoring scales). 21 PART IV. ASSESSMENT RESULTS TABLE 8: SUMMARY OF ASSESSMENT RESULTS Use the following table to report the student results on the assessment. If you prefer, you may report outcomes using the rubric(s), or other graphical representation. Include a comparison of the outcomes you expected (from Table 7, Column 3) with the actual results. NOTE: A number of the pilot assessments did not include expected success rates so there is no comparison of expected and actual outcomes in some of the examples below. However, projecting outcomes is an important part of the assessment process; comparison between expected and actual outcomes helps set benchmarks for student performance. TABLE 8: SUMMARY OF ASSESSMENT RESULTS Desired student learning outcomes: (Copy from, Column 1,Table 6 above; include Curricular and/or General Education Objectives addressed) Student achievement: Describe the group achievement of each desired outcome and the knowledge and cognitive processes demonstrated. Gen-Ed objectives See Table 9. Communicate effectively through reading, writing, listening and speaking. Curricular objectives N/A 22 TABLE 9. EVALUATION AND RESULTING ACTION PLAN In the table below, or in a separate attachment, interpret and evaluate the assessment results, and describe the actions to be taken as a result of the assessment. In the evaluation of achievement, take into account student success in demonstrating the types of knowledge and the cognitive processes identified in the Course Objectives. A. Analysis and interpretation of assessment results (oral ability) What does this show about what and how the students learned? 23 students completed the speaking task in Hebrew, and their performance was rated according to five parameters: listening comprehension, fluidity, pronunciation, vocabulary and grammar. The average score achieved for listening comprehension was 2.34; the average score for pronunciation was 2.28. The average score for fluidity was 2.28; the average score for proficient use of vocabulary was 1.95; the average score for accuracy of grammar was 2.69. In these parameters students on average almost met expectations (the average was quite close to the 3.0 mark). The total points a student could achieve (adding up all five parameters, each worth a maximum of four points) was 20. The scale agreed upon beforehand by the Assessment Committee defined the following ranges: 0-5 points= student performance does not meet expectations; 6 – 10 points= student performance almost meets expectations; 11-15 points= student performance meets expectations; 16-20= student performance exceeds expectations. The average of total points achieved by students tested in Italian was 11.04. According to the scale, this average falls in the 11-15 range, signifying that student overall performance on the task meets expectations. The Assessment Committee also predicted that 75% of students tested would achieve a score indicating that their performance meets expectations. The results for percentage of students who achieved each performance level in Italian are the following: 4.34% of students scored at Level 1, 0-5 points, and their performance does not meet expectations; 30.43% scored at Level 2, 6-10 points, and their performance almost meets expectations; 56.52% scored at Level 3, 11-15 points, and their performance meets but does not exceed expectations; finally, 8.69% scored at Level 4, and the quality of their performance exceeds expectations. Adding together Levels 3 and 4 gives us a total of 64.52% of students who meet or exceed expectations, as defined by the scoring rubric B. Evaluation of the assessment process: What do the results suggest about how well the assignment and the assessment process worked both to help students learn and to show what they have learned? B1) Evaluation of students’ results The results of the speaking task in Hebrew show that, as predicted, at least 64.52% of students tested meet or exceed expectations on overall performance of the task: the actual percentage of students tested in Fall 2009 who meet or exceed expectations is 81%. Adding Levels 1 and 2 together, the percentage of students whose performance does not meet expectations is 35%. 23 If we look at students’ performance according to individual parameters, the highest average score was in pronunciation (1.78%). The second highest scoring parameters were ‘listening comprehension’ (2.34%). The other three parameters: fluidity, vocabulary and grammar all scored very similarly, respectively 2.28, 1.95, and 2.69. B2) Evaluation of the assessment tools The results obtained seem to indicate that the assessment tools used and the assessment process undertaken have been successful in accurately determining our students’ oral proficiency level. Not only has it shown that more than2/3 of the students had met the expectation, but it has also provided them with the appropriate tool that would facilitate their learning process. The great majority of students, in fact, took the speaking task assessment very seriously, and studied extensively for it. C. Resulting action plan: Based on A and B, what changes, if any, do you anticipate making? Further actions The overall average achieved by students on the speaking task in Italian shows a satisfactory result, with 65% of students meeting expectations as defined by the Assessment Committee. These results are lower than those obtained in the Oral Assessment conducted in fall 09 (81%). The Assessment Committee, in consultation with the faculty of the department, will discuss these results and make recommendations concerning curriculum and classroom practice for the 2010-2011 academic year. A. Analysis and interpretation of assessment results: (auditory, vocabulary, grammar, reading and writing abilities) What does this show about what and how the students learned? The data of LH111 students that completed the final exam (assessment tool) in Hebrew was gathered. Their performance was rated according to five categories: listening comprehension, vocabulary, grammar, reading and writing. The results are as follows: SCORING SHEET‐ LH 111Total Final Overall Listening Vocabulary Grammar Reading Writing 16 16 36 16 16 15.43 14.04 25.62 12.73 13.47 23 23 23 23 23 Total 81.29 24 The average score achieved was 81.29% The total points a student could achieve (adding up all five categories) was 100. The scale agreed upon beforehand by the Assessment Committee defined the following ranges: 0-60points= student performance does not meet expectations; 61–78 points= student performance almost meets expectations; 79-89 points= student performance meets expectations; 90-100= student performance exceeds expectations. OVERALL. Student performance … Does not meet Almost meets expectations expectations Meets expectations Exceed expectations 0‐60 0 61‐78 1 79‐89 8 90‐100 14 0 4% 35% 61% The overall student performance shows that only 61% of students meet or exceed expectations. The results for percentage of students in each category are as follows: LISTENING. Student performance… Does not meet expectations 9.5 Almost meets expectations 10 to 12.5 Meets expectations 13 to 14.5 Exceed expectations 15‐16 0 0 4 19 0% 0% 17.5% 82.50% VOCABULARY. Student performance … Does not meet expectations 9.5 Almost meets expectations 10 to 12.5 Meets expectations 13 to 14.5 Exceed expectations 15‐16 1 4 5 13 4.34% 17.39% 21.73 56.54 GRAMMAR. Student performance … Does not meet Almost meets expectations expectations Meets expectations Exceed expectations 0‐21.5 22‐28 28.5‐32 32.5‐36 6 7 5 5 26.08% 30.46% 21.73% 21.73% 25 READING. Student performance … Does not meet Almost meets expectations expectations Meets expectations Exceed expectations 9.5 10 to 12.5 13 to 14.5 15‐16 2 8 5 8 8.7% 34.78 21.74% 34.78% WRITING. Student performance … Does not meet Almost meets expectations expectations Meets expectations Exceed expectations 9.5 10 to 12.5 13 to 14.5 15‐16 1 8 4 10 4.35% 34.78% 17.39% 43.48% B. Evaluation of the assessment process: What do the results suggest about how well the assignment and the assessment process worked both to help students learn and to show what they have learned? The results in the vocabulary, grammar, reading and writing categories show that, (except for vocabulary) contrary to the prediction, less than 75% of students tested meet or exceed expectations on the performance of each task. The only category where the students meet or exceed expectations is listening. The actual percentage of students tested who meet or exceed expectations in vocabulary is 78.27%, in grammar is 43.46%, in reading is 56.52%, in writing is 60.87% and in listening is 82.5% B2) Evaluation of the assessment . The results obtained seem to indicate that the assessment tools used and the assessment process undertaken have been effective in determining our students’ proficiency level. C. Resulting action plan: Based on A and B, what changes, if any, do you anticipate making? The overall average achieved by students on the assessment in Hebrew is 81.29, but the averages achieved on each individual category in Hebrew show that all areas need to improve except for listening. The weakest area is grammar. These results need to be addressed. The committee will meet in the fall to discuss the results and determine recommendations. It will then share and discuss the recommendations with the faculty. 26 APPENDICES APPENDIX I SITUATIONS FOR ORAL ASSESSMENT IN HEBREW GREETINGS A new student joined your class. You want to know him better and you ask him/her following information Possible questions: 1. 2. 3. 4. 5. 6. 7. 8. Where do you live? How do you feel today? How old are you? What is your nationality? What are your hobbies? What do you like or dislike? Did you have good friends in your last school? Could you describe them? ? איפה אתה גר.1 ? איך אתה מרגיש היום את/בן כמה אתה ? איזה אזרחות יש לך ת לעשות בזמן הפנוי/את אוהב/מה אתה ? את לא אוהבת לעשות/מה אתה י עליהם/היו לך חברים טובים בתיכון? תספר 27 DESCRIPTION OF LODGINGS Describe your room/apartment at the University or at home. Describe the pieces of furniture in it, what colors are used. Explain if you need or want other pieces of furniture. Tell why you like or do not like the room/apartment: Possible questions: 1. 2. 3. 4. 5. 6. 7. Do you live in a house or an apartment? How big is it? How many rooms? Do you like living with your parents or alone with roommates? Where does your family live? How is your room? Do you have any pets? Do you like to do your homework at home or in the school library? תאור מגורים י/ תסביר. הצבעים, תאר את הריה וט.תארי את המגורים שלך באוניברסיטה או בבית/תאר .י אם אתה אוהב או לא אובה את המגורים שלך/ תגיד.אם אתה צריך או רוצה ריהוט נוסף ? אתה גר בדירה או בבית ? כמה גדול זה? כמה חדרים ? אתה אוהב לחיות עם ההורים שלך או לבד עם חברים ? איפה המשפחה שלך גרה ? איך החדר שלך ? יש לך חיות בבית ?ספר-אתה אובה לעשות שעורי בית בבית או בספריה של הבית 28 APPENDIX II HOLISTIC RUBRIC FOR ORAL ASSESSMENT IN HEBREW Listening Comprehension Performance exceeds expectations (4 points) Performance meets expectations (3 points) Performance almost meets expectations (2 points) Performance does not meet expectations (0 – 1 points) Fluidity Pronunciation Vocabulary Grammar Rich use of vocabulary Correct use of basic language structures (1-5 errors) Does not interfere with communication Adequate and accurate use of vocabulary for this level Adequate use of basic language structures (6-10 errors) Speech choppy and/or slow with frequent pauses. Few or incomplete thoughts Occasionally interferes with communication Somewhat inadequate and/or inaccurate use of vocabulary Emerging use of basic language structures (11-15 errors) Speech halting and uneven with long pauses or incomplete thoughts Frequently interferes with communication Inadequate and/or inaccurate use of vocabulary Inadequate and/or inaccurate use of basic language structures (more than 16 errors) Student understands the examiner’s questions and responds easily and without probing Speech continuous with few pauses or stumbling Student understands the examiner’s questions and knows how to respond but needs occasional probing Some hesitation but manages to continue and to complete her/his thoughts Student only understands the examiner’s questions after probing Student fails to understand most questions even after probing Enhances communication 29 APPENDIX III SCORING SCALES Grammar Task Scale (Range: 0 to 36) GRAMMAR SECTION ACCURATE USE OF THE GRAMMATICAL STRUCTURES LEARNED IN THE COURSE Performance More than 90% completion of the task exceeds (32.5- 36 points) expectations Performance Between 79% and 89% completion of the task meets (28.5- 32 points) expectations Performance Between 64% and 78% completion of the task almost (22 – 28 points) meets expectations Performance Between 0% to 60% completion of the task does not (0 - 21.5 points) meet expectations Vocabulary Task Scale (Range: 0 to 16) VOCABULARY ACCURATE AND ADEQUATE USE OF THE VOCABULARY SECTION LEARNED IN THE COURSE Performance exceeds expectations More than 90% completion of the task (15 - 16 points) Performance meets expectations Between 79% and 89% completion of the task (13 – 14.5 points) Performance almost meets expectations Between 62% and 78% completion of the task (10 – 12.5 points) Performance does not meet expectations Between 0% to 61% completion of the task (0- 9.5 points) 30 Listening Task Scale (Range: 0 to 16) LISTENING SECTION UNDERSTAND PHRASES, EXPRESSIONS AND SHORT MESSAGES RELATED TO THE TOPICS COVERED IN THE COURSE Performance More than 90% completion of the task exceeds expectations (15 – 16 points) Performance Between 79% and 89% completion of the task meets (13- 14.5 points) expectations Performance almost meets expectations Performance does not meet expectations Between 65% and 78% completion of the task (10 – 12.5 points) Between 0% to 60% completion of the task (0 to 9.5 points) Reading Task Scale (Range: 0 to 16) READING SECTION UNDERSTAND SHORT AND SIMPLE MESSAGES RELATED TO THE TOPICS COVERED IN THE COURSE Performance More than 90% completion of the task exceeds expectations (15 – 16 points) Performance Between 79% and 89% completion of the task meets (13- 14.5 points) expectations Performance Between 65% and 78% completion of the task almost (10- 12.5 points) meets expectations Performance Between 0% to 60% completion of the task does not (0 to 9.5 points) meet expectations 31 APPENDIX IV WRITING TASK HOLISTIC RUBRIC Exceeds expectations Meets expectations Almost meets expectations Does not meet expectations Task Completion Level of Discourse Vocab. Grammar Superior completion of the task. Ss fully address the information requested, and provide additional details Completion of task. Ss fully address the information provided, but do not provide additional details Partial completion of task. Ss complete no more than 60% of the information requested Sentences are fully developed and interconnected with conjunctions (e.g. AND, BUT, or BECAUSE ) Rich use of vocabulary Perfect control of the syntactic structures required (accuracy level 90% - 100%) Sentences are fully developed. Cohesive devices, however, are sporadically used Adequate and accurate use of vocabulary Sentences are somewhat complete. Rare use of cohesive devices Somewhat inadequate and/or inaccurate use of vocabulary Adequate control of the syntactic structures. Some grammatical errors (accuracy level 79% - 89%) Emerging control of syntactic structures. Several grammatical errors (accuracy level 61% - 78%) Minimal completion of task. Ss complete less than 40% of the information requested. Sentences are mostly incomplete. No use of cohesive devices Inadequate and/or inaccurate use of vocabulary Minimal control of syntactic structures. Numerous grammatical errors (accuracy level 0% - 60%) Conclusions & Action Plan 32 Generally speaking, data portray a uniform acquisition scenario among the foreign languages evaluated, namely, Chinese, French, German, Hebrew, Italian and Spanish. As Table 1 shows, all six language groups display similar percentages of students reaching or exceeding the proficiency levels set up for a Beginning I language course. Table 1 Percentage of students reaching or exceeding the required proficiency levels for each language group 80 70 Chinese 60 French 50 German 40 Hebrew 30 Italian 20 Spanish 10 0 However, contrary to our predictions, no language group reported more than 75 percent of their students meeting the proficiency standards of the course. In fact, an average of less than 2/3 of our learners reached the desired proficiency levels. In any case, a closer look at the data reveals an interesting acquisition phenomenon. Our L2 learners seem to encounter less difficulty in developing the required speaking and auditory skills than their reading or writing abilities. Despite their limited amount of instruction and exposure to the target language, a relatively high number of students (avg. 87%), are able to speak and converse at the expected level of a beginning course, irrespective of the language they are learning. Furthermore, their ability to comprehend oral messages seems to be equally developed. In fact, the percentage of students meeting the standards of the course is 75 percent (see Table 2 below). Table 2 33 Percentage of students reaching or exceeding the required level of oral, auditory, reading and writing proficiency for each language group 100 90 80 70 Oral Auditory Reading Writing 60 50 40 30 20 10 0 Chinese French Hebrew German Italian Spanish The reading and writing skills, however, do not seem to follow a similarly expedite developmental pattern. Data indicate that an average of only 63 percent of the students write at the level required by the course. If we eliminate the high percentage displayed by the students of Chinese (92%), the number reaches worrisome levels (55%). These results are quite interesting and intriguing. Normally, one would expect just opposite outcomes given that performance limitations of psychological nature (i.e. nervousness, anxiety) are usually more visible in generating oral than written language. Such a clear discrepancy in students’ oral and written use of their target language (henceforth: L2) calls for a re-analysis of the oral assessment tools used, mainly with regard to their reliability and usefulness in generating accurate data. This re-evaluation should help us eliminate or, at least reduce, the effects of external factors that might have altered the results obtained. The personal and oral nature of this assessment process, in fact, may be easily subject to human errors or bias. With that in mind, the department will organize some training sessions for the instructors administering the oral test in order to ensure that the assessment tool is used uniformly and appropriately. In any case, the low percentage of students meeting the writing standards of the course is also reflected in the appropriate use of L2 syntactic structures. Data indicate that an average of 44 percent of the students tested were able to reach the accuracy standards required by the course. And again, if we eliminate the percentage of the students of Chinese from our calculations, the percentile lowers to 38 percent. Interestingly, such a lower accuracy level is also encountered in L2 oral production. Across the six language groups, the appropriate use of grammatical structures is one of the weakest oral abilities. That being the case, one could assume that internalizing L2 grammar rules is problematic. Such difficulties are displayed in both oral and written language modes. 34 Fortunately, this does not seem to be case when students need to master the required vocabulary. Lexical items do not appear to be acquisitionally as problematic as the grammar rules. As Table 3 indicates, the overall percentage of students using the required vocabulary is clearly higher (62%), even though the Italian and Spanish learners are still showing some problems (33% and 43%, respectively) Table 3 Percentage of students reaching or exceeding the required levels of lexical and syntactic proficiency for each language group 80 70 60 50 Grammar Voc. 40 30 20 10 0 Chinese French Hebrew German Italian Spanish In sum, students have shown to be struggling with the mastering of L2 grammar rules. Their behavior is quite consistent, equally involving the oral and written use of their target language. The reasons justifying this lack of accuracy may be various and of different nature. The groups that have shown greater number of problems are those learning morphologically rich languages such as Italian, French Spanish and German. At a beginning level, the acquisition of grammar is mostly morphologically-based. It is well known that morphology is a linguistic component that is usually acquired very late. However, the similarly lower number of students of Italian and Spanish reaching the desired level of lexical knowledge seems to highlight a general behavior of poor study skills and habits. In fact, L2 learners appear to fall short whenever the skill requires a more attentive participation and thorough analysis of the language. With this in mind, the department will implement a greater variety of grammar activities and tasks in the curricula of the beginning language courses along with a closer monitoring of students' homework and lab assignments. 35