Annual Assessment Report to the College 2010-2011 College: __Science and Mathematics____________________ Department: ___Chemistry and Biochemistry____________ Program: ______Chem BA, BS/MS, Biochem BS/MS_______________ Note: Please submit report to your department chair or program coordinator and to the Associate Dean of your College by September 30, 2011. You may submit a separate report for each program which conducted assessment activities. Liaison: _____Thomas Minehan_______________________ 1. Overview of Annual Assessment Project(s) 1a. Assessment Process Overview: Provide a brief overview of the intended plan to assess the program this year. Is assessment under the oversight of one person or a committee? Assessment in our department is under the oversight of a four-person committee, currently comprised of Curtis, Schrodi, Minehan (liaison), and Nguyen-Graff. The following assessment activities were planned for this year: a. Implement rubrics for the assessment of (SLO4b): Work effectively in a laboratory environment, including the ability to maintain a proper lab notebook. b. Develop a rubric for the evaluation of written research/scientific reports (SLO2, 3) c. Assess basic knowledge in biochemistry and general chemistry (SLO1) using standardized exam questions in course finals. d. Review results from the implementation of a rubric for the assessment of (SLO2): classroom presentations and graduate student seminars. e. Assess students’ ability to perform Quantitative Chemical Analysis (SLO6). f. Evaluate previously developed and implemented Personnel Procedures for non-tenure track faculty 1b. Implementation and Modifications: Did the actual assessment process deviate from what was intended? If so, please describe any modification to your assessment process and why it occurred. Although a “customizable” lab notebook rubric has been crafted for the department, the adoption/use of this rubric by lab coordinators has occurred in only a limited number of courses, and this data is presented below. As a result, we will further publicize the rubric and evaluate SLO4b in next year’s assessment report as well. Assessment of lab notebooks in the course Chem 321L has occurred for several years without the new rubric, and these results will also be presented below. 2. Student Learning Outcome Assessment Project: Answer questions according to the individual SLO assessed this year. If you assessed an additional SLO, report in the next chart below. 2a. Which Student Learning Outcome was measured this year? SLO 1: Assess basic knowledge in the following areas of chemistry: general chemistry and biochemistry. 2b. What assessment instrument(s) were used to measure this SLO? For general chemistry (Chem 101): 25 multiple-choice questions from an ACS standardized exam in general chemistry were embedded in the course final. Relevant questions were chosen from the ACS general chemistry exam, since the comprehensive exam covers topics not presented in Chem 101. For biochemistry (Chem 464): 5 multiple-choice questions from an ACS standardized exam in biochemistry were embedded in the course final. 2d. Describe the assessment design methodology: For example, was this SLO assessed longitudinally (same students at different points) or was a cross-sectional comparison used (comparing freshmen with seniors)? If so, describe the assessment points used. A cross-sectional design methodology, in which freshman are compared with seniors, was used: General Chemistry: Chem 101 is our gateway course taken by both majors and non-majors, and is typically populated by freshman. The foundational concepts introduced in this course are crucial for student success in all subsequent courses in the major. Thus, assessment at this introductory stage will allow us to establish a baseline level of student performance useful for comparison with assessments done in later courses in our undergraduate program. Biochemistry: for the chemistry seniors who make up a significant portion of the students in Chem 464, this class typically would be one of the last courses they take in the major, and thus the assessment in this course may provide information on how our students have been able to both understand and apply fundamental concepts in chemistry they have learned throughout the program. 2e. Assessment Results & Analysis of this SLO: Provide a summary of how the data were analyzed and highlight important findings from the data collected. General Chemistry: it was found that 50% of the students in the section performed above the national average, and the average score for the section was slightly better than the national average (58.3% vs. 56.4%). In addition, there was a strong correlation between the results on the assessment questions and the overall course grades. This may indicate that the course difficulty is roughly at the correct level compared to national courses. Thus, at the introductory level our students appear to be on par with their peers across the country in terms of their grasp of foundational chemical concepts. Biochemistry: 55% of the students taking Chem 464 (22/40) got 3 or more out of the 5 assessment questions on the final exam correct (a score of 3/5 has been identified as a benchmark level for student success). The average class score was 54% (2.7/5 correct). While it is encouraging that more than half of the class has achieved the benchmark success level, there are no previous results to compare this performance to since this was the first semester of assessment for biochemistry. Taking the national average as a benchmark for success in Chem 101, a slightly greater percentage of students in Chem 464 are achieving the benchmark level than in Chem 101. With future assessments and more data, we would hope to see an even greater percentage of students in one of our capstone courses achieve the benchmark level of performance. 2f. Use of Assessment Results of this SLO: Think about all the different ways the results were or will be used. For example, to recommend changes to course content/topics covered, course sequence, addition/deletion of courses in program, student support services, revisions to program SLO’s, assessment instruments, academic programmatic changes, assessment plan changes, etc. Please provide a clear and detailed description of how the assessment results were or will be used. Based on these results, we have: 1.) Established that in order to improve the percentage of students achieving the benchmark level of success in our assessments, critical thinking and problem-solving skills need be emphasized not only in introductory but also in upper division courses. Since lecture courses are driven by direct information transfer, often the best forum for achieving this emphasis on critical thinking is discussion/recitation sessions or in-class exercises where students work on assigned exercises and practice problems in groups and present their results to the class. 2.) Initiated the process of establishing a required recitation course for Organic Chemistry I. The results of previous assessment activity in our organic courses have demonstrated the strong need for such problem-solving sessions. Optional sessions are currently in place, and once the mandatory session for Organic Chemistry I is established we will re-assess student performance in the organic sequence to evaluate the impact of recitation on student performance. 3.) Archived this data for future comparison. If the mandatory recitation sessions in organic chemistry lead to an improvement in the percentage of students achieving the benchmark level of success on our assessments, then appropriate changes in instructional/teaching methods in other upper division (300 or 400 level) chemistry courses may be brought up for consideration by the faculty. 2. Student Learning Outcome Assessment Project: Answer questions according to the individual SLO assessed this year. If you assessed an additional SLO, report in the next chart below. 2a. Which Student Learning Outcome was measured this year? SLO 2m: Organize and communicate scientific information clearly and concisely, both verbally and in writing. 2b. What assessment instrument(s) were used to measure this SLO? A rubric to evaluate the graduate student seminars was crafted and approved by the department. Beginning spring 2010, this rubric was used by department faculty in evaluating graduate student seminars. 2c. Describe the participants sampled to assess this SLO: discuss sample/participant and population size for this SLO. For example, what type of students, which courses, how decisions were made to include certain participants. The sample included all graduate students giving either a literature seminar (Chem 691) or thesis seminar (Chem 692) for the Fall semester 2010 and Spring/Summer semesters 2011. In Fall 2010 there were 3 literature seminars and 4 thesis seminars. In Spring/Summer 2011 there were 4 thesis seminars and 4 literature seminars. 2d. Describe the assessment design methodology: For example, was this SLO assessed longitudinally (same students at different points) or was a cross-sectional comparison used (comparing freshmen with seniors)? If so, describe the assessment points used. Graduate students giving literature seminars are at (approximately) the half-way mark in their progress towards the master’s degree; they have taken several upper division courses, some or most of which require in-class oral presentations. Graduate students giving their thesis seminar have completed all required graduate coursework as well as experimental work. Thus, it is hoped that the sequential evaluations of literature and thesis seminars for individual students will allow an evaluation of the growth of a student’s oral communication abilities. 2e. Assessment Results & Analysis of this SLO: Provide a summary of how the data were analyzed and highlight important findings from the data collected. The scoring rubrics had five categories: organization, understanding of scientific content, style/delivery, use of visual aids, and ability of answer questions, and performance in each category could be rated with a score of 0-20 points. The rubric provided descriptions for excellent (16-20 points), good (11-15 points), marginal (6-10 points), and inadequate (0-5 points) performance. Faculty attending the seminars filled out the rubrics and forwarded them to the seminar coordinator. The seminar coordinator tabulated the results for each category and an average score for literature seminars and thesis seminars was obtained. Fall 2010 results: organization, average scores 18.3 (lit) and 17 (thesis); understanding of scientific content, average scores 17.3 (lit) and 17 (thesis); style and delivery, average scores: 17.3 (lit) and 16.75 (thesis); use of visual aids, average scores: 17.3 (lit) and 16.75 (thesis); and ability to answer questions, average scores: 15.7 (lit) and 15.75 (thesis). Spring/Summer 2011 results: organization, average scores 18.75 (lit) and 18.75 (thesis); understanding of scientific content, average scores 18.25 (lit) and 18.5 (thesis); style and delivery, average scores: 17.75 (lit) and 18.25 (thesis); use of visual aids, average scores: 17.25 (lit) and 18 (thesis); and ability to answer questions, average scores: 17.25 (lit) and 17 (thesis). The results indicate that, on the whole, the graduate students are doing well in their oral seminars, since the average scores in most categories are in the 17-18 range. The weakest category in all cases is the ability to answer questions, with average scores ranging from 15.7-17.3. In the past this has been attributed to a lack of depth of scientific understanding on the part of the student; however, average scores on the understanding of scientific content category (17-18.5) do not necessarily corroborate this idea. Compared to the data collected in Spring 2010, average scores in literature seminars have improved while average scores in thesis seminars have gone down: Organization, up 0.8 points for lit, down 0.35 for thesis; understanding, up 0.65 for lit, down 1.0 for thesis; style, up 2.0 for lit, did not change for thesis; visuals, up 0.6 for lit, down 0.85 for thesis; questions, up 1.2 for lit, down 0.9 for thesis. It is thought that the reason for this trend is that faculty are now following the scoring rubric more closely than before (vide infra). 2f. Use of Assessment Results of this SLO: Think about all the different ways the results were or will be used. For example, to recommend changes to course content/topics covered, course sequence, addition/deletion of courses in program, student support services, revisions to program SLO’s, assessment instruments, academic programmatic changes, assessment plan changes, etc. Please provide a clear and detailed description of how the assessment results were or will be used. Based on these results, we have: 1.) Advised students to practice their literature and thesis seminars before one or two faculty members who can provide questions and feedback prior to their actual presentation may also improve their ability to answer questions. This suggestion will be given to students in the information form they receive before preparing their seminars and can be reiterated by both the seminar coordinator and supervising faculty. These strategies will be implemented in Spring 2012. 2.) Crafted a new rubric scoring sheet in which faculty need to write in a score for each category (rather than an overall score), requiring them to think more specifically about what to give in each instance. Previously, the seminar coordinator suggested that perhaps the high average scores in each category indicate that faculty are not grading strictly according to the rubric or are unwilling to give poor grades to students on the thesis/literature presentations. It is felt that this recent simple change in the rubric scoring sheet has resulted in faculty scoring more accurately, as reflected in the greater similarity of the average scores for literature and thesis seminars in the 2010-2011 academic year. 2a. Which Student Learning Outcome was measured this year? SLO4b: Work effectively in a laboratory environment, including the ability to maintain a proper lab notebook. 2b. What assessment instrument(s) were used to measure this SLO? A general rubric to evaluate the laboratory notebooks of undergraduate students taking laboratory courses in our program was crafted. The rubric was distributed to all faculty at the beginning of the fall 2010 semester with the intent that individual course coordinators could modify the rubric at will to suit the needs of their individual courses. Categories in the rubric included pre-lab preparation, in-lab qualitative observations, tabulation of data and results, and post-lab reflection/conclusion. A desired benchmark score of 75% has been set for this rubric. Prior to the development of this rubric, instructors in Chem 321L have evaluated the quality of each student’s laboratory notebook for many years. The focus of the 321L notebook evaluation, while covering the essential points of the new rubric above, is on completeness and the evaluation tends to be more detailed and comprehensive. The desired benchmark score for this evaluation has thus been set at 70%. 2c. Describe the participants sampled to assess this SLO: discuss sample/participant and population size for this SLO. For example, what type of students, which courses, how decisions were made to include certain participants. At this initial stage of implementation of the new rubric, only a few instructors have adopted its use and data is available only from Organic Chemistry II lab. Notebooks were collected at the end of Spring 2011 semester from all Chem 334L sections (a total of 162 students) and 23 notebooks of chemistry/biochemistry majors and minors were reviewed by course teaching assistants. All chemistry and biochemistry majors and minors take Chem 321L. Notebook data from four sections were analyzed: Spring 2005 (17 students), Fall 2006 (18 students), Fall 2007 (17 students) and Fall 2009 (15 students). The evaluation is done in two parts, with the first check made during week 6 and the second check done at the end of the semester. 2d. Describe the assessment design methodology: Was this SLO assessed longitudinally (same students at different points) or was a crosssectional comparison used (comparing freshmen with seniors)? If so, describe the assessment points used. Organic chemistry II lab is taken by students after a full semester of Organic Chemistry I lecture and lab. The course is taken approximately halfway through the program for chem/biochem majors, and a significant percentage of students in this course are biology majors. Since by this time the students would have taken two general chemistry labs (Chem 101L and Chem 102L) and one organic chemistry lab (Chem 333L), we feel that assessment at this intermediate point through the program should provide valuable information about how well our students have learned to keep a proper lab notebook, which is an essential skill for practicing scientists. Students taking Chem 321L generally have completed General Chemistry and quite a few have also completed Organic Chemistry. As such these students are approximately mid-way through the program, and thus it may be expected that they have developed some fundamental skills in notebook keeping that may be assessed at this intermediate time point. 2e. Assessment Results & Analysis of this SLO: Provide a summary of how the data were analyzed and highlight important findings from the data collected. For the 334L notebooks reviewed, an average score of 15/20 (75%, the benchmark level) was obtained from rubric data gathered from 4 teaching assistants. Of the five categories the students were evaluated in, the weakest category (with an average of 2.4/4 points) involved recording in-lab qualitative observations. The teaching assistants noted that student mistakes during an experiment (such as solvent/reagent spillages during extractions, overheating/underheating reaction mixtures, etc) were often not recorded in the notebooks. A further weak category (with an average 2.7/4 points) was post-lab reflection/conclusion. Some teaching assistants noted that while observations and conclusions were often omitted altogether, detailed procedures carefully copied from the lab manual were included in notebook entries for each experiment. While the overall notebook-keeping ability of our majors and minors may be judged satisfactory at this intermediate point through the program, students need to concentrate more on the process of recording their data, observations and conclusions in their notebook during lab, rather than on doctored notebook mechanics and cleanliness out of lab. For Chem 321L Spring 2005, 41.2% of students achieved the benchmark score of 70% at check 1 and 29.4% achieved benchmark at check 2. For Fall 2006, 16.7% achieved benchmark at check 1 and 61.1% achieved benchmark at check 2. For Fall 2007, 35.3% achieved benchmark at check 1 and 26.7% achieved benchmark at check 2. Finally, for Fall 2009, 40% achieved benchmark at check 1 and 26.7 % achieved benchmark at check 2. These results suggest the following: 1.) the sometimes large variation in scores seems to reflect the small class size and its varying skill level; 2.) sometimes there is a significant improvement throughout the semester, however, when this does not occur it may reflect student attitude in a particular section; 3.) although the class average occasionally exceeds the benchmark level, improvement is needed in the way students maintain a laboratory notebook in Chem 321L 2f. Use of Assessment Results of this SLO: Think about all the different ways the results were (or could be) used. For example, to recommend changes to course content/topics covered, course sequence, addition/deletion of courses in program, student support services, revisions to program SLO’s, assessment instruments, academic programmatic changes, assessment plan changes, etc. Please provide a clear and detailed description of each. Based on these results, we have: 1.) Established benchmark levels for student success overall (75%) and in each category (3/4) of the new rubric. 2.) Re-emphasized the importance of the in-lab recording of experimental observations in all laboratory courses, starting at the 100 level. Although sample notebook entries are provided in course lab manuals, teaching assistants will be encouraged to make it a point to review what a proper lab notebook entry for an experiment looks like on the first day of class. This will be communicated to course coordinators at the beginning of each semester from now on. 3.) Suggested that TA’s do notebook spot-checks periodically through the semester (with points assigned) in order to remind students of the importance of in-lab qualitative observations. This practice has been in place for a while in the organic lab sequence (Chem 333L and Chem 334L), but assigning a greater number of points to the spot-checks may reflect the importance of notebook maintenance in the overall lab course grade. If successful, this strategy could be employed in other lab courses in our program. 4.) Considered providing Chem 321L students with more detailed examples of a proper lab notebook, as well as making the notebook score a more significant part of the course grade to further emphasize the importance of this skill. 2a. Which Student Learning Outcome was measured this year? SLO6: Perform Quantitative Chemical Analysis 2b. What assessment instrument(s) were used to measure this SLO? The emphasis in Chem 321L is on quantitatively analyzing samples using a variety of wet chemical and instrumental techniques. Students perform 8-9 analyses each semester and receive a score between 5-10 based on how close their numerical result is to the accepted value. The level of accuracy required for a top score depends on the technique(s) used, however, a grade of 10 generally requires a relative error of only a few parts per thousand. The acceptable benchmark performance has been set at a class average of 80% for each experiment. 2c. Describe the participants sampled to assess this SLO: discuss sample/participant and population size for this SLO. For example, what type of students, which courses, how decisions were made to include certain participants. All chemistry and biochemistry majors and minors take Chem 321L. Each section of Chem 321L contains 15-18 students. Data were collected from four sections: Spring 2005 (17 students), Fall 2006 (18 students), Fall 2007 (17 students), and Fall 2009 (15 students) 2d. Describe the assessment design methodology: Was this SLO assessed longitudinally (same students at different points) or was a crosssectional comparison used (comparing freshmen with seniors)? If so, describe the assessment points used. Students taking Chem 321L generally have completed General Chemistry and quite a few have also completed Organic Chemistry. As such these students are approximately mid-way through the program, and thus it may be expected that they have developed some fundamental skills in quantitative chemical analysis that may be assessed at this intermediate time point. 2e. Assessment Results & Analysis of this SLO: Provide a summary of how the data were analyzed and highlight important findings from the data collected. For Spring 2005 average scores on experiments 1-8 ranged from a low of 76 (experiment 5) to a high of 94 (experiment 4). For Fall 2006, average scores on experiments 1-8 ranged from a low of 68 (experiment 5) to a high of 95 (experiment 7). For Fall 2007, average scores on experiments 1-8 ranged from a low of 79 (experiment 5) to a high of 95 (experiments 4 and 7). For Fall 2009, average scores on experiments 1-9 ranged from a low of 81 (experiments 6 and 7) to a high of 93 (experiment 8). It is noteworthy that experiment 5 has historically resulted in much poorer scores with the average generally falling below the 80% benchmark. Overall, however, this history of assessment strongly suggests that the average student in Chem 321L is capable of performing careful quantitative analyses using a variety of techniques 2f. Use of Assessment Results of this SLO: Think about all the different ways the results were (or could be) used. For example, to recommend changes to course content/topics covered, course sequence, addition/deletion of courses in program, student support services, revisions to program SLO’s, assessment instruments, academic programmatic changes, assessment plan changes, etc. Please provide a clear and detailed description of each. Based on the results for Spring 2005, Fall 2006, and Fall 2007, which pointed to a problem with experiment 5 (i.e., student results always fell below the benchmark class average of 80%), we have implemented a new experiment (starting in Fall 2009) that avoids problems related to the equipment used for the previous experiment 5. The results (as can be seen for the Fall 2009 data above) are now more in line with other experiments. This change also provided additional lab time so that a further new experiment (#9) was added in Fall 2009. 3. How do your assessment activities connect with your program’s strategic plan? Department assessment most closely aligns with the plan to improve the quality of teaching in our Department. By identifying areas of weakness for our students with respect to our SLO’s, our full time faculty can make necessary changes to course content and instructional approaches or suggest new preparatory, pre-requisite or co-requisite courses (such as discussion or recitation sessions) to assist students in gaining the necessary foundational knowledge so that they can progress through the core courses in their degree. As mentioned above, we have recently initiated the process of creating a mandatory recitation session for our Organic Chemistry I course, since our assessment results clearly indicated a need for our students to practice problem solving and acquire critical thinking skills. We anticipate that this program modification will greatly improve student learning and allow students to progress through to the upper division courses in a more timely fashion. Furthermore, the assessment activities described in section 5b below help ensure that the teaching activities of our non-tenure track faculty are of a high standard. 4. Overall, if this year’s program assessment evidence indicates that new resources are needed in order to improve and support student learning, please discuss here. Assessment during 2010-2011 did not indicate the need for new resources. 5. Other information, assessment or reflective activities not captured above. a.) One of the members of our departmental assessment committee (Yann Schrodi) has developed a rubric for assessing written assignments in chemistry, relevant to department student learning outcomes 2 and 3. Categories in the new rubric include Abstract/Introduction/Theory, Materials and Methods, Discussion, Conclusions, References, Grammar, and Formatting. It is hoped that the use of this new rubric will be implemented in the Fall 2011 semester, especially for evaluating student research reports in Chem 499 and Chem 495. b.) Several years ago the Department developed and adopted a specific set of guidelines for the evaluation of non-tenure track faculty. The department continues to thoroughly review and provide feedback on the teaching of all non-tenure track faculty who teach lecture sections for the Department. This evaluation consists of a class visit, a comprehensive review of all written instructional materials including syllabi, quizzes, handouts, and tests, and an examination of student evaluations for all lecture sections taught. The evaluation report is discussed with each faculty member under review by a member of the Department Evaluation Committee, and each faculty member is encouraged to share this information with his or her faculty mentor. The Department is in the third year of this annual evaluation process. In most instances the faculty under review have improved in areas of concern noted in their evaluation. This process did identify an instructor who was not re-hired by the Department because of a failure to address concerns that were repeatedly raised. 6. Has someone in your program completed, submitted or published a manuscript which uses or describes assessment activities in your program? Please provide citation or discuss. No