2014-2015 Annual Program Assessment Report Please submit report to your department chair or program coordinator, the Associate Dean of your College, and to james.solomon@csun.edu, director of assessment and program review, by September 30, 2015. You may, but are not required to, submit a separate report for each program, including graduate degree programs, which conducted assessment activities, or you may combine programs in a single report. Please identify your department/program in the file name for your report. College: Science and Mathematics Department: Chemistry and Biochemistry Program: BA/BS/MS Assessment liaison: Thomas Minehan 1. Please check off whichever is applicable: A. ___√_____ Measured student work. B. ____√____ Analyzed results of measurement. C. ________ Applied results of analysis to program review/curriculum/review/revision. 2. Overview of Annual Assessment Project(s). On a separate sheet, provide a brief overview of this year’s assessment activities, including: an explanation for why your department chose the assessment activities (measurement, analysis, and/or application) that it enacted if your department implemented assessment option A, identify which program SLOs were assessed (please identify the SLOs in full), in which classes and/or contexts, what assessment instruments were used and the methodology employed, the resulting scores, and the relation between this year’s measure of student work and that of past years: (include as an appendix any and all relevant materials that you wish to include) if your department implemented assessment option B, identify what conclusions were drawn from the analysis of measured results, what changes to the program were planned in response, and the relation between this year’s analyses and past and future assessment activities if your department implemented option C, identify the program modifications that were adopted, and the relation between program modifications and past and future assessment activities in what way(s) your assessment activities may reflect the university’s commitment to diversity in all its dimensions but especially with respect to underrepresented groups any other assessment-related information you wish to include, including SLO revision (especially to ensure continuing alignment between program course offerings and both program and university student learning outcomes), and/or the creation and modification of new assessment instruments Preview of planned assessment activities for next year. Include a brief description and explanation of how next year’s assessment will contribute to a continuous program of ongoing assessment. 3. 2. Overview of Annual Assessment Project(s). Provide a brief overview of this year’s assessment activities. The following assessment activities took place this year: •Measure Student Work a. Assess basic knowledge in general, analytical, and biochemistry (SLO1) using standardized exam questions in course finals. b. Implemented a signature assignment for longitudinal assessment of knowledge (SLO1) in the gateway (Chem 321) and capstone (Chem 401) courses c. Assess graduate students’ scientific oral communication abilities in literature and thesis seminars, relevant to SLO2m: Organize and communicate scientific information clearly and concisely, both verbally and in writing •Analyze Results of Measurement d. An analysis of student performance trends in general chemistry, analytical chemistry, and biochemistry was undertaken. e. The results of the signature assignment were reviewed and suggestions for changing questions in that assignment have been made. f. Review evidence pertaining to SLO2m: Organize and communicate scientific information clearly and concisely, both verbally and in writing. The department chose these activities so as to 1.) move forward with a longitudinal assessment program, which was previously planned but had yet to be implemented, and 2.) encourage faculty to continue doing assessment in their courses each year so as to identify trends over multiple semesters and weaknesses in student comprehension that need to be addressed at the individual course level and in the program as a whole. Option A: Measure Student Work SLO 1: Assess basic knowledge in the following areas of chemistry: general chemistry, chemistry, biochemistry, and analytical chemistry, both at the individual course level and for the program as a whole. SLO’s addressed: 2 a. 1. Assess basic knowledge in general chemistry (SLO1) using standardized exam questions in course finals. Alignment with core competencies: critical thinking, quantitative literacy. General Chemistry Assessment: Students taking Chem 101 in fall 2014 and spring 2014 were assessed using a 24 question pre- and post-test taken from the course textbook testbank. The assessments were administered either in class via a paper exam or online via Moodle quiz. For fall 2014, 47 students took the pre-test and 28 students took the post-test; with a benchmark level of 66% of questions answered correctly, 15% of students performed at or above benchmark for the pre-test and 54% of students performed at or above benchmark for the post-test. The class average also improved 15% from pre-test to post-test. For spring 2015, 66 students took the pre-test and 55 students took the post test; whereas only 3% of students scored at or above benchmark for the pre-test, 29% of students performed at or above benchmark for the post test. Furthermore, the class average improved from 41% to 60% from pre-test to posttest. For students taking Chem 102, in spring 2015, the 70-question ACS standardized exam in General Chemistry was administered at the end of the semester, and 69 students took the exam. 14% of the students scored 66% or higher, and the class average was 50%, which was below the composite norms average by about 10%. The class average compared with the Composite Norms is at the 31st percentile. In summer 2015, 18 questions from the ACS standardized exam in General Chemistry were administered as a pre- and post test to 51 students. For the pre-test, 2% of the class scored at or above the benchmark (66%); for the post-test, 12% of the students scored at or above the benchmark. The class average on the assessment also improved from 29% to 43%. d. Analysis of the Results of Measurement Our general chemistry courses are doing a good job in improving student comprehension of the foundational concepts of chemistry. For both Chem 101 and Chem 102, students are clearly improving in their understanding of the material as they proceed through the courses, with significant gains evident in both the class average on the assessments and in the number of students achieving the benchmark level of performance. Nonetheless, the results from the ACS standardized exam given at the completion of Chem 102 in the spring semester indicates that there is still much room for improvement; the majority of our students are still performing below the national averages. Although it is not desirable to “teach to the exam”, it may be helpful for general chemistry instructors to review the ACS exam questions and identify general topics which the students seem to have more difficulty with. This activity may result in changing the emphasis on some topics covered in the corresponding lecture course. 3 b. 2. Assess basic knowledge in analytical chemistry (SLO1) using standardized exam questions and perform qualitative and quantitative analysis (SLO6). Alignment with core competencies: critical thinking, quantitative literacy. The analytical chemistry courses Chem 321 and Chem 422 are taken in sequence by all Chemistry BS, Chemistry BA and Biochemistry BS majors. Thus, the majority of students in Chem 422 in Fall 2014 took Chem 321 in Spring 2014. During Fall 2014, the same 5-question assessment poll was given on the first day of class in both Chem 321 and Chem 422; the questions were taken from an ACS standardized exam in analytical chemistry. In Chem 321, of the 30 students polled, 20% of the students obtained the benchmark score of 3/5 or better. In Chem 422, of the 26 students polled, 50% of the students obtained a score of 3/5 or better. d. Analysis of the Results of Measurement The results indicate that students learn the material from Chem 321 and seem to perform better on the ACS questions after having taken the course. Selection should be considered, as the only students taking Chem 422 successfully passed Chem 321 and the students that did not pass Chem 321 were not assessed. The study performed was pseudolongitudinal (the same set of students was not used). However, we plan on tracking the Chem 321 students through Chem 422 in future semesters to provide a fully longitudinal study as students progress through the major. c. 3. Assess basic knowledge in biochemistry (SLO1) using standardized exam questions. Alignment with core competencies: critical thinking, quantitative literacy. Chem 464 is a one-semester course in biochemistry taken primarily by non-majors; Chem 462 is the second-semester of a two-semester biochemistry course (Chem 461 and Chem 462) taken by biochemistry majors. In Spring 2015, 37 students in Chem 464 were administered a 5-question pretest on the first day of class, with the questions taken primarily from the ACS standardized exam in biochemistry; the same 5 questions were also included on the final exam. The average score for the pre-test was 1.67/5, whereas the average for the same questions on the final exam was 3.23/5. For Chem 462, 21 students were administered 10 questions from the ACS standardized exam in biochemistry at the end of the course. 81% of the students in this course achieved the benchmark level of performance (6/10 questions answered correctly) or higher in this assessment. This compared favorably to the results of the same assessment given in previous years: 45% at benchmark or higher for spring 2012 and 64% at benchmark or higher for spring 2012. 4 d. Analysis of the Results of Measurement In Chem 464, it is evident that students are significantly improving in their knowledge of biochemistry from the pretest to the post-test: the class average on the assessment questions almost doubled. In Chem 462, for the 10 ACS questions used for assessment, this was the best student performance with respect to the benchmark level in recent years. One suggestion to allow comparison of our students’ performance with national averages is to administer the entire ACS standardized exam in biochemistry on the last day of class, time permitting. This has been a useful comparison for general chemistry instructors and organic chemistry instructors, allowing them to gauge where our students stand with respect to others assessed across the nation in the same subject area b. Signature Assignment Creation and Administration: twenty multiple-choice questions from all subdisciplines of chemistry (general, organic, inorganic, analytical, physical, and biochemistry) were assembled to create an assignment with the input of the department faculty (see appendix A). Although this assessment has now been implemented into Moodle, for the 2014-2015 academic year an in-class paper exam/scantron was given in Chem 321 (Analytical Chemistry I, sp 2015) and Chem 401 (Inorganic Chemistry, sp 2015). The assignment was administered to 51 students in Chem 321 (31 majors and 20 non-majors) and 42 students in Chem 401 (majors and “unclassified” graduate students who would like to demonstrate proficiency in inorganic chemistry). •Results of assessment: In Chem 321, the average score for the non-majors who completed the assignment was 7.3 correct out of 20. The average score for chemistry and biochemistry majors was 7.9 correct out of 20. For Chem 401, the average score for all of the students taking the exam was 8.4 correct out of 20. While there is a better average performance of the students in Chem 401 than in Chem 321, the difference between Chem 321 majors and Chem 401 students is quite small (0.5 points) and this was an unexpected result. We were able to identify a subset of the questions that the students had the most difficulty with: #2, 4, 11, 17-20. Numbers 17-19 deal with concepts in inorganic chemistry that the students were most recently exposed to. Question number 2 had a typo that may have led to some confusion and thus the high rate of incorrect answers. Questions 4, 11, and 20 deal with fundamental concepts introduced in general and organic chemistry. 5 e. Analysis of the Results of Measurement The small difference in the average performance of Chem majors in Chem 321 (gateway course) and students in Chem 401 (capstone course) is alarming. Since this is the first semester such an assignment has been given, there are no previous data to compare to. In discussion of the results with department faculty members, it was suggested that some of the questions be revised (including the correct/incorrect answer options) to further clarify what is being asked (question #2 and #4) and this will take place before the assessment is implemented again this year. Beyond, this, the results indicate that our students are struggling with concepts not only encountered in upper division courses (questions 17-19), but also with foundational knowledge related to molecular structure, acid-base chemistry, and orbital theory (questions 4, 11, and 20). Faculty have suggested that students may not be motivated to take the assessment as seriously as course exams, since it is not directly impacting their course grade. A solution to this “motivation” problem that has been put forward is to provide “bonus” points to all students who take the exam, and extra bonus points for those performing above a certain benchmark level. If the students see the assessment as positively impacting their grade, they are more likely to put in the effort to do as well as they can. These results are very useful for instructors in our lower division classes (especially Chem 101/Chem 102) and upper division classes in terms of identifying the challenging topics that our students struggle with throughout the program. It is anticipated that, after reflection on these results, our department faculty will direct more attention to these problem areas in their respective courses. d. c. Assess graduate students’ scientific oral communication abilities in literature and thesis seminars, relevant to SLO2m: Organize and communicate scientific information clearly and concisely, both verbally and in writing. Alignment with core competencies: oral communication, information literacy. The oral presentation rubric developed in the department of chemistry and biochemistry (see appendix) was used to assess the literature and thesis seminars given by 12 MS chemistry students in the 2014-2015 academic year. The scoring rubric has five categories: organization, understanding of scientific content, style/delivery, use of visual aids, and ability to answer questions. Performance in each category could be rated with a score of 0-20. The rubric provided descriptions for excellent (16-20 points), good (11-15 points), marginal (6-10 points), and inadequate (0-5 points) performance. Faculty attending the seminar filled out the rubrics and forwarded them to the seminar coordinator. The 6 seminar coordinator then tabulated the results for each category and an average score for literature and thesis seminars was obtained. •Results for 2014-2015: For the five literature seminars, the average score for the category of organization was 17.2/20; the average score for the category of understanding of scientific content was 16.6/20; the average score for the category of style and delivery was 16.3/20; the average score for the category of use of visual aids was 16.6/20; and the average score for the category of ability to answer questions was 16.2/20. The total score for the literature seminars was 82.9/100. For the seven thesis seminars, the average score for the category of organization was 19.0/20; the average score for the category of understanding of scientific content was 17.0/20; the average score for the category of style and delivery was 16.3/20; the average score for the category of use of visual aids was 17.3/20; and the average score for the category of ability to answer questions was 15.7/20. The total score for the thesis seminars was 85.3/100. For comparison, results from the 2013-2014 academic year are also provided: for the nine literature seminars, the average score for the category of organization was 16.8/20; the average score for the category of understanding of scientific content was 15.6/20; the average score for the category of style and delivery was 16.6/20; the average score for the category of use of visual aids was 16.9/20; and the average score for the category of ability to answer questions was 14.4/20. The total score for the literature seminars was 80.3/100. For the seven thesis seminars, the average score for the category of organization was 17.6/20; the average score for the category of understanding of scientific content was 17.1/20; the average score for the category of style and delivery was 16.7/20; the average score for the category of use of visual aids was 17.2/20; and the average score for the category of ability to answer questions was 16.0/20. The total score for the thesis seminars was 84.6/100. f. Analysis of the Results of Measurement As can be seen, the overall scores for both literature and thesis seminars increased during the 2014-2015 academic year as compared to the 2013-2014 academic year. In both years, the thesis seminar grades are higher than the literature seminar grades. The results indicate that, on the whole, graduate students are doing well in their oral seminars, since the average scores in most categories are in the 17-18 range. The weakest category tends to be the ability to answer questions. It should be noted that one student failed the literature seminar each year for the 20132014 and 2014-2015 academic years, and so the scores are fairly good for the literature seminars considering that is the case. 7 Another useful comparison to assess student performance is to determine whether the quality and content of the students’ seminars improved between their literature presentation and thesis seminar. The following is a survey of average results from the 7 students who defended their theses in the 2014-2015 academic year: Organization (literature)=17.4, (thesis)=19.0; understanding of scientific content (literature)=16.0, (thesis)=17.0; style and delivery (literature)=16.5, (thesis)=16.3; use of visual aids (literature)=17.4, (thesis)=17.3; ability to answer questions (literature)=14.4, (thesis)=15.7; total (literature)=81.7, (thesis)=85.3. These data show that the graduate students who presented their thesis seminars over the past year showed definite improvement in the overall quality of their presentation compared to their literature smeinars (which were presented at least one semester prior to their thesis seminar). Interestingly, the “style and delivery” and “use of visual aids” categories showed no significant change. Again, the weakest category is the ability to answer questions. Perhaps a useful suggestion to improve performance in this category is to have the student give their practice presentations before an audience consisting not just of the individual’s research group students and professor, but rather including students and/or professors from other research groups and subdisciplines, so that a broad variety of questions about their literature and thesis topic may be encountered prior to the actual seminar. 3. Preview of planned assessment activities for next year. Include a brief description and explanation of how next year’s assessment will contribute to a continuous program of ongoing assessment. In the next year we plan to continue our assessment of student subject knowledge (SLO1) in individual courses as well as throughout the program through our signature assignment administered in the gateway and capstone courses. For 2015-2016, we will also focus our efforts on assessment of our laboratory courses to evaluate SLO’s 4-6 (4: work effectively and safely in a laboratory environment, including the ability to follow experimental chemical procedures and maintain a proper lab notebook; 5: effectively utilize modern chemical instrumentation to obtain data and perform research; 6: perform qualitative and quantitative chemical analysis). We will be encouraging our lab instructors to utilize our customizable lab notebook rubric for notebook evaluations. We will also be obtaining useful data on student learning from our upper division labs that involve quite a bit of chemical analysis (Chem 321, Chem 422, Chem 411) and the use of chemical instrumentation (Chem 321, Chem 422, Chem 411 and Chem 433). 8 Appendix A: Longitudinal Assessment Assignment, Revised 2015 1. Which best represents a step in the mechanism for the saponification of methyl acetate? 2. A mixture of the following four compounds is dissolved in diethyl ether and shaken with a 2M NaOH solution. Which compound(s) remain in the organic (ether) phase? (A) (B) (C) (D) A, B and D B and C B, C, and D B and D 3. The correct chair conformation of the following structure is: 4. Which of the following molecules are expected to have a net dipole moment? (A) (B) (C) (D) BF3 CCl4 CH2O CH4 Useful Information: Group pKa Arg Sidechain 12.5 Asp Sidechain 3.7 Glu Sidechain 4.3 Lys Sidechain 10.5 His Sidechain 6.0 Pepide Amine Group 8.0 Peptide Carboxylate Group 3.4 5. For the dipeptide shown below (Glu-Ala), which structure represents the most abundant form at physiological pH (pH = 7.4)? (A) (B) (C) (D) 10 6. For the dipeptide in question 1 above, what is the overall charge at pH = 7.4? (A) (B) (C) (D) -1 +1 0 -2 7. You are given a solution containing a mixture of two dipeptides, Glu-Ala and Val-Ala (nonionic structures shown below). You have access to an anion exchange chromatography column, a cation exchange chromatography column, a pH = 7.4 buffer, a pH = 4.0 buffer, a pH = 9.0 buffer, and a chromatography system with a fraction collector. Which purification strategy below would be best for separating the two dipeptides into two different flasks using chromatography? Glu-Ala (A) (B) (C) (D) Val-Ala Dilute the solution containing Glu-Ala and Val-Ala into the pH 4.0 buffer, load it to the anion exchange column. Elute the column first with pH 7.4 buffer collecting one fraction, then with pH 9.0 buffer collecting a second fraction. Dilute the solution containing Glu-Ala and Val-Ala into the pH 9.0 buffer, load it to the anion exchange column. Elute the column first with pH 7.4 buffer collecting one fraction, then with pH 4.0 buffer collecting a second fraction. Dilute the solution containing Glu-Ala and Val-Ala into the pH 4.0 buffer, load it to the cation exchange column. Elute the column first with pH 7.4 buffer collecting one fraction, then with pH 9.0 buffer collecting a second fraction. Dilute the solution containing Glu-Ala and Val-Ala into the pH 9.0 buffer, load it to the cation exchange column. Elute the column first with pH 4.0 buffer collecting one fraction, then with pH 7.4 buffer collecting a second fraction. 11 8. Which of the four dipeptides below would be expected to be the most soluble in aqueous solution at pH = 12.0? Note: for uniformity, the dipeptides are depicted in their nonionic forms, but your answer should be based on the appropriate states of ionization that would be adopted by the ionizable groups at pH = 12.0. (A) (B) (C) (D) 9. You desire quantitative information about a protein sample of unknown concentration and decide to use a dye-binding assay. You obtain the standard curve data shown below. A 10 L aliquot of your protein sample is diluted to 1.0 mL with the appropriate assay buffer and the absorbance measured, giving a visible absorbance of 0.34. What is the protein concentration of the original protein solution? (A) (B) (C) (D) 0.35 mg/mL 35 mg/mL 0.68 mg/mL 33 mg/mL 12 10. Which solution should be mixed with 50.0 mL of 0.050 M HF to make an effective buffer? (A) 50.0 mL of 0.10 M NaOH (B) 25.0 mL of 0.10 M NaOH (C) 50.0 mL of 0.050 M NaOH (D) 25.0 mL of 0.050 M NaOH 11. In the dissociation of a monoprotic weak acid in aqueous solutions, HA, which leads to the assumption that [H+] ≈ [A-]? (A) The autoprotolysis of water is the dominant source of H+ (B) The concentration of the acid is sufficiently high such that the dissociation of the acid is the dominant source of [H+] (C) The concentration of the acid has no effect on the approximation; [H+] is always approximately equal to [A-] 12. The confidence interval for a set of measurements represents for a defined confidence level that (A) The true value lies within a certain range about the mean value (B) The results from two different methods will agree with each other (C) The variances of two different methods will agree with each other (D) An experimental value lies within a certain range about the mean value 13. Which choice is not a requirement of a primary standard solid? (A) Its reactions must be known and stoichiometric (B) It can be dried to remove surface moisture (C) Its purity must be accurately known (D) It must have a low formula weight 14. Glass pH electrodes must be soaked in water prior to conducting measurements in order to (A) Clean the glass surface (B) Allow the internal reference electrode to reach its equilibrium potential (C) Hydrate the glass to allow ion-exchange between the sample solution and the glass surface (D) Adjust the concentration of the internal filling solution with respect to the external sample solution 15. The addition of traces of antimony to crystalline silicon produces a material having (A) (B) (C) (D) (E) a lower conductivity than silicon. a higher conductivity than silicon. no conductivity superconductivity metallic conductivity 13 16. The bonding interactions in SmF3 are stronger than the bonding interactions in SmI3. Therefore, one can conclude that I. Sm3+ is a soft acid. II. Sm2O3 will be more stable than Sm2S3. III. Sm3+ is a hard acid. (A) (B) (C) (D) only I only III I and II II and III 17. What substitution mechanism is most common for square planar complexes? (A) (B) (C) (D) associative dissociative migratory insertion reductive elimination 18. In a dissociative reaction, how does an increase in the nucleophilicity of the incoming ligand affect the rate of the reaction? (A) (B) (C) (D) The rate of the reaction increases The rate of the reaction decreases The rate of the reaction is unchanged The change in the rate of the reaction depends on the oxidation state of the metal. 19. Square planar complexes containing metal atoms in a low oxidation state typically undergo (A) (B) (C) (D) ligand dissociation migratory insertion oxidative addition reductive elimination 20. How does “negative overlap” in Molecular Orbital Theory lead to “antibonding” between atoms? (A) (B) (C) (D) (E) The opposing charges of two orbitals cancel each other out. The opposing phases of two orbitals cancel each other out. The like charges of two orbitals repel each other. The concentration of electron density between nuclei pushes them apart. The negative phases of two orbitals repel each other. 14 Appendix B: Chemistry and Biochemistry Oral Presentation Rubric Scoring Rubric for Oral Scientific Presentations Level of Achievement Excellent 16-20 points Good 11-15 points Marginal 6-10 points Inadequate 0-5 points Organization • Well thought out with logical progression • Use of proper language • Significance clearly stated • Content level appropriate for audience • Abstract and bibliography are well constructed • Talk easy to follow • Use of proper language • Significance clearly stated • Content level not always appropriate • Abstract and/or bibliography have some errors • Talk somewhat disorganized • Shows some effort to use proper language • Significance somewhat unclear • Includes some irrelevant content and inappropriate content level • Abstract and bibliography are not well constructed • Talk difficult to follow • Unclear language • Does not understand significance of work • Inadequate content • Abstract and bibliography lack proper content and construction Understanding of Scientific Content • Identifies the research question/research field • Has advanced understanding of the experimental approach and significance • Critically evaluates results, methodology and conclusions • Scientifically rigorous and well researched • Identifies the research question/research field • Has basic understanding of the experimental approach and significance • Limited evaluation of results, methodology and conclusions • Well researched • Research question/research field somewhat unclear • Description of experimental approach somewhat confusing • Results and conclusions stated but not critically evaluated • Does not integrate outside readings • Does not understand the research • Does not understand the experimental approach • Does not understand conclusions or recognize implications for future work Style/Delivery • Uses time wisely • Speaks with good pacing and enthusiasm • Makes eye contact and does not read information • Uses engaging tone and appropriate vocabulary • Speaks well, but often repeats comments • Exhibits few disfluencies (“ahs”, “uhms”, etc.) • Makes eye contact • Uses good vocabulary and tone • Presentation poorly timed • Some hesitation and uncertainty are apparent • Exhibits many disfluencies • Makes little eye contact and looks at notes • Monotone and nonengaging delivery • Presentation poorly timed • Makes no eye contact and reads from notes • Hesitation and uncertainty are very apparent • Speaks too quietly or quickly for audience to hear and understand Use of Visual Aids • Tables/graphs summarize data and/or conclusions • Size and labels are clear • Very little text • Figures and images explained and described well • Presentation has no misspellings or grammatical errors • Makes limited and effective use of laser pointer • AV set up properly • Text appropriately sized • Very little text • Most figures and images explained and described well • Presentation has an occasional misspelling or grammatical error • Uses laser pointer effectively • AV set up properly • Labels and legends somewhat unclear • Text size somewhat small • Too much detail on slides • Blocks of text on slides • Figures are explained • Presentation has multiple misspellings and/or grammatical errors • Uses laser pointer unnecessarily • AV mishaps resolved • Labeling is not clear • Size is too small to see • No logical placement of information • Mostly text and very few images • Figures are not explained • Presentation has numerous misspellings and/or grammatical errors • Use of laser pointer is distracting • AV mishaps unresolved Ability to Answer Questions • Anticipates audience questions • Understands audience questions • Can integrate knowledge to answer questions • Thoroughly responds to questions • Does not anticipate audience questions • Understands audience questions • Can integrate knowledge to answer questions • Thoroughly responds to most questions • Does not anticipate audience questions • Makes an effort to address question • Can address some questions • Often responds poorly to questions • Either makes no effort to respond to questions or does so poorly