Program Assessment Report PROGRAM INFORMATION Degree Program(s): Department Chair: Department: B.A. Political Science James Brent Phone: 4-5550 Report Prepared by: Ken Nuger Phone: 4-5346 Next Self-Study due: 2017 E-mail: ken.nuger@sjsu.edu Note: Schedule is posted at: http://www.sjsu.edu/ugs/programplanning/ ARCHIVAL INFORMATION Location: Clark Hall #471 Person to Contact: (Bldg/Room #) James Brent or Ken Nuger 4.5550 4.5346 (Name) (Phone) Does the information (e.g., Mission, Goals, and/or Learning Outcomes) posted on the web (see, http://www.sjsu.edu/ugs/assessment/programs/ ) for this program need to be updated? No If yes, please submit changes to jacqueline.snell@sjsu.edu SCHEDULE OF ASSESSMENT ACTIVITIES* Please complete the schedule of assessment activities below by listing all program Student Learning Outcomes (SLOs) by number down the left column. . When will data be collected? (must be Semester the SLO is focus of efforts (no at least once each program planning PP cycle, each semester put this cycle) semester by the appropriate SLO) SLOs 1 Breadth, ongoing Spring, 2011 2 Application, ongoing 3 Disciplinary Methods, ongoing 4 Communication Skills, ongoing Spring, 2010 5 Citizenship, ongoing Page 1 of 10 Program Assessment Report Learning Objective 4, Communication Skills: Written Communication, Spring and Fall, 2009 BE SURE to enter the semester. This will become a progressive document to which you add information each semester. Initial Evidence of Student Learning: The current political science department five-year program review is being finalized for submission in the spring, 2010 and therefore, will begin a new program review period. We are mindful of the recent suggestions and pleased with this new, simplified form provided by Undergraduate Studies to construct our annual program assessment summary. Based on the assessment guidelines for Spring 2009 to Spring 2010, a memorandum to the Chairs and Assessment Coordinators from Jackie Snell dated February 24, 2009 as well as a College of Social Sciences Assessment meeting conducted by Bob Cooper, in his capacity as director of assessment for COSS and Ron Rogers, in his capacity as director of Assessment for Undergraduate Studies, our department thought it prudent to accept WASC's recommendation to assess written communication skills, which is our department's fourth learning objective, for our Spring 2010 report. We last visited this learning objective in 2006. As we are all aware, both from department, college and university level discussion, as well communications from WASC, having our students write well remains a serious challenge. Since 2006, our department has had ongoing discussions about how to try and improve our students' writing skills. The following represents a summary of our efforts. In our department's annual program assessment report dated October 1, 2006, our focus was written communication. Using the rubric we developed at that time, we evaluated the written communication skills of our seniors using the final research papers they submitted in their senior seminar, a culminating and rigorous requirement for all of our graduating seniors. The rubric is attached at the end of this report. At that time, our department's representatives on our curriculum committee read all 14 senior seminar papers and using our rubric, averaged the composite evaluation of the papers assessed by the committee. On a five point scale, five representing competence well beyond normal expectations for baccalaureate students in political science, to one, where a student does not meet normal expectations, the first writing assessment of our graduating seniors seminar papers were averaged as follows: 1. Student's work is free of technical errors: 2.47 2. Student clearly identifies and explains concepts related to politics, public administration, public policy and law: 2.62 3. Student logically and clearly develops the thesis, arguments and/or theme of the written assignment: 2.38 4. Student correctly uses an acceptable attribution style, including footnotes, or parenthetical references, as well as a complete bibliography or works cited section: 2.64 5. Composite assessment: 2.53 To summarize our first attempt to assess our students' writing abilities in 2006, our senior seminar students fell between 3, student fully meets expectations for baccalaureate students in political science and 2, student generally meets but in some respects, falls short of expectations for baccalaureate students in political science. Page 2 of 10 Program Assessment Report Change(s) to Curriculum or Pedagogy: Our initial efforts led us to conclude that most of our students' written communication skills were, for the most part, barely adequate for a graduating senior in political science. We had hoped for evidence of stronger written communication skills and since that initial assessment, our department has had ongoing discussions about what we could reasonably do to help students improve their writing skills. While not mandating changes with formal departmental policy, we started to be more vigilant in advising students to take 100w in their junior year and we attempted to more vigorously monitor and supervise student writing and encourage students to utilize the various writing support services offered at San Jose State. As of this spring, 2010 semester, our department implemented a modest policy requiring a statement in all syllabi about the importance of correct writing. While we are not requiring that everyone utilize the same wording, we did create a model statement that would serve as a guideline for faculty to construct their own commitment to correct writing statements for their syllabi. The model statement is attached in the appendix. Similarly, the department created a one-page handout, identifying the Writing Center, the Peer Mentor Center and the Learning Assistance Resource Center as valuable on site resources students can utilize to improve their writing. In addition, the handout identifies Owl, the Purdue writing Lab, A Research Guide and the University of Wisconsin Writing Guide as three online resources we believe will help students better understand what is expected of university level scholarly writing and offer opportunities students can utilize to improve their writing skills. While these changes are too new to currently assess, it is our hope that by having the department and faculty continue to more openly discuss the importance of correct writing, we will make students more aware early in their academic careers that writing skills are important to have and important to practice. We hoped that our focus on student writing would help similarly make students focus more attention to writing well. Assessing our current students’ writing abilities affords us an opportunity, as limited as it may be, to make a judgment about whether our informal efforts are possibly helping our students’ writing skills. Evidence of Student Learning after Change. Our department continues to use our senior seminar final research papers as a means to assess our department's five learning objectives. Per WASC and the university's recommendation, and using the same writing rubric we used in 2006, we revisited our fourth learning objective, written communication skills by assessing senior seminar papers from the spring and fall, 2009 semesters. The following summarizes the curriculum committee’s efforts and findings. This year, the committee was composed of six faculty members including the department chair. The six members were collapsed into three groups of two, the goal being that each group of two faculty would independently read the same eight senior seminar papers which we hoped would address data reliability issues. We hoped to evaluate 24 papers from two semesters of senior seminars, eight papers per group and then take the average scores from each group for each of the four writing criteria in our rubric. We fell short of that number, but ultimately were able to use 13 senior seminar papers that were independently evaluated by two of the three groups. One group evaluated seven papers, the eighth being an essay exam and not a paper; another group evaluated six papers, with some confusion about which papers that group had to read that reduced the number of papers evaluated by that group to six. Unfortunately the third group, consisting of the chair and another committee member was supposed to also independently read a group of eight papers but from the chair's perspective, there was overwhelming evidence to suggest his group mate did not carefully read the papers, if at all, so at the discretion of the chair, the average from that group was omitted from our initial evaluation. The following represents the average scores of 13 of 19 senior seminar papers from the fall, 2009 semester. 1. Student work is free of technical errors: 3.57 2. Student clearly identifies and explains concepts related to politics, public administration, public policy and law: 4.11 Page 3 of 10 Program Assessment Report 3. Student logically and clearly develops the thesis, arguments and/or theme of the written assignment: 3.8 4. Student correctly uses an acceptable attribution style, including footnotes, or parenthetical references, as well as a complete bibliography or works cited section: 3.65 5. Composite assessment average: 3.78 If we include the scores of two papers read by only one committee member in one group and the scores from eight papers evaluated by the chair from the department’s spring, 2009 senior seminar, the averages for our four written communication goals are adjusted as follows: 1. Student work is free of technical errors: 3.5 2. Student clearly identifies and explains concepts related to politics, public administration, public policy and law: 3.88 3. Student logically and clearly develops the thesis, arguments and/or theme of the written assignment: 3.52 4. Student correctly uses an acceptable attribution style, including footnotes, or parenthetical references, as well as a complete bibliography or works cited section: 3.47 5. Composite assessment average: 3.59 Our committee was encouraged with these averages as they represent a significant increase from the aggregate written communication scores from senior seminar papers evaluated in 2006. Fully recognizing the methodological weaknesses raised by not using the first group of seniors evaluated, and instead, making a judgment about our efforts to improve our students’ written communication skills from a different group of seniors, possibly with different writing abilities, our committee was still encouraged with the new data. At the very least, we are able to say that our sample of senior seminar papers generally meet and in some areas exceed our department’s expectations for a baccalaureate degree in political science. We believe that our faculty’s modest efforts to identify students who need help with their writing, coupled with our recent efforts, stated above, to stress the need for polished writing has benefitted our students and will inspire them to continually work at improving their writing skills. As our annual reviews proceed and our focus turns to other department program objectives, the department will continue to spot monitor our students' writing skills. Of course we shall continue to carefully monitor our students writing skills as they progress through our program but recent evidence suggests that our students are literate and possess effective writing skills. Page 4 of 10 Program Assessment Report Appendices Appendix 1, Writing Rubric Rubric for assessing written communication skills of our graduating majors The following criteria shall be used to assess each identified written communication skill identified below. 5 = Student demonstrates competence well beyond normal expectations for baccalaureate students in political science 4 = Student fully meets and in some regards exceeds expectations for baccalaureate students in political science 3 = Student fully meets expectations for baccalaureate students in political science. 2 = Student generally meets but in some respects falls short of expectations for baccalaureate students in political science. 1 = Student does not meet normal expectations for baccalaureate students in political science. Written communication skills 1. The student's written work is free of technical writing errors. 2. The student clearly identifies and explains concepts related to politics, public administration, public policy or law. 3. The student logically and clearly develops the thesis, argument and/or theme of the written assignment 4. The student correctly uses an acceptable attribution style, including endnotes, footnotes or parenthetical references, as well as a complete bibliography or works cited section. Page 5 of 10 Program Assessment Report Appendix Two, Writing handout Department of Political Science Writing assistance resources Do you need assistance with your written assignments? The university provides excellent resources to help you write high quality papers. You can also find some excellent resources online. We hope you will find this list helpful. At San Jose State: 1. The Peer Mentor Center, first floor Clark Hall The peer mentor center is staffed with expertly trained undergraduate students, whose job is to assist students in many different capacities, including writing assistance. You can drop in and arrange to meet with one of the many peer mentors on duty. You can find out at: http://www.sjsu.edu/muse/peermentor/. 2. Writing Center, Clark Hall, Suite #126 The writing center will work with students in a one on one setting to help students in all phases of writing. You can find out more at: http://www.sjsu.edu/writingcenter/. 3. Learning Assistance Resource Center, Student Services Center #600 The staff here will be happy to work with you and help you with your assignments. You can find out more at: http://www.sjsu.edu/larc/resources/. In addition to the excellent resources on campus, there are several academic websites that have developed excellent online writing assistance programs. Three of the many well regarded of these online cites are: 1. Owl, The Purdue University writing lab: http://owl.english.purdue.edu/. 2. A research guide for students: http://www.aresearchguide.com/index.html. 3. University of Wisconsin writing guide: http://writing.wisc.edu/Handbook/index.html. In addition, reading a high quality newspaper on a regular basis will expose you to good, if not always great writing and is highly recommended, especially for a political science major. The Wall Street Journal, New York Times, Washington Post and realistically, any major sized city newspaper read on a regular basis will, in the long run, help your writing skills. Page 6 of 10 Program Assessment Report Appendix Three, model writing statement for course syllabi: Students of political science should develop the ability to write in clear, grammatical English. Spelling and grammar count! The following guidelines and resources will help students avoid common mistakes in spelling and grammar and provide useful information to help you craft university level scholarship. Students must ensure that appropriate citations are used. Direct quotations must be so indicated with quotation marks and a specific reference to the page in the source from which it was taken. Ideas from others must also be referenced although quotations marks are inappropriate. Failure to cite your sources constitutes academic misconduct which carries with it serious sanctions. A tutorial on citations is available at the library website at http://www.sjlibrary.org/services/literacy/info_comp/citing.htm. For further writing assistance: The SJSU Writing Center is staffed by professional instructors and upper-division or graduatelevel writing specialists from each of the several SJSU colleges. Writing specialists have met a rigorous GPA requirement, and they are well trained to assist all students at all levels within all disciplines to become better writers. It is your responsibility to arrange a time to meet with the instructors. Consult with your professor in advance to determine whether you need assistance with your writing. Arguably the best way to improve your own writing is to read quality material on a regular basis. To that end, students are encouraged to read articles from scholarly sources but also newspapers such as The Wall Street Journal, The Financial Times, The New York Times, and The Economist. These publications also contain articles of direct relevance to their political science classes. Page 7 of 10 Program Assessment Report Learning Objective One, Breadth, Spring and Fall, 2010, for March, 2011 annual report Initial Evidence of Student Learning: For Spring and Fall, 2010, our department focused on assessing our breadth learning objective, which essentially attempts to determine how well our graduating seniors learned the substance of our major. Additionally, we spot checked 12 senior seminar papers to compare our newest graduating seniors' writing skills for the spring and fall, 2010 semesters with those students submitting their papers in 2009. To assess our students' mastery of our first learning objective, breadth, our department has formulated a senior seminar exit exam that utilizes 60 multiple choice questions divided into our four subdisciplines, American Politics, Comparative Politics, International Relations and Political Theory. While we are troubled with how best to accurately assess how much our students learned, we believe our senior seminar exit exam is a good first step toward uniformly judging our students' overall mastery of our material. Our concern primarily is how we can reliably gauge how well our students demonstrate mastery of our broad curriculum when many of the courses are often taught by different professors. Of course, there is much material that should be covered by any professor teaching a particular subject and we attempted to draft exam questions that would reflect the core curricula in each of our four subfields. It is naïve, however, to assume that students learn the same material regardless of the professor teaching the seminar. Also, while most of our focus is on upper division coursework, it is similarly naïve to assume that transfer students will get the same background training as our students who enter our program as frosh. Finally, it is difficult to determine if better or worse exam scores are a result of our efforts as faculty, or the natural variations in talent from different, relatively small groups of seniors from different semesters. Those caveats stated, we have diligently worked on devising a senior seminar exit exam that would assess how well our students appear to master the core substance of political science. The exam has been used since 2008 to see if our students' mastery is either similar from one semester to the next and preferably, to see if our students do better one year to the next. Our department offered two senior seminars in the spring, 2010 and one in the fall, 2010. Regrettably, we are unable to supply any data from Professor Harris’ senior seminar offered last spring because he was unable to properly administer our 75 question multiple choice exam. In his seminar, students provided most of the demographic data in the exam's first 15 questions, but left unanswered almost all of the remaining 60 questions covering four sub disciplines. That left our department with data from Professor Quill's senior seminar for the spring, 2010 semester and Professor Haas' senior seminar for the fall, 2010 semester. The exam itself explores student responses categorized multiple ways. We attempt to discern our students' broad knowledge of our discipline from several perspectives. We compare transfers with non transfer students, as well as compare scores based on a variety of student characteristics including gender, race, age, part and full time student, average hours of work per week, where core political science courses were completed, sub discipline specialty, to name some but not all of the variables examined. This year, we compared our senior seminar exit exam results with those of 2008. For the purpose of this annual assessment report, one measure, our students' mean score on the exam, is summarized. There were 78 students examined in 2008 and 41 students examined in 2010, not including the 20 plus students from Professor Harris' seminar whose scores were unusable. Page 8 of 10 Program Assessment Report The overall mean score for 2008 was 52% and 56% in 2010. While these mean scores seem low to us, justifying a discussion of what to make of these scores, we do note an improvement from 2008. The median score was 55% in 2008 and 57% in 2010. Within our four sub disciplines, students had a mean score of 61% in American Politics, 50% in Comparative Politics, 65% in International Relations and 50% in Political Theory. The corresponding mean scores in these sub disciplines in 2008 were, 61%, 44%, 59% and 47% respectively. Our committee does not yet know what to make of what seems to us to be relatively low scores across all sub disciplines. Should we be alarmed? We simply do not know yet. We were cautiously optimistic with the improvement from two years ago and will monitor future administrations of this exam for a hoped for upward trend. Also, while we are chagrined at the low score, we are confident that our senior seminar exam is a good instrument to use to assess breadth of political science knowledge. The exam was carefully written by our faculty to include the core curriculum within each of our four sub disciplines. Using this test every year, at least for a few years, will enable us to compare each year's group of graduating seniors using the same criteria. After a few administrations, we believe we will be in a better position to understand what modifications to our curriculum and/or our pedagogy, if any, may be warranted. In sum, using this senior seminar exit exam to assess our breadth program learning objective has piqued our curiosity and has started serious department discussions about how well our students broadly know the discipline of political science. In time, we hope the data will bring some clarity to our understanding of this learning objective and that we may be in a better position to know what to do differently to increase our majors' broad knowledge of the discipline. As a follow up to last year's report, our curriculum committee spot read 12 senior seminar papers from the Professor's Quill's, Harris' and Haas' seminars. Contrary to university expectations, Professor Harris chooses not to use his university assigned email and therefore is unable to receive electronic copies of his students' culminating papers. In attempt to minimize the damage to our assessment efforts brought on from Professor Harris' defiant choice to not appropriately conduct his university responsibilities, our committee solicited his students to send us their papers. Through their benevolence, we were able to collect over half of the papers in Professor Harris' seminar. Both Professor Quill and Haas collected all of their students’ papers in their respective seminars. The committee read four papers randomly from each of the three senior seminar sections. Of the dozen spot checked, the results are as follows: 1. The student's written work is free of technical writing errors. 3.58 2. The student clearly identifies and explains concepts related to politics, public administration, public policy or law. 3.16 3. The student logically and clearly develops the thesis, argument and/or theme of the written assignment. 3.416 4. The student correctly uses an acceptable attribution style, including endnotes, footnotes or parenthetical references, as well as a complete bibliography or works cited section. 3.25 5. Composite average, 3.375 With the exception of question #2, which was significantly lower this year than last, the scores are very similar to our average scores last year and all at above basic competence for graduating seniors. Our department feels confident that our students, at least as graduating seniors, are meeting our demands for basic writing competency in political science. Change(s) to Curriculum or Pedagogy: Page 9 of 10 Program Assessment Report With regard to written communication, it is increasingly evident that standards for seminar papers vary greatly from professor to professor. Of particular concern is the variation our students have in question #2 in which we assess how well our students demonstrate how well they can identify and explain concepts related to politics, public administration, public policy or law. Perhaps this is in part due to how different the demands are of our students by their professor for writing a culminating senior seminar paper. We will discuss this issue and attempt to resolve it. However, it is not our department's mission to micro manage how faculty teach their courses. Still, for the foreseeable future, the department shall initiate discussions on how we may be able to minimize the variation of expectations for research papers from different faculty teaching senior seminar. Within the next few weeks, our goal is to draft specific policy guidelines for culminating senior seminar papers and also develop standards for the prospectus assignment in our 195 methods class that we will use next year to evaluate next year's learning objective, application skills. Evidence of Student Learning after Change As explained earlier, it is our department's collective view that significant changes are in order only after we are clearly convinced that any change is warranted and meaningful. Therefore, we are consciously attempting to create continuity in our efforts for this program assessment cycle with the ultimate goal of considering changes after we are more confident where changes might be needed. To merely react to small variations from year to year in our assessment efforts seems ill advised. For the moment, however, one issue we are becoming more aware of is the different standards imposed by different professors for their students culminating research papers. We will note any modifications we may make to try to ensure that regardless of which professor a student takes for senior seminar, the standards for their culminating papers shall be roughly equivalent. Final observation for this 2011 report The committee wishes to note that in the last year or two, our department's assessment efforts, which is overseen by a five person curriculum committee who actively involve other faculty who, in turn, are becoming increasingly aware of and used to our assessment efforts, appear to be slowly building a department wide culture that is at the very least, amenable to assessment. At the assessment workshop on February 11, one of the topics addressed was how to get the broad faculty to "buy into assessment." By spreading the various assessment responsibilities to many faculty, and then funneling the data from their efforts to the curriculum committee so it may analyze their efforts and prepare these reports, most faculty in our department are indeed buying into the concept. Most of the tenured/tenure track faculty are actively involved in some aspect of our overall assessment efforts and increasingly, our adjunct faculty are becoming more aware of and participating in our assessment responsibilities. While there are certainly challenges still to come, our department is confident that we are building a long range plan that will allow us to reflect on how well we are teaching our students what we want them to learn and over the course of a few cycles, make adjustments to our curricula and pedagogy. Page 10 of 10