Bachelor of Arts, Department of Journalism and Public Communications Academic Assessment Plan Adopted by The Journalism and Public Communications faculty: October 2010, revised spring 2013 to the Office of Academic Affairs: June 14, 2013 for review by The Academic Assessment Committee of the Faculty Senate Bachelor of Arts, Department of Journalism and Public Communications Page 1 of 32 TABLE OF CONTENTS Mission Statement __________________________________________________________________ 4 Program Introduction _______________________________________________________________ 4 Assessment Process Introduction ______________________________________________________ 5 Student Learning Outcomes ___________________________________ Error! Bookmark not defined. Table 1: Association of Assessment Measures to Student Learning Outcomes __________________ 7 Assessment Measures _______________________________________________________________ 9 Table 2: Assessment Measures and Administration _______________________________________ 9 Assessment Process ________________________________________________________________ 11 General Implementation Strategy ________________________________________________________________ 11 The Department of Journalism and Public Communications, which had 160 majors in fall 2011, will continue to diligently collect data to satisfy university mandates and Standard Nine of Accrediting Council of Education in Journalism and Mass Communications. _________________________________________________________ 11 In the past year, the Department has started to rely more on its Community Advisory Council about preparation for careers in journalism and public communications. The department has also relied on numerous outcomes assessment tools, some of which are studier and more meaningful than others. JPC faculty also meet regularly to discuss curriculum and assessment. The standard university instructor evaluation forms completed by students in every course at semester’s end can also be valuable feedback. _______________________________________ 11 Other ongoing assessment measures include writing and reporting assignments and products to demonstrate mastery including media production of campus and community events such as the Great Alaska Shootout, UAA Ice Hockey, news programming for broadcast by campus television and campus radio KRUA, True North magazine, department newsletters, news articles for publication by The Northern Light, digital media for service and professional organizations, and strategic communications campaigns. ______________________________ 11 The department also tracks outcomes through student-produced multimedia projects; a mentorship program through the Alaska chapter of the Public Relations Society of America; and outcome of competition in local, regional and national awards programs, such as the Hearst Foundation Awards Journalism Awards Competition and the Alaska Press Club annual competition. ___________________________________________________ 11 Description of Faculty Involvement ______________________________________________________________ 11 Modification of the Assessment Plan _____________________________________________________________ 12 Appendix A: Alumni Surveys _______________________________________________________ 13 Measure Description: _________________________________________________________________________ Factors that affect the collected data: _____________________________________________________________ How to interpret the data: ______________________________________________________________________ Factors that affect the collected data: _____________________________________________________________ How to interpret the data: ______________________________________________________________________ Pertinent questions from Alumni Survey: __________________________________________________________ 13 13 13 14 14 14 Appendix B: JPC A202 Pre- and Post-test _____________________________________________ 15 Measure Description: _________________________________________________________________________ 15 Factors that affect the collected data: _____________________________________________________________ 15 How to interpret the data: ______________________________________________________________________ 15 Appendix C: JPC A201 Pre- and Post-test _____________________________________________ 17 Measure Description: _________________________________________________________________________ 17 Factors that affect the collected data: _____________________________________________________________ 17 How to interpret the data: ______________________________________________________________________ 17 Appendix D: JPC A101 Pre- and Post-test _____________________________________________ 19 Measure Description: _________________________________________________________________________ 19 Bachelor of Arts, Department of Journalism and Public Communications Page 2 of 32 Factors that affect the collected data: _____________________________________________________________ 19 How to interpret the data: ______________________________________________________________________ 19 Appendix E: Internship Supervisor Surveys____________________________________________ 22 Measure Description: _________________________________________________________________________ 22 Factors that affect the collected data: _____________________________________________________________ 22 How to interpret the data: ______________________________________________________________________ 22 Appendix F: Awards Won by JPC Students ____________________________________________ 24 Measure Description: _________________________________________________________________________ 24 Factors that affect the collected data: _____________________________________________________________ 24 How to interpret the data: ______________________________________________________________________ 24 Appendix G: JPC Integrative capstone course, JPC A403 ________________________________ 26 Measure Description: _________________________________________________________________________ 26 Factors that affect the collected data: _____________________________________________________________ 26 How to interpret the data: ______________________________________________________________________ 26 Appendix H: Senior Exit Survey _____________________________________________________ 28 Measure Description: _________________________________________________________________________ 28 Factors that affect the collected data: _____________________________________________________________ 28 How to interpret the data: ______________________________________________________________________ 28 Appendix I: Senior Portfolios _______________________________________________________ 30 Measure Description: _________________________________________________________________________ 30 Factors that affect the collected data: _____________________________________________________________ 30 How to interpret the data: ______________________________________________________________________ 30 Appendix J: Writing Rubric ________________________________________________________ 31 Measure Description: _________________________________________________________________________ 31 Factors that affect the collected data: _____________________________________________________________ 31 How to interpret the data: ______________________________________________________________________ 31 Bachelor of Arts, Department of Journalism and Public Communications Page 3 of 32 MISSION STATEMENT The mission of the Department of Journalism and Public Communications is to prepare students for professional careers and graduate study while conveying an understanding and appreciation of the vital role that free expression and mass communication play in a global society. Through research, creative activities, and community engagement, we enrich our community and our profession. The department also strives to meet Alaska’s communication needs. Our goal is to contribute to the development of the economic and social environment of the state, with an emphasis on the Southcentral region served by the University of Alaska Anchorage. The department aims to foster in its students a strong resolve to make the flow of news and information more accurate, informative, complete, fair, and ethical. To accomplish this goal, the department seeks to teach students theory, skills, and ethical principles of journalism and professional communications that will endure as fundamentals in a world where the methods of conveying information are undergoing significant change. PROGRAM INTRODUCTION The Department of Journalism and Public Communications is nationally accredited by the Accrediting Council on Education in Journalism and Mass Communications (ACEJMC). The department places great emphasis on preparing graduates for careers in professional communications and media industries. The department was first accredited in 1990. It was reaccredited in 1996, 2002, and 2008 and will be up for reaccreditation in 2013-2014. It is one of more than 100 national and international programs accredited by ACEJMC. ACEJMC accreditation is voluntary and reflects a commitment to ACEJMC’s standards for instruction, service, and faculty research/scholarship/creative activity. The student learning objectives of the Department of Journalism and Public Communications are mapped to ACEJMC core values and competencies listed under its curriculum and instruction standard (Standard 2). JPC expects that its graduates will: 1. Understand and apply the principles and laws of freedom of speech and press for the country in which the institution that invites ACEJMC is located, as well as receive instruction in and understand the range of systems of freedom of expression around the world, including the right to dissent, to monitor and criticize power, and to assemble and petition for redress of grievances; 2. Demonstrate an understanding of the history and role of professionals and institutions in shaping communications; 3. Demonstrate an understanding of gender, race, ethnicity, sexual orientation and, as appropriate, other forms of diversity in domestic society in relation to mass communications; 4. Demonstrate an understanding of the diversity of peoples and cultures and of the significance and impact of mass communications in a global society; 5. Understand concepts and apply theories in the use and presentation of images and information; Bachelor of Arts, Department of Journalism and Public Communications Page 4 of 32 6. Demonstrate an understanding of professional ethical principles and work ethically in pursuit of truth, accuracy, fairness and diversity; 7. Think critically, creatively and independently; 8. Conduct research and evaluate information by methods appropriate to the communications professions in which they work; 9. Write correctly and clearly in forms and styles appropriate for the communications professions, audiences and purposes they serve; 10. Critically evaluate their own work and that of others for accuracy and fairness, clarity, appropriate style and grammatical correctness; 11. Apply basic numerical and statistical concepts. 12. Apply current tools and technologies appropriate for the communications professions in which they work, and to understand the digital world. ASSESSMENT PROCESS INTRODUCTION This document explains educational objectives and expected outcomes for the Department of Journalism and Public Communications (JPC) and outlines a plan for assessing the achievement of our objectives and outcomes. This plan was originally developed at the request of the university, which was directed by the Board of Regents and the Northwest Association of Schools and Colleges (NWASC)1 to develop a means for demonstrating the effectiveness of programs. This plan relies in part on objectives, outcomes, standards and assessments contained in JPC’s most recent self-study report from 2007-2008,2 prepared for accreditation by the Accrediting Council on Education in Journalism and Mass Communications (ACEJMC). A new self-study is in draft form as of this writing. This plan incorporates accrediting standards adopted by the ACEJMC Committee on Standards and Assessment, which were adopted in September 2012 and will be implemented in September 2013. These standards require that an accredited unit such as JPC “regularly evaluates instruction, using multiple measures that include student input.”3 The development of this plan also consisted of: Discussions during regular faculty meetings. 1 UAA Academic Assessment Policies and Procedures draft. March 31, 2011, page 1. Self-study report: Department of Journalism and Public Communications, University of Alaska Anchorage, 2007-2008. 3 Accrediting Council on Education in Journalism and Mass Communications. Retrieved June 6, 2013. http://www2.ku.edu/~acejmc/PROGRAM/STANDARDS.SHTML#std9. 2 Bachelor of Arts, Department of Journalism and Public Communications Page 5 of 32 Discussions with the JPC Community Advisory Board The appointment of a JPC subcommittee to research objectives and outcomes of other programs accredited through ACEJMC. JPC is accredited through ACEJMC. Research regarding direct and indirect measures of educational outcomes for programs in journalism and mass communications by Prof. Paola Banchero and former chair Dr. Fred Pearce. Contributions by ACEJMC 2007 site-visit team chair, Dr. Trevor Brown. This plan is the latest iteration of one drafted in May 2003. It was revised in April 2005 and again in October 2006. The plan was submitted as Standard 9, Program Outcomes and Assessment as part of JPC’s ACEJMC reaccreditation in 2007-2008. JPC faculty adopted the plan on Feb. 15, 2008. Faculty members have revised the plan to match ACEJMC standards since that date. Upcoming modifications to our curriculum will require we revisit the plan more extensively. Bachelor of Arts, Department of Journalism and Public Communications Page 6 of 32 TABLE 1: ASSOCIATION OF ASSESSMENT MEASURES TO STUDENT LEARNING OUTCOMES This table is intended to help organize outcomes and the measures that are used to assess them. Each measure contributes information on the students’ achievement of a different set of outcomes. That contribution is tracked in this table. Senior portfolios* 1 1 0 1 0 1 0 1 Demonstrate an understanding of the history and role of professionals and institutions in shaping communications. 1 0 0 1 1 0 1 0 1 Demonstrate an understanding of gender, race ethnicity, sexual orientation and, as appropriate, other forms of diversity in domestic society in relation to mass communications. 0 0 0 0 1 0 1 1 1 Demonstrate an understanding of the diversity of peoples and cultures and of the significance and impact of mass communications in a global society. 0 0 0 1 0 0 1 1 1 Understand concepts and apply theories in the use and presentation of images and information. 1 1 1 0 1 0 1 1 1 Bachelor of Arts, Department of Journalism and Public Communications won by JPC Capstone students class, JPC 0 JPC A202 pre- and post-test JPC A201 reporting & writing instrument JPC A101 pre- & post-test Internship supervisor surveys Awards Understand and apply the principles and laws of freedom of speech and press for the country in which the institution that invites ACEJMC is located, as well as receive instruction in and understand the range of systems of freedom of expression around the world, including the right to dissent, to monitor and criticize power, and to assemble and petition for redress of grievances. Outcomes Alumni surveys A403exit Senior survey This table also forms the basis of the template for reporting and analyzing the combined data gathered from these measures. Page 7 of 32 Demonstrate an understanding of professional ethical principles and work ethically in pursuit of truth, accuracy, fairness and diversity. 0 1 0 0 1 1 0 1 1 1 0 0 1 0 1 1 0 1 Conduct research and evaluate information by methods appropriate to the communications professions in which they work. 0 0 1 0 1 1 1 0 1 Write correctly and clearly in forms and styles appropriate for the communications professions, audiences and purposes they serve. 0 0 1 0 1 1 1 1 1 Critically evaluate their own work and that of others for accuracy and fairness, clarity, appropriate style and grammatical correctness. 0 0 0 0 0 0 1 0 0 Apply basic numerical and statistical concepts. 0 0 0 0 1 0 0 1 1 0 1 1 Think critically, creatively and independently. Apply current tools and technologies appropriate for the communications 1 0 0 0 1 0 professions in which they work, and to understand the digital world. 0 = Measure is not used to measure the associated outcome. 1 = Measure is used to measure the associated outcome. *Senior portfolios are not yet in use in the department. Bachelor of Arts, Department of Journalism and Public Communications Page 8 of 32 ASSESSMENT MEASURES A description of the measures used in the assessment of the student learning outcomes and their implementation are summarized in Table 2 below. The measures and their relationships to the student learning outcomes are listed in Table 1, above. There is a separate appendix for each measure that shows the measure itself and describes its use and the factors that affect the results. TABLE 2: ASSESSMENT MEASURES AND ADMINISTRATION Meas ure Alumni Survey JPC A202 pre- & posttest JPC A201 reporti ng and writing instru ment JPC A101 preand posttest Interns hip supervi sor survey s Award s won by JPC studen Frequency/ Start Date Collection Method Administered in fall 2001; administered in fall 2006; to be administered in 2013 Previously a mailed survey; now an online survey JPC staff Students in JPC A202 are given a pretest that measures their basic understanding of journalistic principles and ethical values. A post-test was given to the cohort during the last week of class in JPC A403, which is the capstone class for JPC majors. We changed that in 2011-12. Began in fall 2008; each cohort year Pre-test instrument is distributed in class, saved and compared to post-test instrument for each cohort. JPC faculty direct Pre-test and post-test administered in JPC A201. Began in spring 2008 and administered through 201112; each section of JPC A201 Pre- and posttest administered in class JPC faculty direct Fall 2007; every semester Pre-test instrument is distributed in class and compared with post-test JPC faculty direct Survey JPC internship director indirect Entries are prepared by students with department JPC faculty and staff indirect Description Alumni are surveyed every five years to learn their views on the achievement of program objectives. This social science survey is given in the first week of JPC A101, JPC’s social science GER. The pre-test measures the students’ understanding of how mass communications fits into the social sciences. A post-test is given to the same class the last week of the semester. Internship field supervisors are surveyed about the achievements of each JPC intern. In 12-13, some interns were visited at their internship site to gather qualitative data about their experience and that of their workplace. JPC students’ work is entered for competition among students and professional peers. Began fall 1999; administered each semester, including summers Administered throughout the year since the unit’s inception Administere d by 4 Direct or indirect4 indirect William G. Christ, ed. Assessing media education: A resource handbook for educators and administrators (NJ: Lawrence Erlbaum Associates, 2006), 329-459. Bachelor of Arts, Department of Journalism and Public Communications Page 9 of 32 ts Capsto ne class, JPC A403 JPC students work on projects and presentations, applying research methods and basic statistical analysis. Senior exit survey JPC graduating seniors or recent graduates complete an exit survey to learn their views on the achievement of program objectives based on coursework within the department. Senior portfoli os Randomly selected senior students’ work will be evaluated for their ability to produce professional communications, depending on their concentration in journalism, strategic communications, telecommunications and film, or integrated media. Writing rubric JPC students in select writing-intensive classes will have their work measured against a rubric, comparing assignments completed early in the semester to assignments completed late in the semester for mechanics and content. Fall 2008 and revamped in fall 2010; each semester course is taught Collected each semester beginning in fall 2001 through 2007; brought back in 2012 JPC has waited for UAA to institute eportfolios, but plans to carry out its own assessment beginning in 2015. Piloted in 2012 in Reporting and Writing News, to be implemented in fall 2013. and faculty support Projects and presentations are prepared by students and evaluated by faculty JPC faculty Previously a mailed survey; now an online survey JPC staff Electronic portfolios will be produced for review by faculty and external committee of professionals JPC faculty and members of the Anchorage professional media community Class assignments measured against rubric JPC faculty Bachelor of Arts, Department of Journalism and Public Communications Page 10 of 32 direct & indirect indirect direct direct ASSESSMENT PROCESS General Implementation Strategy The Department of Journalism and Public Communications, which had 145 majors in fall 2012,5 will continue to diligently collect data to satisfy university mandates and Standard Nine of Accrediting Council of Education in Journalism and Mass Communications. In the past two years, the Department has relied more on its Community Advisory Board about preparation for careers in journalism and public communications. The department has also relied on numerous outcomes assessment tools, some of which are studier and more meaningful than others. JPC faculty also meet regularly to discuss curriculum and assessment. The standard university instructor evaluation forms completed by students in every course at semester’s end can also be valuable feedback. Other ongoing assessment measures include writing and reporting assignments and products to demonstrate mastery, including media production of campus and community events such as the Great Alaska Shootout, UAA Ice Hockey, news programming for broadcast by campus television and campus radio KRUA, True North magazine, department newsletters, news articles for publication by The Northern Light, digital media for service and professional organizations, and strategic communications campaigns. The department also tracks outcomes through student-produced media projects; and outcome of competition in local, regional and national awards programs, such as the Hearst Foundation Awards Journalism Awards Competition and the Alaska Press Club annual competition. Description of Faculty Involvement The faculty of the Department of Journalism and Public Communications meet, usually in August, to review data collected using the assessment tools and to consider changes to our assessment process. However, questions of assessment surface in many faculty meetings over the course of a given academic year. Faculty make recommendations designed to improve the program’s objectives and outcomes. Results of data collection, interpretation of the results, and recommended program changes are passed on to the Office of Academic Affairs by June 15 every year. This process has driven programmatic changes, such as the wholesale revision of the curriculum in 2006 and adjustments that were made in the 20112012 academic year. We are again looking at curricular issues given the rapid changes in the communications and media industries. Through this plan, faculty will continue to review data to refine curriculum to meet program objectives and outcomes. These proposed programmatic changes may be any action or change in policy that the faculty identifies as being necessary to improve performance relative to program outcomes, such as changes in course content, scheduling, prerequisites, means of delivery, advising, and facilities. 5 UAA Office of Institutional Research Bachelor of Arts, Department of Journalism and Public Communications Page 11 of 32 Modification of the Assessment Plan The JPC faculty, after reviewing the collected data and the processes used to collect it, may decide to alter the assessment plan. For example, we had originally planned to test students on their understanding of First Amendment issues in JPC A202 and again in JPC A403, the Integrative Capstone class. We decided it made more sense to make the instrument a pre- and post-test for JPC A202. JPC faculty may make changes to any component of the plan, including the outcomes, assessment measures, or any other aspect of the plan. The modified assessment plan will be forwarded to the dean/director’s office and the Office of Academic Affairs. This plan includes the revised ACEJMC core values and competencies, which go into effect in September 2013. Previously, 11 competencies were required to be measured. Now, 12 competencies will be measured in the revised plan. Faculty members continue to evaluate and revise this plan as necessary, in some cases evaluating assessment models from other ACEJMC-accredited programs. Bachelor of Arts, Department of Journalism and Public Communications Page 12 of 32 APPENDIX A: ALUMNI SURVEYS Measure Description: The JPC alumni survey asks graduates to report their JPC concentration area6 and list the three most recent jobs the respondents have held since graduation. The survey also asks alumni to report how well they believe a JPC education prepared them for their first job. Surveys were last distributed to alumni in 2006. Another is being administered this summer via email. Department staff members administer the surveys. Factors that affect the collected data: Low return rates. In 2001, surveys were sent to 350 alumni for whom addresses were available; 81 responses were received for a 23% response rate. Accuracy of the results is reduced with a low survey return rate. Most of the responses were from graduates still in Alaska. The data are skewed to the viewpoint of this subset of graduates. How to interpret the data: The survey had two main goals that potentially limit its value here. The survey’s purpose is to verify the major job categories of our graduates and to gather anecdotal information in response to the openended question about how well a JPC education prepared graduates for their first job. Results from the 2013 survey, which should have been completed in 2012, will not be available till later in the year. The instrument is a fair to good source of perceptions in some outcome areas focusing on alumni. While the survey did not seek data on other program outcomes, responses in 2006 revealed key findings7: Alumni believe the department’s instruction in basic skills of reporting and writing prepared them very well for a variety of jobs. The Atwood professors were exceptional in providing real-world training. Internships were invaluable. 6 Program emphasis areas since fall 2006 are journalism, strategic communications, telecommunications and film, and integrated media. 7 Self-study report: Department of Journalism and Public Communications, University of Alaska Anchorage, 2007-2008 Bachelor of Arts, Department of Journalism and Public Communications Page 13 of 32 Factors that affect the collected data: We will have to wait until results are in later in summer 2013 to see what the trends are. How to interpret the data: Although we don’t have current data, we hope to identify where our graduates are going to work and how their JPC degree influenced what kind of experiences they have in their first few positions. Pertinent questions from Alumni Survey: 1. Please list the jobs you have had since graduation, starting with current job. 2. What was your concentration in the Department of Journalism and Public Communications? Bachelor of Arts, Department of Journalism and Public Communications Page 14 of 32 APPENDIX B: JPC A202 PRE- AND POST-TEST Measure Description: The JPC Principles and Ethics Instrument has been administered since 2008-09 in JPC A202 First Amendment and Media Ethics, which is generally taken when students are sophomores. The plan initially was to have students in the Integrative Capstone course, JPC A403, take the post-test. But when a different faculty member began teaching the capstone in fall 2010, we re-evaluated the plan. We decided to start using the instrument as a pre- and post-test in JPC A202. Because the class is offered only in the spring, spring 2012 is the first year we used this model. Therefore, this is the first year we collected post-test data. JPC faculty developed the instrument, but needs some changes to reflect more of the ethical issues presented in JPC A202. The survey’s main goal is to gather information about direct outcomes of JPC’s program educational effectiveness, especially about First Amendment principles and the law appropriate to professional practice and professional principles such as accuracy, fairness and diversity. Data is tabulated by the JPC assessment coordinator and administrative assistant and presented to faculty to gauge curriculum effectiveness. Factors that affect the collected data: We had no post-test data until 2011-12, meaning most of the data from 2008-09 until then is not very valuable. We lacked a solid way to identify students for data collection while keeping them anonymous in the first years of implementation. We have since corrected this issue. How to interpret the data: We believe we only have one year of paired data for this instrument given the reasons mentioned above. The spring 2012 results indicate that instruction is helping students understand First Amendment concepts. Results from spring 2012: Section 001: Pre-test: Average score was 14.13 out of 20 points possible, or 71%, with a standard deviation of 1.73. Post-test: Average score was 15.23 out of 20 points possible, or 76%, with a standard deviation of 3.39. Section 002: Pre-test: Average score was 14.33 out of 20 points possible, or 72%, with a standard deviation of 2.03. Post-test: Average score was 15.43 out of 20 points possible, or 77%, with a standard deviation of 1.28. Bachelor of Arts, Department of Journalism and Public Communications Page 15 of 32 Results from spring 2013: Section 001: Pre-test: Average score was 15.1 out of 20 points possible, or 75.5%, with a standard deviation of 1.59. Post-test: Average score was 15.65 out of 20 points possible, or 78.3%, with a standard deviation of 1.69. This was not a statistically significant increase. Section 002: Pre-test: Average score was 13.83 out of 20 points possible, or 69.2%, with a standard deviation of 3. Post-test: Average score was 15.67 out of 20 points possible, or 78.4%, with a standard deviation of 2.3. This was a statistically significant difference. Bachelor of Arts, Department of Journalism and Public Communications Page 16 of 32 APPENDIX C: JPC A201 PRE- AND POST-TEST Measure Description: The JPC Reporting and Writing instrument is administered as a pre-test and post-test in each section of JPC A201 Reporting and Writing News. JPC faculty developed the instrument. It was administered beginning spring 2008 in one section and was introduced to other sections in fall 2009. Survey results are gathered by the JPC assessment coordinator and staff and used for improving the course. The JPC Reporting and Writing instrument was designed to determine direct educational outcomes of JPC A201, particularly in journalistic writing conventions, the application of basic knowledge of communications law and the ability to identify inaccuracies or inconsistencies in reporting—a key critical thinking skill. Factors that affect the collected data: Faculty members inexperienced with the assessment process did not create a corresponding rubric to grade this instrument for the first few years the assessment was in effect. A rubric has been developed to measure writing, the application of basic communications law and accuracy. The design of the instrument makes data collection and interpretation difficult. We did not use this assessment in 2012-13, but rather used a rubric for each writing assignment in one section of JPC A201. See below for more information on that tool. How to interpret the data: Students can score up to 6 and as low as -6 on the assessment. An overall positive score means that the student advanced in understanding and application of skills between the pre-test and the post-test. An overall negative score indicates that the student made no gains or failed to demonstrate competencies shown at the beginning of the semester. In 2011-12 the mean score was -1, which is interpreted to mean that students lost ground over the course of the semester. However, the students spent far less time on the post-test than they did on average on the pre-test. On the pre-test, students spent approximately 50 minutes writing the brief news report. On the post-test, students spent approximately 24 minutes writing the report. Most students did not pose journalistic questions as instructed, which reduced their overall score. Another problem was that in one section, only three students took both the pre- and post-test, impacting the reliability of the results. The scant results give some indication that JPC 201 instructors need to spend more time helping students understand the importance of questioning material presented as fact. That was the greatest weakness in the results. Writing skills, for the most part either stayed consistent or improved. Bachelor of Arts, Department of Journalism and Public Communications Page 17 of 32 JPC Reporting and Writing Instrument Please write a news story using the following facts, all provided by the police chief. You do not have to use all of them. If you question any statements from the chief, put your questions at the end of your story. Be alert for inconsistencies. 1. Three vehicles collided around noon at 40th St. and Old Cheney Rd. in suburban Arlington, VA. Three people were killed and five were injured. 2. The driver of the other vehicle, a 1999 Volkswagen, was Robin Beck. 3. The Jaws of Life tried to rescue Jane Foster, but failed. The van she was riding in burst into flames and was totally destroyed. 4. Dead are: Alfred Jones, 72, driver of the 1987 Chevy van. Anne Jones, 40, Jones’ wife. Jane Foster, 24, a friend of the Joneses and passenger in the Jones van. 5. Damage to the vehicles involved totaled $26,000. There was $11,000 damage to the Volkswagen; $8,500 to the Chevrolette van; and $6,500 to the Pagel car. 6. Beck’s dog, Poof, was thrown from the van and died after hitting a fire hydrant. 7. Injured were: Ted Morkel, 60 years old. He suffered a fractured leg. Jim Smith, 43 years old. He suffered multiple internal injuries. George Gregory, 27. He suffered lacerations. Megan Smith, 3 years old. She suffered lacerations and contusions. 8. “It’s a good thing Alfred Jones died in this wreck,” Police Chief Paul Stevens commented. “Otherwise, he would have faced vehicular homicide charges and we would have put him away for good. Not only that, but he was drunk. Not sort of drunk, but smashed.” 9. A crowd of almost 250 persons gathered around the accident scene as firemen from the Arlington Fire Department tried to resuscitate Poof the dog. 10. A passerby, Barbara Wells, said, “They are going to have to put a stop sign at that intersection soon. This is the 17th fatal accident there this year. That number includes three dogs and one cat.” 11. The dead and injured were taken to Arlington Memorial Hospital. An investigation is continuing. Stevens said the intersection was not unsafe. Rather, he said, “people drive like fools. What do you expect?” 12. "Something has to be done about that intersection," the chief said. "I have asked over and over for electric lights to be put up, but the city says it just doesn't have the money. With the loss of life it has caused, we can't afford not to put lights there." 13. Robin Beck was charged with vehicular homicide and leaving the scene of an accident. She will be arraigned Monday. Bachelor of Arts, Department of Journalism and Public Communications Page 18 of 32 APPENDIX D: JPC A101 PRE- AND POST-TEST Measure Description: The Department of Journalism and Public Communications has one social science GER. It is JPC 101 Media and Society. This is a survey course that describes the evolution of newspapers, magazines, photography, film, radio, television, Internet and the development of telecommunications and information technologies in the last 100 years. The emphasis is on social, cultural, political and economic effects of media. In 2012-13 the social sciences GER outcomes were tweaked. We made basic edits to the instrument in August 2012, but are planning more adjustments our instrument and to the course to reflect these changes. Factors that affect the collected data: A new adjunct faculty member inexperienced with the assessment process conducted the survey in fall 2012. Collection has been ongoing, but faculty members assigned to teach this course have not always carried out assessment consistently. How to interpret the data: In 2012-2013, JPC administered a pre- and post-test to students enrolled in this course. In fall semester, we offered one section capped at 60 students. In spring 2013, we offered two sections capped at 74 each. Each section had approximately 50 students. Students on the pre-test in fall scored a mean of 11.67, or 58.33%. Students on the post-test scored a mean of 12.26, or 61.29%. In spring 2013, students in section 001 scored 11.41, or 57.06% on the pre-test and 12.09, or 60.44% on the post-test. Section 002 scored 11.39, or 56.94% on the pre-test and 12.28, or 61.39 on the posttest. Both semesters and all three sections show a modest but statistically significant increase in these assessment scores. Bachelor of Arts, Department of Journalism and Public Communications Page 19 of 32 JPC A101 Social Science GER Instrument 1. Social Science methods are based on what principles of inquiry a. Socratic methods b. Random methods c. Empirical methods d. Hypothetical methods 2. Mass Communications a. Cannot be an empirical discipline b. Is not a discipline c. Is outdated d. Is the study of what is communicated, by whom, to what affect 3. Mass Communications research a. Has only one method of analysis b. Uses many methods of analysis c. Is based on quantitative analysis only d. Is based on rhetorical theory only 4. Mass Communications theory a. Doesn’t exist b. Is designed to explain and predict c. Cannot be established d. Is a solved problem 5. Internal validity in Mass Communications research a. Is impossible to establish b. Is a function of proper definitions and logic c. Is something that one should not worry about d. Is arbitrary 6. External validity in Mass Communications research a. Is arbitrary b. Is a function of convention c. Meets the test of other research d. Is something that should be of no concern 7. Quantitative methods in mass communications research are used a. Never b. Always c. When dealing with large data d. When convenient 8. Qualitative methods in mass communications research are used a. Never b. Always c. When analyzing manifest content d. When convenient 9. Which kind of research method reveals correlations between two variables? a. Surveys b. Experiments c. Content analysis d. Focus group interviews 10. Correlation cannot establish a. Causality b. Meaning c. Tradition d. Relationships 11. Which kind of research method employs a control group for comparison? a. Surveys b. Experiments c. Content analysis d. Focus group interviews Bachelor of Arts, Department of Journalism and Public Communications Page 20 of 32 12. Institutional Review Boards are important in Mass Communications research a. Because they provide funding b. Because they provide protection to human subjects c. Because they manage grants and contracts d. The IRB is not important to research 13. Moving images and visual communications a. Are impossible to empirically study b. Require unique methods of empirical analysis c. Need critical attention d. Cannot be codified 14. For this kind of method, researchers systematically code and measure media content a. Surveys b. Experiments c. Content analysis d. Focus group interviews 15. The Arbitron and Nielsen ratings systems use what principles of research a. Lucky guesses b. Representative samples c. Non-representative samples d. Random samples 16. Newspaper readership studies a. Are arbitrary b. Are necessary to understand changes in audience c. Are a waste of time and money d. Are not taken seriously 17. Radio formats usually target special audiences according to a. Age and income b. Gender c. Race or ethnicity d. All of the above 18. VALS research for advertising refers to a. Emotional, social and economic audience profiles b. Profiles of advertising by geographic region c. Viceroy’s Analysis of Life Systems d. The study of audience attention spans 19. Psychographics involves the study of a. Sex b. Age c. Socioeconomic class d. Values and lifestyles 20. Which line of research has generally demonstrated that that mass media don’t tell people what to think as much as they tell people what to think about? a. Cultivation analysis b. Agenda-setting c. Social learning theory d. Audience studies Bachelor of Arts, Department of Journalism and Public Communications Page 21 of 32 APPENDIX E: INTERNSHIP SUPERVISOR SURVEYS/INTERVIEWS Measure Description: The JPC Internship Survey asks supervisors of student interns to evaluate interns in areas that reflect professional core values and competencies. All interns seeking academic credit are evaluated at the end of their project, usually semester’s end. Survey results are gathered by the JPC internship director and used to help form the basis of the intern’s final grade. Surveys typically are followed up with a phone call or e-mail message from the internship director. We have changed the evaluation form to yield more uniform information about how interns did, not matter what kind of internship post they held. Factors that affect the collected data: Low volume of surveys. The program typically accommodates fewer than 25 interns annually. Some internship supervisors are loath to write in detail about a student’s performance or lack thereof. Follow-up phone calls can often generate more useful data about how a student met or failed to meet outcomes. We added some site visits this year to gather qualitative information about the intern and the internship experience. How to interpret the data: The survey’s main goals are to help ensure that internship supervisors remain engaged over the course of the internship; duplicate a real-world working relationship in which a subordinate is evaluated by a supervisor; and to help the internship director gauge an intern’s progress in the field. Workplace supervisors also are asked for ways in which the internship program may be improved. The instrument tends to be a good indirect measure of the following student learning outcomes: 1, 5, 6, 7, 8, 9, 10, and 12. It could be better if we tailored the questions to better make instrument reflect the ACEJMC student learning outcomes. Field mentor survey data collected over the past five years reveal key findings: Exemplary grades awarded by internship supervisors are an indication that the program has lived up to its goal of recommending only work-ready students for internships. Internship supervisors are being more candid in their assessments because we have asked them to be. However, we want to ask for more details in their feedback. Internship supervisors typically agree that they would accept other interns from JPC in the future. That does not mean that they would hire the intern should a position become available in their organization. Site visits, which are done when the schedule permits for both the internship director and the workplace supervisor, have yielded positive results. Supervisors are more candid in person and also share what might be limitations outside the student’s control. Bachelor of Arts, Department of Journalism and Public Communications Page 22 of 32 JPC PRACTICA/INTERNSHIP EMPLOYER EVALUATION FORM (Mo/Yr) (Mo/Yr) RATING ELEMENTS RATINGS (CHECK ONE) Excellent Rate each intern’s performance by entering an “X” under one of the factors in the rating section at the right for each element. Unacceptable Marginal Satisfactory Practicum or Intern Student’s Name: __________________________________________ Job Title: ______________________ Company: _________________________________ Company Website: ________________________________________________________ Address:________________________________________________________________ Supervisor: _____________________________________Title:_____________________ Email: _________________________________ (If different from above) Internship Contact: ______________________ Email: _________________________________ Evaluation Period: __________ to __________ Above average JOB PERFORMANCE: Consider the quality, quantity and timeliness in accomplishing tasks. Did the intern work without constant supervision, show initiative and interest in work? WORK HABITS: Consider ability to manage time, professional attitude, and willingness to learn. Did the intern seek out and utilize appropriate resources, as well as accept constructive criticism and increasing responsibility? KNOWLEDGE OF WORK: Consider the intern’s knowledge and skills appropriate for the profession. Did the student demonstrate an understanding of concepts and practices of the profession? COMMUNICATION: Consider the intern’s ability to write effectively and efficiently in the form and style appropriate to the profession. Was the intern able to evaluate his/her own work and that of others for accuracy and fairness? PROBLEM SOLVING: Consider the intern’s ability to think critically, creatively and independently. Was the intern able to collect and evaluate various forms of information? PROFESSIONALISM: Consider the intern’s sense of values, respect for the profession as well as the employees in the company. Did the intern show respect for the diversity within the profession and the company? GENERAL CONDUCT: Consider the intern’s punctuality, adherence to work schedules, appropriateness of dress for the position, manner and courtesy on the job and relations with the public. OVERALL EVALUATION: Did the intern meet established employer expectations? Consider the ratings for all of the personal and performance elements above. COMMENTS: (You are invited to attach a separate sheet with your comments) DUTIES/RESPONSIBILITIES: (You are invited to attach a separate sheet with your comments) WOULD YOU ACCEPT OTHER INTERNS FROM THIS DEPARTMENT IN THE FUTURE? YES? NO? Supervisor’s Signature: _________________________ Intern’s Signature: ________________________ Phone: ________________________________ Internship Contact Title: __________________ Phone: ___________________________ Thank you for your participation in our practica/internship program. Please return this form by fax or mail to: University of Alaska Anchorage Department of Journalism and Public Communications 3211 Providence Drive Anchorage, AK 99508-4614 Fax: (907) 786-4179 Bachelor of Arts, Department of Journalism and Public Communications Page 23 of 32 APPENDIX F: AWARDS WON BY JPC STUDENTS Measure Description: This indirect measure of program outcomes looks at how students performed in local, regional and national competitions, mostly in journalism and strategic communications. Factors that affect the collected data: We lack a precise way of keeping track of awards, scholarships and other accolades our students have received. However, we have developed a database and our administrative assistant asks faculty to report to her student success. How to interpret the data: Historically, we have not had a lot of submissions for the Hearst Foundation Awards as some ACEJMC programs do. We have had a relatively strong showing when students have submitted for these awards. We have gathered information about other local, regional and national awards. This data provides only indirect data about our program objectives and outcomes. In 2011-12, We had one second-place winner of the Hearst Foundation Journalism Award in radio reporting. The student was the first from JPC to place in the prestigious competition since 2005. We three other submissions, but none were named finalists. The Northern Light was named a national Pacemaker for 2010-11, when it was led by JPC students and largely staffed by JPC students. Two JPC students’ papers were accepted for presentation at national conferences. A student won first place in the statewide competition for feature story writing from the Alaska Professional Communicators. In 2012-13, One student won a third-place award for feature writing from the Alaska Press Club. Two students won the UAA Student Showcase, an annual competition to recognize students’ creativity activity and research. Three students were asked to present their research at the Undergraduate Honors Conference in Communication at DePauw University. Bachelor of Arts, Department of Journalism and Public Communications Page 24 of 32 One student took second place in the Herbert A. Kassner Journalism Contest for human-interest story. The contest was for public affairs specialists all over the U.S. Army Corps of Engineers. One student was awarded the Ward Sims Journalism Scholarship. Bachelor of Arts, Department of Journalism and Public Communications Page 25 of 32 APPENDIX G: JPC INTEGRATIVE CAPSTONE COURSE, JPC A403 Measure Description: The primary objective of JPC A403 is to introduce students to communications and media research. Course outcomes are designed to transition students from consumers of research to producers of original scholarship. As the JPC program’s capstone course, JPC A403 outcomes fulfill a wide range of assessment items related to student learning throughout the curriculum. Students are required to submit assignments resulting in a 25-30 page paper. These assignments help to appraise students’ ability to write, evaluate their work and that of others, conduct research, and demonstrate critical, creative and independent thinking. Class exercises also incorporate the application of basic numerical and statistical concepts related to the research process. Moreover, students undergo Institutional Review Board training to help them understand ethical considerations related to truth, accuracy, fairness and diversity in the media and communications industries. In the course of research design, students must apply concepts and theories related to their research topic. Students must also demonstrate an understanding of the history and role of professionals and institutions in shaping communications, as well as systems of freedom of expression around the world. Within the context of the United States, First Amendment principles are highlighted. In addition to the paper, students must also demonstrate their ability to present information orally by participating in a poster session and applying the tools and technologies needed for poster design and presentation execution. For both the paper and poster session, Association for Education in Journalism and Mass Communication standards are used for evaluation. Factors that affect the collected data: We had not developed a precise way of measuring success in scholarship in this course until fall 2010. Student research projects vary widely in this course, making comparisons difficult. How to interpret the data: Several benchmarks are used to assess student success. These benchmarks include student participation in the UAA Student Showcase Conference, UAA University Honors College Undergraduate Research Symposium, publication in the UAA Student Showcase Journal and presentations at AEJMC-level national communication conferences. These benchmarks, along with JPC 403 class outcomes allow the department to evaluate whether students have gained the necessary skills and abilities from the program. In 2009, no student enrolled in two sections of JPC A403 participated in university, regional or national research conferences. In 2010: -One student was selected to present her research at the Association for Education in Journalism and Mass Communication (AEJMC) national convention. Bachelor of Arts, Department of Journalism and Public Communications Page 26 of 32 -One student was selected to present his research at the Broadcast Educators Association national convention. -Four students were featured in the 2011 Student Showcase. -Two students were chosen to participate in the 2011 Undergraduate Research Symposium. In 2011: -Five students were chosen to participate in the 2012 Undergraduate Research Symposium. -Five students were chosen to participate in the 2012 Student Showcase. In 2012: -Six students were chosen to participate in the 2013 Student Showcase. Two students won the competition. -Three students were chosen to participate in the 2013 Undergraduate Honors Conference in Communication Conference at DePauw University. Bachelor of Arts, Department of Journalism and Public Communications Page 27 of 32 APPENDIX H: SENIOR EXIT SURVEY Measure Description: JPC graduating seniors complete an exit survey to learn their views on the achievement of program objectives based on coursework within the department. Factors that affect the collected data: Survey was collected each semester beginning in fall 2001 through 2007; it has been brought back in summer 2012. How to interpret the data: We are lacking a rubric to tell us whether students are developing into people who can be successfully employable in journalism and public communications. In 2012, student respondents to the survey reported that they felt they had emerged from the program with the understanding of First Amendment law and professional ethics we demand. They also responded they felt prepared in terms of the technological and practical skills for their entry-level jobs. This data provides only indirect data about our program outcomes. Bachelor of Arts, Department of Journalism and Public Communications Page 28 of 32 SENIOR EXIT SURVEY QUESTIONS RELATED TO ASSESSMENT Students were asked to measure their agreement on a five-point scale. JPC prepared me to understand and apply First Amendment principles and the law appropriate to professional practice while comparing and contrasting the range of systems of freedom of expression around the world. JPC prepared me to demonstrate an understanding of the history and role of professionals and institutions in shaping communications. JPC prepared me to understand and appreciate the value of diversity in the media and in a global society. JPC prepared me to understand concepts and apply theories in the use and presentation of images and information. JPC prepared me to demonstrate an understanding of professional principles and work ethically in pursuit of truth, accuracy, fairness and diversity. JPC prepared me to demonstrate critical, creative and independent thinking by contributing to the communications needs of the community and the state through projects, internships and service. JPC prepared me to conduct research and evaluate information by methods appropriate to the communications professions. JPC prepared me to write correctly and clearly in forms and styles appropriate for the communications professions, audiences and purposes they serve. JPC prepared me to evaluate my own work and that of others for accuracy and fairness, clarity, appropriate style and grammatical correctness. JPC prepared me to apply basic numerical and statistical concepts. JPC prepared me to apply tools and technologies appropriate for the communications professions. Co-curricular programs such as The Northern Light, KRUA, UAA Seawolf Hockey productions prepared me for my career. Bachelor of Arts, Department of Journalism and Public Communications Page 29 of 32 APPENDIX I: SENIOR PORTFOLIOS Measure Description: The JPC Senior Portfolio will examine select and representative senior portfolios to judge the preparedness of JPC students for the professional quality of their work. We have delayed the implementation of this assessment process because the university was working on an e-portfolio project, and we did not want to duplicate efforts. Unfortunately, the university initiative has stalled. We must implement a process for portfolios, which will likely be built into the curriculum. If and when the university implements a process, we will ask students to migrate to the software UAA chooses. We have the buy-in from our community advisory board and will try to launch a pilot assessment in 2013-14. These portfolios will be submitted voluntarily by students, with the goal of unveiling a full senior portfolio metric in 2014-15. A senior portfolio review instrument will be developed to quantify this process. These responses from the committee will be saved for analysis. The senior portfolio review process will be confidential. Factors that affect the collected data: We have no data to report at this time. How to interpret the data: We have no data to report at this time. The senior portfolios will be submitted by students and collected and maintained by staff and the assessment coordinator, who receive results and tabulate them for use in faculty outcomes review. Data will be analyzed beginning summer 2013 and presented to faculty at the following August faculty meeting. We are developing a rubric to assess portfolios. Bachelor of Arts, Department of Journalism and Public Communications Page 30 of 32 APPENDIX J: WRITING RUBRIC Measure Description: This is our newest assessment tool, and it needs improvement. It was developed to try to gather information about students’ writing skills and critical thinking. We used this assessment in one section of JPC A201 Reporting and Writing News in fall 2012. Each student’s writing assignments were measured against the rubric. The goal was to see a substantial improvement in students’ writing and critical thinking skills. Ideally, this rubric would be modified and could be used in several courses that are writing-intensive. Factors that affect the collected data: We do not have meaningful data to report at this time because we have not had time to analyze the information. Bachelor of Arts, Department of Journalism and Public Communications Page 31 of 32 PILOT WRITING RUBRIC Lead 10 points Writing 15 points Is it concise? Avoids irrelevant detail, long lead-in clauses that delay action) and specific? Does it effectively summary the story and convey the uniqueness of story? Uses simple sentence structures that have one main idea per sentence and avoid too many clauses, buried subjects/verbs, misplaced modifiers. Uses effective word choice includidng wording that is specific rather than vague; employs active verbs, does not have any editorializing or loaded (opinionladen) words; avoids unexplained jargon, cliches, journalese, redundancies and first-person constructions; presents key facts clearly and early in story and uses a coherent, logical ordering of facts. 10 points Story has information significant/important to audience, is based in the classic elements of news judgment and avoids inclusion of irrelevant information; story has no key details missing (5Ws and H) and no unanswered questions for reader 15 points All sources (human or documentary) are qualified to address subject. All sources that should be included are included. All information properly attributed to sources, attributions of quotes clearly identify speakers and all sources properly identified on first reference, subsequent references. Reporting is thorough, accurate and fair. 5 points Written in concise, effective fashion (headline 10 words or fewer; blurb 25 words or fewer) and summarizes story without repeating the lead. Completeness Sourcing/Attribution Web extender Bachelor of Arts, Department of Journalism and Public Communications Page 32 of 32