PROVOST OFFICE ACTION STEPS PROGRAM ASSESSMENT NOTE: The Provost’s Office is requesting that Faculty Senators and Deans read the complete document for the context of the changes. However, we are particularly requesting feedback on the details associated with Action Step #1 detailed on pp. 7-9. March 10, 2016 Action Steps based on Recommendations in May 2015 Biennial Assessment Working Group Report Dr. Sandra Grunwald interim Associate Vice Chancellor for Academic Affairs Table of Contents I. II. III. I. Executive Summary Background of the Review of Programmatic Assessment Processes Provost Office 2015-2017 Programmatic Assessment Initiatives Executive Summary Based on substantial review of processes and engagement with individuals highly invested in programmatic assessment over several years, this report delineates the background and components of the review, the areas identified for improvement, and several initiatives (with action steps) in progress. Recommendations guiding the assessment initiatives associated with the Provost’s Office for the 2015-2017 biennium are grouped into three key areas. 1. Recommendations for Increasing Communication and Knowledge ● ● ● ● Establish an assessment sharing process that allows departments to benefit from cross-disciplinary assessment efforts. Articulate a shared conceptual framework and common language for assessment. Provide opportunities for academic programs to showcase program-level assessment. Make more public program assessment results Action Item: Academic Affairs sponsored a university-wide assessment event at UWL on January 20, 2016 -- Assessment Commons: Asking Questions, Sharing Practices, Finding Answers. Approximately 180 faculty and staff attended the event. 2. Recommendation on Reward and Accountability Structures: ● Incentivize assessment through the Provost Office. Action Item: In January 2016 the Provost Office put out a call for Program Assessment Initiative Grant proposals to provide monetary support for gathering, analyzing, discussing and acting on evidence of student learning in programs. Ten proposals were received from departments/programs that spanned a range of level of program assessment development (as self reported). Funding requests spanned proposals associated with department/program retreats, conference attendance, and support for the development of assessment tasks, evaluation rubrics or plans. 3. Recommendations on Consistency of Feedback: ● Create an Assessment Associates review group for a university-level, facultybased review process for program assessment reports. ● Develop a university-level program assessment report feedback rubric. ● Establish a common timeline to provide and receive feedback on program assessment. Action Items: 1. Consolidate review of program assessment to a university-wide, faculty-based program. ○ includes consultation with Faculty Senate, development of rubrics, timelines and membership guidelines 2. Move to a 3-year program assessment cycle 3. Move program assessment reporting to Taskstream 1 II. Background of the Review of Programmatic Assessment Processes In Spring 2015, Provost Macpherson led a comprehensive review of the program assessment reporting and review process in place at UWL. College biennial assessment reports and individual program/departmental biennial assessment reports were reviewed in addition to a meeting dedicated to the topic on February 17, 2015 during which Deans, Associate Deans, faculty leaders within the colleges, and the UWL Assessment Coordinator shared their views of the program assessment process. From this comprehensive review, a Working Group on Biennial Assessment was formulated and charged with providing recommendations in response to the following guiding questions: 1. Does the timing of review and feedback of assessment within the academic programs meet the needs of all parties involved? 2. Does the current structure of review and feedback of assessment within the academic programs meet the needs of all parties involved? 3. If these conditions (both or just one) are not met, what substantive changes could be taken to improve the process? The working group surveyed department chairs and reviewed existing information on the assessment sections of Academic Program Review reports to investigate these questions. In their summary report submitted to Provost Macpherson in May 2015, the working group stated Based on a careful review of the feedback from chairs, the working group identified several barriers to effective biennial program assessment at UW-L: ● Defining assessment and associated expectations o Understanding what program assessment is o Unclear communication about program assessment o Expectations not clearly articulated ● Faculty buy-in/ownership o Faculty resistance to program assessment o Limited time to engage in program assessment activities ● Failure to close the loop o Lack of sufficient communication about the results of program assessment o Not using program assessment data to inform decision-making The working group described the overall state of assessment as follows: ● There is a need to improve the overall culture of program assessment ● Many programs are struggling with program assessment (see Appendix B), based on an analysis of the past 8 years of APR report program assessment results ● There is a need for improved communication at all levels, as evidenced by o Confusion between annual and biennial review (CLS) o Dissonance between college-level and dean-level review and feedback o Decentralized processes for program assessment - each college goes about things a different way that lead to a common biennial report 2 The working group made the following four overarching recommendations to improve the program assessment process, including specific action steps related to each recommendation. 1. Improve consistency of feedback regarding program assessment a. Consolidate review process to a university-level faculty peer group. Identify faculty to serve as Assessment Associates who complete the review of the assessment reports. See the University of Northern Iowa model. b. Develop a university-level program assessment report feedback rubric. c. Establish a common timeline to provide and receive feedback on program assessment. 2. Encourage increased communication and knowledge about program assessment at all levels a. Articulate a shared conceptual framework and common language for assessment. b. Communicate feedback about program assessment to entire department. c. Establish an assessment sharing process that allows departments to benefit from cross-disciplinary assessment efforts. d. Clearly articulate the three levels of assessment - course, general education and program - and the similarities and differences of each. e. Develop and disseminate clear expectations for program assessment. f. Clarify the roles of different individuals involved with assessment on campus. g. Encourage attendance at ongoing professional development opportunities provided by CATL to enhance the culture of assessment at UW-L. 3. Identify strategies to increase faculty buy-in a. Incorporate program assessment into faculty promotion files. b. Incorporate program assessment into digital measures headings. c. Encourage departments to put assessment issues on agenda at departmental meetings. 4. Establish a reward and accountability structure a. Establish an assessment recognition process to identify and celebrate academic programs for outstanding work in assessing student learning outcomes. b. Provide opportunities for academic programs to showcase program-level assessment. c. Make programs more accountable for program assessment. d. Incentivize assessment through the Provost Office. e. Make more public program assessment results 3 III. Provost Office 2015-2017 Programmatic Assessment Initiatives The Provost Office carefully reviewed the report and has subsequently developed and supported several initiatives in the 2015-17 biennium that address recommendations within the communication / knowledge and reward recommendation categories as described below. Furthermore, thorough investigation into the improving consistency of feedback recommendation has occurred and a plan has been formulated with action steps as described below. 1. Recommendations for Increasing Communication and Knowledge ● Establish an assessment sharing process that allows departments to benefit from crossdisciplinary assessment efforts. ● Articulate a shared conceptual framework and common language for assessment. ● Provide opportunities for academic programs to showcase program-level assessment. ● Make more public program assessment results Action Item: Academic Affairs sponsored a university-wide assessment event at UWL on January 20, 2016 -- Assessment Commons: Asking Questions, Sharing Practices, Finding Answers. ● In Summer and Early Fall 2015, Betsy Knowles (Economics), gathered feedback through one-on-one discussions and follow-up surveys from a variety of stakeholders at UWL to inform planning of an assessment day event. A summary of stakeholder input was submitted to the Provost and informed the faculty-based committee that planned the event. ● A Planning Committee made up of five faculty members spent several months developing and carrying out the Assessment Commons event that was made up of a variety of workshops, panel discussions and presentations that address current concerns and needs of the university community with respect to assessment practices (see full event program). The goals of the event were to 1) provide a forum for faculty to share and collaborate on methods to measure and improve student learning, 2) to improve understanding of assessment efforts within the context of the university, including shared language, expectations, and vision, 3) provide an opportunity for department members to work together on those aspects of assessment that are most relevant to them. ● As part of the program, attendees were provided a Reference of Assessment Related Terminology and Resources which included 1) a glossary of common assessment related terms, 2) a glossary of UWL assessment related acronyms and terminology, and 3) references for two national assessment initiatives and the resources they provide. These documents will be more broadly distributed. ● Approximately 180 faculty and staff attended the event. Initial feedback and reviews show that the event was highly successful and very well received. 4 Follow-up: The faculty organizers would recommend offering this type of workshop on a yearly basis. 2. Recommendation on Reward and Accountability Structures: ● Incentivize assessment through the Provost Office. Action Item In January 2016 the Provost Office put out a call for Program Assessment Initiative Grant proposals to provide monetary support for gathering, analyzing, discussing and acting on evidence of student learning in programs. Ten proposals were received from departments/programs that spanned a range of level of program assessment development (as self reported). Funding was requested to support department/program retreats, conference attendance, and faculty support for development of assessment tasks, evaluation rubrics or plans. Follow-up: Pending available funding, the Provost’s Office would like to continue and refine some form of programmatic assessment grants. 3. Recommendations on Consistency of Feedback: ● Consolidate review process to a university-level faculty peer group. ● Develop a university-level program assessment report feedback rubric. ● Establish a common timeline to provide and receive feedback on program assessment. Steps taken to involve campus community in the discussion of a consolidation of the review process: Step 1: Solicit feedback from Department Chairs ● At the Fall 2015 Chair’s meeting, the Working Group findings regarding program assessment reporting and review were presented to department chairs, along with all recommendations. ● Chairs were then asked to discuss and provide their feedback on the value and challenge of the above three recommendations from the working group report. ● Feedback indicated that chairs found value in having a university-level, faculty peer group doing program assessment report review especially in regards to 1) consistency and 2) faculty-based. Several challenges that were identified were 1) finding faculty with enough knowledge of program assessment to serve as reviewers and 2) recognizing the programs that are outside-the-norm (i.e. DPI-associated programs, CBA programs, Library) Step 2: Video conference with University of Northern Iowa to learn about their universitywide, faculty-based program assessment report review process. Associate Deans, Working Group faculty members, IRAP members and SEC members were invited to attend. Also an open invitation was sent to each dean to invite others. 5 Video conference attended by CLS: Charles Martin-Stanley, Linda Dickmeyer CBA: Glenn Knowles, Laurie Miller, Betsy Knowles SAH: Barb Bennie IRAP: Natalie Solverson, Patrick Barlow Provost Office: Sandy Grunwald In discussion with faculty/staff who attended the videoconference, agreement was that such a centralized process had value and that UWL should move forward with a similar process. Step 3: Discussion of next steps at the October 2015 Dean’s Council Meeting 6 Action Item #1 - Create an Assessment Associates review group for a university-level, faculty-based review process for program assessment reports. Under development Detail Suggested Deadline Composition of the Assessment Associates review group May 15 ● Ten faculty members within various ranks, representing the three academic colleges and SOE, and also representing both graduate and undergraduate education. ● Potential Assessment Associates names will be gathered from the following: 1) an open call from the Provost to all faculty asking for self-nominations, 2) soliciting nominations from academic deans, 3) soliciting nominations from academic chairs. It is expected that potential Associates will be nominated based upon their previous involvement in assessment activities, preferably within the program area in their department, and their support for assessment of student learning. ● The Provost Office will review the nominations, feedback will be solicited from Deans, SEC, etc… and a final list formulated. The final Assessment Associate list will be reviewed by the Deans and SEC for affirmation. ● Committee terms will be three-years, with the first set of terms being two or three years to provide continuity in membership at the beginning of the process. ● The initial group of committee members will be named by the first week of September. ● Training and guided practice in review of reports will occur by mid-October. Workload and recognition of Assessment Associates ● It is anticipated that service as an Assessment Associate will involve approximately 15-20 hours of time during the fall semester. ● Assessment Associates will be expected to: ■ Be informed and knowledgeable with respect to the review process; ■ Work to provide supported and supportive reviews of program assessment reports; ■ Keep ratings confidential and discuss reports/evaluations only with Assessment NA 7 Associates or faculty involved with a specific program area report. ● Assessment Associates should document their work as a reviewer within Digital Measures to be considered in both the merit and promotion process. Work Expectation ● Each Assessment Associate will be assigned about six Oct 20 reports to review. ● There will be two reviewers for each report, one from the college or the general program discipline. ● Each reviewer will have approximately one month to review individually the assigned reports. Associates will utilize a customized template within Taskstream to conduct the review process. The template will include ratings and opportunities for comments on each required section of the report. ● After Associates have completed their assigned reports, they will confer with the other reviewer for the same reports, with the goal of creating a single evaluation of the report for each program area reviewed. ● Final evaluation reports are to be completed before December 1st. Evaluation reports will be sent to the department chair, program director, and dean. Aggregate reports will be created for college and university levels. These evaluation reports should also feed into the centralized program review process. They could be included as an important appendix within a program’s 7-year APR self-study. Development of a university-level program assessment report feedback rubric. Need to develop review template within TaskStream ● See possible draft template. (Associates will be involved in finalizing review template.) Sept ●Perspectives of the review process will be collected from departmental programs, Deans, and the Assessment Associates. ●Assessment Associates will also meet in January to discuss how the evaluation process worked and make suggestions for improvement to the process for the next academic year. Jan 15 Evaluation of the new process 8 Explore the need for an exemption process for programs that are externally accredited and have a periodic review process already in place. TBD May 15 Review of assessment plans Involvement of assessment associates in this process? TBD Action Item #2 - Move to a 3-year program assessment cycle ● Provides time for gathering at least two years worth of assessment data and then time for programs to review results and implement action steps as needed. ● Timeline will be created such that certain “tasks” are completed at certain points within the 3-year cycle to facilitate a full report at end of 3 years. ● At least two complete cycles would be completed within the 7-year APR cycle. ● Submission years of program assessment reports will be staggered so that all reports are not due in the same year. ● Program Assessment reports will be due October 15th of the submission year (changed from the current June 30th deadline). Feedback from chairs indicated that the October submission date would greatly increase communication, feedback, and review of the report by the whole department before submission (versus just the chair and perhaps one other faculty member completing, reviewing and submitting the report). Program Assessment Report review process ● Program assessment reports will be completed in Taskstream by October 15th of submission year. Submission years will be staggered among programs with approximately half of program reports due in 2016 and again in 2019, with the other half due in 2017 and again in 2020. The initial submission year schedule will be based on 1) the program’s APR schedule and 2) the level of development of the program’s assessment (based off of the program’s last self study report). Action Item #3 - Move program assessment reporting to Taskstream ● Taskstream interface will be created to capture information that is currently requested on biennial assessment form. ● Benefit: Replaces paper/electronic copies of biennial assessment reports that are currently uploaded into D2L Biennial Assessment site. Taskstream also facilitates the review of program assessment reports by providing a central repository for reviewers to access and a central place to provide and access comments. Next steps to convert program assessment reporting to Taskstream. 9 January/February 2016 - Patrick Barlow completed development of Taskstream program assessment portal. February - Sandy Grunwald reviewed portal. February/March - Group of five faculty representing CBA, CSAH, and CLS (Bonnie Bratina, Laurie Miller, Glenn Wright, Terry Lilley, Kim Vogt) will be provided training on the program assessment portal and will review and provide suggested changes on direction language, etc… Necessary changes will be made per feedback received. February - Communicate with SEC regarding changes to process. March - Deans will be provided update on process. March 10 - Dept chairs, program directors and assessment coordinators will be advised to 1) review their program SLOs that have been pre-inputted into the system and make necessary changes, 2) input current curriculum map into system (if applicable), and 3) input current program assessment plan into the system by April 1st. During this communication a draft of the submission year schedule will be circulated for comment. March 14-25 - Patrick Barlow will provide four training sessions on the Taskstream program assessment portal. 10