ASSESSMENT PEER CONSULTANT TRAINING St. Cloud State University January 28 and 29, 2008 Trainers: Patricia Aceves, Elaine Ackerman, Wendy Bjorklund, Lisa Foss, Sandra Johnson, Jim Knutson-Kolodzne, Joe Melcher, Jim Sherohman www.stcloudstate.edu/assessment INTRODUCTIONS OF PARTICIPANTS AND TRAINERS Introductions Name and current work assignment Why you want to be an assessment peer consultant A concern, question, or apprehension you have about being an assessment peer consultant Your impressions of Walvoord’s assessment workshop INTRODUCTION TO PEER CONSULTANT TRAINING HLC Assessment Academy 4-year commitment Exempts institutions from accreditation followup visits on assessment Centers on completion of student learning projects SCSU’s Assessment Academy Projects University Assessment System Assessment of Student Learning in Programs General Education Assessment www.stcloudstate.edu/assessment/academy.asp Goals of University Assessment System Project 1) 2) 3) 4) 5) Refine institutional assessment policies Communicate institutional assessment policies Build institutional assessment capacity Integrate assessment into work life of institution Implement assessment of assessment Goals of Project on Assessment of Student Learning in Programs 1) 2) Implement institutional assessment plan at level of major programs Implement institutional assessment reporting system at program level “Train the Trainers” Workshop, May 2007 Did initial planning for peer consultant training Facilitated by Academy mentor Elaine Klein 20 participants from across the campus Goals of General Education Assessment Project 1) 2) 3) 4) Approve the program Implement program structure Collect and analyze data Use data for improvement Goals of the Peer Consulting Program Improve program assessment across the institution Build assessment capacity Increase acceptance of assessment of student learning at the program level Learning Outcomes for Peer Consultant Training Those who complete this training will be able to: Articulate the basic principles of assessment Present the SCSU model of assessment Facilitate the assessment process in departments, programs, and units CONSULTANT EXPECTATIONS AND THE ASSESSMENT PEER CONSULTING PROGRAM Assessment Peer Consultants Assist with any aspect of program-level assessment Respond to requests from programs Work with at least one other peer consultant Are familiar with the resources available at SCSU and use these when appropriate Represent the SCSU Assessment Peer Consulting Program and the Assessment Steering Committee Peer Consulting Process Request submitted to Assessment Office Assessment Director contacts program to obtain additional information about the request Assessment Office selects two (or more) peer consultants with complementary backgrounds Peer consultants meet, then contact the program Peer consultants assist the program Peer consultants provide brief descriptive report to the Assessment Office Program evaluates the peer consulting service Peer Consulting Requests Peer Consulting Request form How will peer consultants be assigned? Can peer consultants decline assignments? What are some good reasons for doing so? Some Tips for Peer Consultants 1) 2) 3) 4) 5) 6) 7) Learn about the program before you visit. Focus on the problem as defined by program faculty/staff. Early on, ask questions that will help you understand the situation from their perspective. If you are not actively facilitating, observe and take notes. Help programs find a way to do assessment work themselves; don’t do it for them. Nudge the program in the direction of recommended policies, to the extent this is appropriate. Request help when needed. Peer Consultants as a Team Should there be a “lead” consultant? How will you share responsibility—drawing upon each other’s strengths, while allowing opportunities for each to learn? How will you communicate in the presence of program faculty/staff? • • • Support each other Avoid the appearance of “ganging up” on program faculty/staff Comment on partner’s line of action without offending program faculty/staff or the partner CONSULTANT TRAINING: ASSESSMENT PRINCIPLES Definition Assessment is an ongoing process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards; using the resulting information to document, explain, and improve performance. Assessment Assessment is an ongoing process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards; using the resulting information to document, explain, and improve performance. Walvoord’s Definition the systematic collection of information about student learning, using the time, knowledge, expertise, and resources available, in order to inform decisions about how to improve learning (Barbara Walvoord, Assessment Clear and Simple) Assessment’s core questions What do you want students to know/do/value as a result of completing your program? How do you know whether your students know/do/value these things? If they know/do/value these things…you’ve got something to tell! If NOT, you have useful information to guide changes to improve learning. Where do programs begin in developing an Assessment Plan? Consultants may assist: Agree on Mission Identify Program Goals MnSCU Work Plan MnSCU, SCSU, College Mission and Goals Professional/Accrediting Criteria Constituencies Recommendations Department Mission Department Goals Program Educational Objectives Standards and Elements ASSESS Student Learning Outcomes Changes to Enhance Student Learning Data Collection and Interpretation Strategies to Accomplish Outcomes Areas and Methods of Assessment ASSESSMENT – THE BOTTOM LOOP ASSESS Changes to Enhance Student Learning Data Collection and Interpretation Student Learning Outcomes STUDENT Strategies to Accomplish Outcomes Areas and Methods of Assessment Walvoord’s Three Steps of Assessment 1. Articulate your goals for student learning “When they complete our program, students will be able to . . .” (Barbara Walvoord, Assessment Clear and Simple) What should consultants know? At SCSU we say: Identify your student learning outcomes Students will <<action verb>> <<something>> Student Learning Outcomes • • • • Learner Centered Specific Action oriented Cognitively appropriate at the program level Possible Outcomes • • • • • Students will appreciate…. Students will be exposed to…. Students will demonstrate knowledge of the history, theories and applications of… Students will attend…. Faculty will provide students with opportunities to…. Possible Outcomes • • • • • Students will appreciate… Students will be exposed to…. Students will demonstrate knowledge of the history, theories and applications of… Students will attend…. Faculty will provide students with opportunities to…. COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Lower division course outcomes COMPREHENSION EVALUATION APPLICATION ANALYSIS SYNTHESIS KNOWLEDGE Cite Count Define Draw Identify List Name Point Quote Read Recite Record Repeat Select State Tabulate Tell Trace Underline Associate Classify Compare Compute Contrast Differentiate Discuss Distinguish Estimate Explain Express Extrapolate Interpolate Locate Predict Report Restate Review Tell Translate Apply Calculate Classify Demonstrate Determine Dramatize Employ Examine Illustrate Interpret Locate Operate Order Practice Report Restructure Schedule Sketch Solve Translate Use Write Upper division Course / Program outcomes Analyze Appraise Calculate Categorize Classify Compare Debate Diagram Differentiate Distinguish Examine Experiment Inspect Inventory Question Separate Summarize Test Arrange Assemble Collect Compose Construct Create Design Formulate Integrate Manage Organize Plan Prepare Prescribe Produce Propose Specify Synthesize Write Appraise Assess Choose Compare Criticize Determine Estimate Evaluate Grade Judge Measure Rank Rate Recommend Revise Score Select Standardize Test Validate Outcome: Reason Scientifically – Geology What is asbestos? Explain how the characteristics of amphibole asbestos make it more conducive to producing lung damage than other fibrous minerals. Given the formula Mg3Si2O5(OH)4, calculate the weight percent of magnesium in chrysotile. Outcome: Reason Scientifically – Geology Two controversies surround the asbestos hazard: (1) it is nothing more than a very costly fabrication all out of proportion to actual risk, or (2) it is a serious hazard that accounts for tens of thousands of deaths annually. What is the basis for each argument? What is your role as a consultant? ASSESSMENT – THE BOTTOM LOOP ASSESS Student Learning Outcomes STUDENT Strategies to Accomplish Outcomes Areas and Methods of Assessment Strategies to accomplish Identify the assignments and activities that demonstrate achievement of each learning outcome Consulting at this stage? Student Learning Outcomes Course 1 Course 2 x Course 3 Course 4 x x x x x x x x x x x x x Program Matrix Other x Walvoord’s Three Steps of Assessment 2. Gather evidence about how well students are meeting the goals. Direct measures Indirect measures “Evidence includes qualitative as well as quantitative information.” (Barbara Walvoord, Assessment Clear and Simple) Gather Evidence Using Direct Measures Directly evaluate student work “Exams, papers, projects, computer programs, interaction with a client, musical performance” (Barbara Walvoord, Assessment Clear and Simple) Gather Evidence Using Indirect Measures Asking students or alumni how well they thought they learned Tracking graduate school or job placement rates (Barbara Walvoord, Assessment Clear and Simple) Challenges in Gathering Evidence Recognize appropriate measures to use Data from multiple courses and instructors interpret, report, and utilize in decision making Relate to outcomes when course or instructorspecific assignments and activities are used Possible Consulting Requests Identify the traits of successful achievement of the outcome What are we looking for to be able to tell whether or not student performance is acceptable? Example OUTCOME: Gather factual information and apply it to a given problem in a manner that is relevant, clear, comprehensive, and conscious of possible bias in the information selected BETTER: Students will apply factual information to a problem. TRAITS: Relevance Clarity Comprehensiveness Awareness of Bias Possible Consulting Requests Identify characteristics of effective/accurate/successful performance for each trait • Scale or description for assessing each of the traits • Two to five-point scale for each trait Performance Rubric Performance Characteristics Traits Does not meet Expectations Meets Expectations Descriptions of Unacceptable, Acceptable, or Excellent Performance Exceeds Expectations Performance Rubric Business Management Performance Characteristics Team’s Customer Satisfaction Skills Punctuality Does not meet Expectations Some team members missed appointments or did not return phone calls. Meets Expectations All team members arrived on time for appointments and returned all phone calls promptly. Exceeds Expectations All team members were always early. Courtesy Some team members were not respectful of firm employees All team members were always courteous and respectful of all firm employees. All employees felt that The team members were very courteous and respectful and fully elicited their ideas. Communication Some team members did not communicate clearly during meetings and phone calls. The team members always communicated clearly during meetings and phone calls. The team members always made an extra effort to make sure that they understood us and that we understood them. ASSESSMENT – THE BOTTOM LOOP ASSESS Changes to Enhance Student Learning Data Collection and Interpretation Student Learning Outcomes STUDENT Strategies to Accomplish Outcomes Areas and Methods of Assessment Walvoord’s Three Steps of Assessment 3. Use the information for improvement Pitfall: Gathering data that no one will use “It is not enough to gather data about student strengths and weaknesses; you need information and hypotheses about the causes of student weaknesses.” (Barbara Walvoord, Assessment Clear and Simple) The Final Challenge Close the loop Use data – develop hypotheses Review assessment methods Change curriculum and/or instruction Possible consulting issues? Thinking Outside the Loop Accredited Programs Special Consideration? Programs? Departments? Colleges? Thinking Outside the Loop More Pitfalls: Mere compliance with external demands Triggering resistance and hostility of faculty Letting administrators do it Making the process too complicated (Barbara Walvoord, Assessment Clear and Simple) Thinking Outside the Loop It’s a passing fad It’s not my job or concern It violates academic freedom It violates student privacy You can’t assess what we do in academe I already assess - it’s called ‘grading’ I don’t have time to do assessment – workload! I don’t have the expertise Assessment might reveal “bad” news INSTITUTIONAL POLICIES AND RESOURCES SCSU Assessment Structure Organization chart Assessment Steering Committee Assessment Office Institutional Effectiveness Continuing Studies Student Life and Development Other areas SCSU Recommended Policies Assessment Plans Annual Assessment Reports These policies apply to all undergraduate major programs, graduate programs, and free-standing minor programs These policies have not been approved by the Faculty Senate, but they are recommended by the Assessment Steering Committee Assessment Plan Mission Student Learning Outcomes Program Matrix – shows which courses or outside-ofcourse activities will be used to assess each student learning outcome Timeline – identifies when (which year) these assessments will take place Annual Assessment Report Student learning outcomes assessed Methods and tools used to assess these outcomes Summary of findings for these outcomes Discussion of findings Use of findings for improvement Plan for next year Other Institutional Policies General education www.stcloudstate.edu/generaleducation/assessment.asp Upper division writing http://www.stcloudstate.edu/generaleducation/docpolicies/U pperDivisionWritingRequirement.asp Institutional learning outcomes? http://www.stcloudstate.edu/actionplanning/academic/workgr oups/outcomes.asp OFFICE OF INSTITUTIONAL EFFECTIVENESS (OIE) Office of Institutional Effectiveness (OIE) The Office of Institutional Effectiveness strives to facilitate the integration of planning, institutional research, and assessment to encourage the continuous and dynamic improvement of the University’s programs and services. The goals of the office include: Create a more balanced and transparent planning process Encourage broad discussion and understanding of university priorities Enhance accountability to internal and external stakeholders Expand use of data in decision-making & budgeting Encourage and support evaluation and assessment that leads to continuous organizational improvement OIE areas of responsibility Institutional Research: http://www.stcloudstate.edu/oie/research/default.asp SCSU Strategic Planning: http://www.stcloudstate.edu/oie/planning/process/default.asp SCSU Action Planning: www.stcloudstate.edu/actionplanning Minnesota State Colleges & Universities reporting Data & reports from OIE Enrollment reports Retention and graduation 10th & 30th day enrollments Student profile Common Data Set Fact Book and Department Data Reports Graduating Senior Survey Custom data requests (http://www.stcloudstate.edu/oie/research/datarequest.asp) Nationally normed reports from OIE National Survey of Student Engagement Self-reported data on student experience of freshmen and seniors Administered spring of odd years (‘01, ‘03, ‘05, ‘07) Collegiate Assessment of Academic Proficiency (CAAP) Measures critical thinking and writing Administered spring 2007 Collegiate Learning Assessment (CLA) Measures critical thinking and writing Administered fall 2007 (freshmen); spring 2008 (seniors) University Data System The University Data System (UDS) is being created to enhance the use of data in university planning and budgeting, to streamline access to disparate data sources for institutional decision makers, and to increase accountability to internal and external stakeholders. The development of the UDS will contribute to evidence-based planning and resource allocation, and to a culture of continuous improvement at SCSU. Cubes currently under development Instructional Workload (May 2008) Assessment (November 2008) Enrollment Management & Retention (May 2009) OIE staff & contact information Lisa Helmin Foss, AVP for Institutional Effectiveness lhfoss@stcloudstate.edu; 320.308.4028 Director of Institutional Research (vacant) Deborah Bechtold, Research Analyst dabechtold@stcloudstate.edu; 320.308.3039 David Sikes, Research Analyst dwsikes@stcloudstate.edu; 320.308.1746 Links to data and reports and information on the Office of Institutional Effectiveness www.stcloudstate.edu/oie STUDENT LIFE AND DEVELOPMENT ASSESSMENT 2007 Student Life and Development The Division of Student Life and Development has established learning outcomes and translated them into understandable outcomes for our students. Student Life and Development Learning Outcomes Diversity and Global Environment. The world we live and interact in is diverse and requires us to interact globally. To effectively engage in today’s world students will learn to… Student Life and Development Learning Outcomes Understand uses of power and nature of oppression, be aware of cultural and personal differences Trust and respect others, empathize, access culturally appropriate resources Be committed to cross-cultural communication, value social responsibility, be committed to social justice Student Life and Development Learning Outcomes Inter and Intra-Personal Competence. Developing awareness of ourselves and the way we interact with others are essential characteristics. The more we know about our strengths and weaknesses and the impact of our behavior on others, the better we are able to develop healthy relationships. To facilitate this growth students will learn to… Student Life and Development Learning Outcomes Gain knowledge about themselves, develop consciousness of self, selfconfidence, feelings of mattering, manage personal emotions, value cultural heritage Apply self-knowledge, practice self-worth , congruence, commitment, identify passions, discuss cultural differences and issues Work with others, practice collaborations, controversy with civility, engage across difference, be committed to ethical action Student Life and Development Learning Outcomes Engagement/Social Responsibility and Community Building. it is essential that we learn what it means to work together toward a common good. These skills are important whether on campus now or in our future communities. It is also imperative that we understand our responsibility as educated citizens in our community. Toward this end students will learn to… Student Life and Development Learning Outcomes Create effective change and practice collective efficacy Develop common purpose Develop civic awareness, value civic responsibility, practice engaged citizenship Develop a sense of connectedness within one’s communities Student Life and Development Learning Outcomes Cognitive and Intellectual Growth. The intellectual and practical skills we need are extensive, sophisticated and expanding with the explosion of new technologies and increasingly complex societies. As we progress through the educational system it is imperative to develop competencies that synthesize specific skills into higher level reasoning capabilities. To develop these qualities students will learn to… Student Life and Development Learning Outcomes Engage with others in constructive ways, engage in principled dissent; accept and appreciate other world views; manage conflict constructively Develop critical thinking/ decision making skills. Be open to change Foster lifelong learning. Integrate academic knowledge into all aspects of living. Student Life and Development Learning Outcomes Leadership Exploration and Development. In today’s world there are many situations in which we will find ourselves in roles requiring leadership skills. While these skills ultimately develop through training, practice and mentoring, understanding what it means to be a leader is multifaceted. To gain these skills students will learn to… Student Life and Development Learning Outcomes Develop personal leadership skills, set individual goals, practice risk-taking, delegate, serve as a role model, manage people and tasks, facilitate group processes View leadership as a process, not a position Student Life and Development Learning Outcomes Critically evaluate leadership models, develop group leadership skills, effectively lead change, develop team building skills, develop leaderships in peers, identify common purpose in groups, help groups set goals, apply problem solving strategies, value recognition and organizational sustainability Student Life and Development Learning Outcomes Life Skills Competence. Life teaches us many things, though learning to live with purpose and responsibility toward self and others can be quite challenging. Despite these challenges, living a satisfied and competent life provides fulfillment and direction for ourselves and those with whom we come in contact. In order to gain these skills students will learn to… Student Life and Development Learning Outcomes Develop effective communication skills and practice effective written and oral communication Develop wellness skills Develop life and career path Understand and develop professionalism and self responsibility As an SCSU student I will: Be globally-aware and support diversity. Be self-aware and strive for meaningful relationships. Be an engaged member of my various communities. Take full advantage of opportunities for learning. Explore leadership. Live a balanced and purposeful life. Student Life and Development Each department of the division is currently in the process of establishing departmental and programmatic learning outcomes and identifying methods of assessment. Student Life and Development The SLD assessment plan includes: a process for reviewing department assessment plans an Assessment Skills survey an audit of assessment practices department feedback process training and professional development to help support the faculty and staff skill sets Student Life and Development Essentially, the challenge is to provide support for the division to develop the competencies inherent to a culture of assessment. Student Life and Development The division must plan proper support for competency development and resources to inspire confidence and move the division forward with regard to assessment, measurement, and effective methodologies. Student Life and Development It is our intent to secure resources to jumpstart the required competency development; identify needs and assess our competency as a division; and provide ‘in house’ support to help each member of the division. Student Life and Development The division is eager to move forward, but has a limited number of personnel who are at a sufficient level of competency The motivation and expectation is present, but the need to measure who needs what support and to provide competency development is critical to an ‘all hands” effort. Student Life and Development This; in turn, sets the expectation and movement to make assessment principles and practices a mainstream part of all program entities in the SLD division and also provides leadership in leading SCSU in a integrated learning direction for the benefit of the students. Tools and Resources ACPA ASK Standards Needs Assessment Ten-Step Matrix in Student Affairs Assessment Ten-Step Matrix in Student Outcomes Assessment Assessment Practice in Student Affairs: An Applications Manual Student Life & Development Assessment website SWOT EXERCISE SWOT Exercise SWOT is a planning tool used to analyze the strengths, weaknesses, opportunities, and threats associated with meeting a particular strategic objective Strengths and Weaknesses are internal to the organization or unit Opportunities and Threats are external to the organization or unit SWOT Discussion Form groups of four. Your group should not include anyone else from your department, program, or unit. Describe to the others in your group one weakness of or threat to assessment in the program you analyzed. Lead a discussion of how peer consultants might respond to this weakness or threat. Allow an equal amount of time for each person (about 5 minutes each). Group Reports Which weaknesses and threats did you discuss in your small group? Which of these were hardest to deal with? Why? Do you see any interesting patterns that you would like to discuss? Conclusion Procedure for obtaining professional development funds from the Assessment Office Biographical information for website? Evaluation of the training What do you feel most and least confident about as an peer assessment consultant? Would you like additional professional development on any topic related to assessment consulting?