1 Cohort Report on Title III Academic Objectives “Focusing on Learning to Increase Learner Success” Angela Breckenridge 10/9/2009 Abstract This report is in partial fulfillment of the goals stated in Delgado’s Federal Title III Grant to improve student achievement, persistence, and retention through the institutionalization of practices piloted and assessed by a cohort who were specifically involved with the academic objectives of the Grant Activity. The report consists of a narrative explanation of the process, product, and impact associated with implementing certain of the academic objectives of the Grant as well as qualitative data elements related to evaluating those processes, products, and impact (U.S. Department of Education [USDE], 2004, pg. 58). Critical discourse analysis was used to determine findings and to categorize the several streams of institutional focus out of which there emerged recommendations for quality improvement and sustainable impact of the Grant activity. The report concludes with a comparative analysis of external data related to the findings and recommendations, including a proposal and timeline for institutionalizing the functions of the Title III Learning Outcomes Specialist. This report is limited to the data collected and controlled by the Learning Outcomes Specialist. It does not fulfill nor replace the Grant’s proposed Internal Evaluation Plan; which is to be designed by Institutional Research and prepared with internal data from the Title III Coordinator, Activity Director, Task Force, Department Chairs and Academic Deans, then submitted to the Title III Steering Committee for review and approval before presentation to the Chancellor and Executive Council (USDE, pg. 59-60). 2 Table of Contents Purpose ................................................................................................................................................................................................................ 3 Title III Academic Activity ........................................................................................................................................................................... 3 Implementation strategies facilitated by the Learning Outcomes Specialist .................................................................. 3 Summary ........................................................................................................................................................................................................ 3 From then to now ...................................................................................................................................................................................... 4 Implementation and Assessment Procedures .................................................................................................................................... 5 Process ................................................................................................................................................................................................................. 6 Title III Courses and Cohorts Spreadsheet ..................................................................................................................................... 6 Delgado SLO/LA Overview and Process .......................................................................................................................................... 6 The Delgado Outcomes/Assessment Methodology and Guidelines .................................................................................... 6 Proposal for a General Education Rationale .................................................................................................................................. 7 Cohort Final Pilot Report ........................................................................................................................................................................ 7 Findings from process data elements ............................................................................................................................................... 7 Product................................................................................................................................................................................................................. 8 “Designs for Learning” Course ............................................................................................................................................................. 8 Course Design Checklist .......................................................................................................................................................................... 8 The SLO/LA Database .............................................................................................................................................................................. 9 Master and Course Syllabus Template and Policy ...................................................................................................................... 9 The Student Learning Outcomes/GenEd Core Competency Matrix .................................................................................... 9 Workshop Presentations ......................................................................................................................................................................10 Findings from product data elements ............................................................................................................................................10 Impact.................................................................................................................................................................................................................12 Course Evaluations ..................................................................................................................................................................................12 Part 1 ........................................................................................................................................................................................................12 Part 2 ........................................................................................................................................................................................................13 Final Pilot Report Feedback ................................................................................................................................................................15 Other Areas Impacted by Services and Activities ......................................................................................................................17 Examining Research & Best Practice...............................................................................................................................................18 Findings from impact data elements ...............................................................................................................................................19 Summary of Results .....................................................................................................................................................................................20 Overall findings ..............................................................................................................................................................................................21 Quality Improvement & Sustainable Impact for Discussion ......................................................................................................23 Appendices .......................................................................................................................................................................................................28 3 Purpose The purpose of this report is to provide information derived from results of the Title III Academic Activity that (a) have bearing upon the strategic focus of the College or that (b) advance existing efforts to strengthen system-wide capabilities in effective planning, outcomes assessment, data analysis, and continual learning through scholarship and practice. Title III Academic Activity Implementation strategies facilitated by the Learning Outcomes Specialist Instigate comprehensive faculty development in creating learning outcomes, active/collaborative learning strategies, instructional design techniques, and classroom assessment strategies; Sustain faculty development over a four-year plan and beyond through mentorships and faculty-led workshops on creating learning outcomes, active/collaborative learning, instructional design techniques, and classroom assessment strategies; Revise and/or develop measurable student learning outcomes for 68 general education courses and freshman seminar over four-year period; Pilot learning outcomes and class assessment methods in these general education courses and freshman seminar in cycles over four-year period; and Institutionalize outcomes and active learning strategies for all GenEd courses over four-year period (USDE, pp. 14-36). Summary The Federal Title III Grant award provided funds to improve Delgado’s learningcentered mission through objectives designed to strengthen our learning-centered capabilities in our services and in the classroom as well as to continually improve the quality of learning for our students in each phase of their academic journey. To this end, the grant specified student services objectives as well as academic objectives to be implemented, evaluated, and institutionalized by faculty, staff, grant personnel, a Task Force and a Steering Committee. Ultimately, Delgado faculty, a learning outcomes and assessment specialist, and an activity director implemented the academic objectives and collected data relevant to assessing those objectives and to institutionalizing practices that resulted from the process. However, the originally proposed management and leadership support team is required in order to successfully institutionalize this work in a way the College sees 4 appropriate. The collaborative effort of the faculty cohort to achieve the academic objectives was intended to increase understanding of student learning in our classrooms. Indeed, these results and findings should enter into existing dialogues regarding organizational development, curriculum and instructional design, learning outcomes throughout the student life cycle, and the systematic use of evidence in planning and assessment at the College. In general, what we learned through the experience and what the data collected suggest is how critical it is to have a shared understanding of outcomes, assessment, and planning. This can be seen in the final results, which suggest the need for system-wide coordination, active and engaged leadership, competent support resources, and a wellcommunicated, meaningful set of guiding principles to serve as the foundation for Collegewide decision-making. The data also suggest recommendations for quality improvement in these areas in order for faculty and staff to successfully fulfill the College’s commitment to sustaining these initial processes, methods, and practices supported by the Federal funding of our Title III Grant. From then to now (see Appendix A, Implementation Recap) In August 2005, General Education (GenEd) Department Chairs selected 16 courses and instructors who taught them as the first cohort in a series of 5 pilots that was intended to begin in the fall of 2005. This did not occur, as just two weeks into that semester, Hurricane Katrina struck. After the storm, in that chaotic spring of 2006, Delgado faculty, staff, and students faced enormous challenges: all-important; all urgent. Among those challenges was the charge to implement the Title III objectives immediately. The alternative was to risk losing the award and its potential to increase student learning and retention. That summer of 2006, in the midst of finding students, rebuilding homes, and recovering from the devastation, 15 instructors agreed to participate in the first cohort. At that time, the prospect of institutionalizing outcome-oriented pilot syllabi through collaborative dialogue and College-wide participation was far-reaching, at best. Most faculty and staff were unaware of Title III; most pilot faculty and their colleagues were unfamiliar with outcomes assessment language and so were unclear about the purpose of enhancing course syllabi through the pilot program. But the first group stumbled their way through the process, trying to understand the language of outcome-oriented syllabus design and how to fulfill the directive we’d been given to pilot student learning outcomes and assessments at the course level for GenEd courses. Three-and-a-half years and 71 additional instructors later, the new language is more familiar to a few more people, and the focus on student learning outcomes has 5 become an expected—and in some cases a natural—mode of operation across many areas of the College. However, as we’ve seen, this understanding and mode of operation is sporadically distributed across the College and requires support from trusted and knowledgeable resources. This is an appropriate time to conduct another baseline survey to determine Delgado’s “assessment climate” for use in a longitudinal comparison of organizational knowledge and application of principles for planning, assessment, and the use of data for both. Implementation and Assessment Procedures (USDE, pg. 61) Objective as per Grant Implementation Strategies actual OutcomeOriented Course Designs Develop Outcomes Design Assessment Design Learning Activities Professional Development and Training Knowledge Sharing Active /Collaborative Learning Strategy Design Core courses with outcomes and active learning strategies All components 2000 Baseline as per Grant Zero Criteria as per Grant Data Elements as per Grant Data Collection Procedures actual Analysis Procedures as per Grant 14 per year 68 total Number of core courses with Outcomes Project Spreadsheet Overview & Process Guidebook GenEd Rationale Final Report Form Comparative Analysis ≧30% 20% and 60 core courses Application Usage Functional Audit Comparative Analysis Institutionalization ≧70% 20% Faculty satisfaction and recommendations Course Materials Syllabus Checklist SLO/LA Database Master Syllabus Policy GenEd Competency Matrix Workshop Presentations Course Evaluations Final Pilot Reports Areas of Impact Table Best Practice Analysis Institutionalization Zero 100% Funding, governance, and structure Program Audit Comparative Analysis Test of significance Comparative Analysis 6 Process Data elements for evaluating the process component of the Activity are listed in the grant as “documents of accomplishments, such as manuals and guides developed to support the implementation process, schedules and rosters developed to support the implementation process, discussions of process reflected through meeting minutes, and financial records” (USDE, pg. 58). Our implementation process yielded the following deliverables that are suitable for use by the institution and as data elements for project evaluation. Title III Courses and Cohorts Spreadsheet (see Appendix B) 86 faculty piloted 88 courses, representing 33 disciplines within the College, 23 of which were GenEd disciplines. The Grant called for cycling a total of 68 GenEd courses through the program and pilot process. The cohort exceeded that with a total of 71 GenEd courses and extended the impact of the Grant with an additional 17 courses from specific academic program areas. Delgado SLO/LA Overview and Process (see Appendix C) Two-semester implementation: Semester 1 – training, development, planning for pilot with support and Title III Resources Semester 2 – implementation of pilot and dissemination of pilot results throughout respective departments The process was adapted and modified with each successive cohort. Initially, the Grant specified pilot implementation to be carried out by Deans, the Task Force, and other instructors. This was one of the challenges in implementing the grant objectives. The cohort also experienced some administrative issues that required interventions outside the scope of the Title III staff or the faculty cohort. The Delgado Outcomes/Assessment Methodology and Guidelines (see excerpt in Appendix D) This comprises the Manual as required by the Grant and was designed for webbased navigation. It is intended to assist faculty in understanding, practicing, designing, and evaluating student learning outcomes, assessments, and learning activities at the course and classroom level. 7 The methodology is based upon current theory and practice in the fields of active/collaborative learning (Henscheid, 2006; Millis, 2008; Michaelson, 2002), classroom assessment techniques (Angelo & Cross, 1993; Huba & Freed, 2000), learning-centered practices (Weimer, 2002; Leamson, 1998), integrated course design principles (D. Fink, 2003), and outcomes/assessment process models (Stiehl & Lewchuk, 2002; 2005). Proposal for a General Education Rationale (see excerpt in Appendix E) It became evident early on that in order to connect the student learning outcomes to a broader system of educational goals, we needed a framework and methodology. This resulted in the use of Delgado’s GenEd Core Competencies as a means of mapping student learning outcomes to the overarching achievement of those goals. In doing so, it also became evident that many of the general outcomes, which faculty identified as important for students to achieve, were not explicitly covered by either Delgado’s GenEd Characteristics or Core Competencies. This led to many conversations about the GenEd outcomes and the possibility of having a College-wide dialogue on revising them to better align with current needs and practice. The GenEd Rationale is a discussion document, should the College wish to pursue further dialogue. Cohort Final Pilot Report (see Appendix F) In order to provide pilot faculty with a tool to help them facilitate dialogue with their colleagues and their department chairs, a final report template was designed. It was used by cohorts 2-5 and adapted based on feedback. It defines the course, its SLOs, their alignment with certain levels of GenEd core competencies, a critical analysis of assessments done during the pilot, final conclusions from the pilot, and recommendations for the course. Findings from process data elements Grant objective to design SLOs for 68 GenEd courses exceeded by o 4% in additional GenEd courses and o 28% in additional academic program courses Process for revising Master Syllabus and designing SLOs/Assessments documented, web-based, and ready for institutionalization1 o Requires presentation to appropriate committees and administrative bodies for discussion, approval, and communication 1 “An Outcomes Assessment Specialist and 4 academic advisors will be supported through Title III funds with Delgado assuming increasing costs beginning in year 3, to institutionalize the positions, reforms and new systems over the course of the grant period.” (USDE, pg. 28) 8 Product Data elements for evaluating the product component of the Activity are listed in the grant as “materials, publications, systems and installations, approvals and related artifacts” (USDE, pg. 58). Our implementation process has yielded the following artifacts that are suitable for use by the institution and as data elements for project evaluation. “Designs for Learning” Course (see Appendix G) The grant objectives specifically called for the design of a professional development program for faculty that increased capabilities in instructional design, use of active/collaborative learning methods, the application of learning theory in the classroom, and assessing student learning. The professional development component went through a dramatic evolution during the grant implementation, based on regular assessment and feedback from the cohort. It now consists of both online and face-to-face delivery as well as individual consultations with a designated resource. It follows a train-the-trainer model, which is in support of recent initiatives to distribute outcomes/assessment knowledge and skill throughout the organization via key point people in different functional and academic areas. These training components have also been absorbed into Delgado’s fledgling faculty development curriculum and have influenced the curriculum design in their emphasis on instructional design principles. The Delgado faculty development curriculum is structured by knowledge areas, one of which is theory based instructional design. Course Design Checklist (see Appendix H) In order to assess the effectiveness of the faculty development component and to determine the quality of course designs and syllabi, we have developed a course design checklist. Varied levels of understanding integrated course design using outcomes and assessment can be seen in the syllabi that faculty have revised through this program. This finding is in keeping with the notion that the process of strengthening capabilities in outcomes and assessment takes time and allocated resources. In other words, an instructor may or may not reach a level of understanding capable of designing a quality outcome-oriented syllabus after a short workshop or series of workshops. This point also becomes clear in the qualitative data analysis further in this report. 9 However, those instructors who had individual consultation in designing their courses were able to produce outcome-oriented syllabi at an expected level of quality. The SLO/LA Database (see sample interface pages in Appendix I) This is an interactive, web-based program for searching course SLOs, courses, and the levels of GenEd core competency they develop. The initial scope of this database included an attempt to follow the GenEd Assessment methodology by being able to search for courses, SLOs, and their corresponding levels of expected GenEd core competence in order to assist in identifying courses for assessment and from which to secure assessment artifacts. It is located on the faculty server. Master and Course Syllabus Template and Policy (see Appendix J) The Title III cohort spent six semesters revising and piloting what has become the newly approved standard for Delgado Community College’s master syllabus template, along with a revised policy and guidelines. This template corresponds with a system that was designed to align learning outcomes with our GenEd goals. It moves us from a very content-oriented style of our previous master syllabus to one that is closer to a fully integrated, outcome-oriented syllabus. Some recommendations made by faculty were too dramatically different from our traditional approach to be successfully integrated at this point. However, the approved template, guidelines, and policy embrace the outcome-oriented methodology and are a successful institutionalization of the cohort’s work. The Student Learning Outcomes/GenEd Core Competency Matrix (see excerpt in Appendix K) Early on in the implementation process, it became clear that for instructors to design meaningful student learning outcomes (SLOs) for GenEd courses, those SLOs must have substance and contribute to larger College and student goals. At the time we began the project, the disciplines that comprised GenEd did not all have clear academic goals to which courses within the discipline could connect in achieving an overarching learning outcome. What we did have were GenEd Characteristics and, from the Office of Institutional Effectiveness and the GenEd Assessment Committee, nine GenEd Core Competencies. The core competencies aligned with the LCTCS GenEd goals and could be defined in terms of proficiency levels. Using Bloom’s Taxonomy of learning to structure three progressive levels of proficiency for each of the GenEd Core Competencies, we were able to map pilot courses and SLOs to the level(s) of core competence for which a particular course SLO was 10 designed to achieve. This provided a framework from which to design the SLOs piloted, as well as to categorize levels of competence across GenEd courses for purposes of assessment. Workshop Presentations (see excerpts in Appendix L) As the implementation progressed, more instructors and program coordinators outside the GenEd scope sought support and guidance for designing SLOs, assessments, and syllabi for programs, those under review and those simply wishing to improve the quality of their program design. “Designs for Learning” provided instruction in course design methodology that could be used across disciplines and throughout the College. In response to the demand by many instructors for greater understanding of these concepts, the “Designs for Learning” course was condensed into a two-hour workshop focused on “Integrated Course Design.” In addition, three other workshops were designed and delivered on the three main campuses and also at both Northshore sites. These included “Understanding SLOs,” “Active Learning in the Classroom,” and “Understanding Classroom Assessment.” Findings from product data elements Training program based on a train-the-trainer model developed for understanding learning theory, SLOs, assessments and currently ready for institutionalization.2 o Supports successful implementation of Master Syllabus Revision initiative o Supports coordinated development of faculty and staff competencies in outcomes/assessment o Findings indicate a need for continued support and development throughout the collaborative and train-the-trainer process by a capable resource(s): Supports proposal for key “point people” with skills in learning theory, SLOs, assessments to serve throughout the College Syllabus checklist used in peer review to establish common set of understood standards. o Results of applying rubric to pilot syllabi suggest more instruction, support, and most of all collaborative input by peers to achieve quality. o Indicates value of collaboration, especially structured collaboration aligned with academic processes and systems. Interactive, web-based search engine designed for referencing and understanding how GenEd skills are built through learning in different courses and with particular SLOs. 2 “Changes in instructional strategies to engage students in the learning process will be institutionalized by project end. A web-based resource center and train-the-trainer model will support continued development and expansion of these initiatives across the College into the future.” (USDE, pg. 28) 11 o Suggested need for College-wide model to communicate with students, faculty, staff, and accrediting bodies the rationale for student learning outcomes at the course level. Revised Master Syllabus Template, guidelines, and policy designed and piloted by the Title III cohort and approved for institutionalization. o Suggested need for updating the rationale behind the Master Syllabus components. Findings also suggested need for open dialogue on GenEd mission, definition, and assessment in order to provide purpose and structure to designing course SLOs for GenEd courses – GenEd Rationale drafted for discussion. Findings suggested the need to “chunk” content from the faculty course and have available for delivery on all campuses and to all departments as needed. o Requires continued resource(s) skilled in outcomes, assessment, and instructional design principles. 12 Impact Data elements for evaluating the impact component of the Activity are listed in the grant as “analyses of student performance, persistence and retention, as well as faculty and staff satisfaction” (USDE, pg. 58)3. The overall impact of this implementation process can be seen in the satisfaction of faculty, their lessons learned, the actions they choose to take in their own teaching, recommendations based upon their experiences, and conclusions drawn from their pilot, all of which were also influenced by an understanding of the College’s history and culture. The following artifacts are suitable for use by the institution and as data elements for project evaluation. Course Evaluations When the grant activity began, it was understood from its record that a Task Force and Steering Committee would collectively design an evaluation and assessment strategy, including required baseline surveys as requested by then Chancellor Alex Johnson. A baseline survey instrument was chosen but not implemented. Neither the Task Force nor the Steering Committee has convened since 2006. It therefore became incumbent upon the Learning Outcomes Specialist and the faculty cohort to provide data and feedback in order to evaluate this portion of the activity. Evaluations were distributed at the end of each training segment (see Appendix M). For the first cohort, this took the form of a series of half-day workshops delivered by external consultants. We distributed evaluations for these workshops to each participant after each session. However, the assessment tool was not designed with the academic objectives in mind. It was therefore an ineffective tool for use in this report. By the second cohort, a better assessment tool was designed by the Learning Outcome Specialist and used with each successive cohort to assess the effectiveness of the training program. The resulting tool combined Likert scale evaluation with open-ended qualitative responses from participants. Part 1 – Likert Scale Survey Items (see Appendix N) The tool used for faculty to evaluate the workshops and “Designs for Learning” development component provided some relevant data on its success in achieving the Grant objectives; however, we did see flaws in the tool that impacted data reliability. Some of this 3 Data elements related to student performance, persistence and retention are not included in this cohort report. They will be included in a more comprehensive report on the Title III Activity, to be compiled by the Title III Director, Activity Directors, Institutional Research and an External Evaluator. 13 had to do with the manner in which the questions were asked, which were apparently interpreted differently by different instructors. But the ambiguity of the Likert scale also limited the ability of this tool to collect thoroughly sound data. That said, we can still see from the overall findings that, in general, faculty learned about and applied learning theory and active and collaborative learning strategies in the classroom as a result of the training. The following findings indicate percentages of individuals who agreed that the program helped them to do or to learn the following items: 93% 92% 91% 84% 96% 92% 93% - understand student learning outcomes - feel confident in creating an outcome-oriented course - understand how active/collaborative methods increase learning - use active and collaborative learning techniques in their course - learn something valuable from their colleagues - see the value that good course design can have on student learning - apply learning theory to instructional design The slightly higher findings that indicate instructors learned something valuable from their colleagues as a result of the program are consistent with other findings throughout the study. They confirm the conclusion that opportunities for faculty to collaborate are key components of a continual improvement system for faculty development. The slightly lower results in the area of using active/collaborative methods as a result of the program suggest at least two possible conclusions, based on their consistency with other findings. First, many faculty claimed that they used these types of methods in the classroom prior to Title III. This may be some of the reason that the program was less likely to influence their use of these methods. Second, other faculty expressed the challenges in implementing these kinds of methods and requested further training beyond what Title III offered. Part 2 – Plus/∆ Feedback (see Appendix O) This portion of the faculty evaluation asked a broad, open-ended question about what participants would keep and what they would change about the faculty development component of Title III. Respondents were to write what they would keep in the program in the left hand column of the form (Plus) and list what they would change in the right hand column of the form (∆). Data from this portion of the faculty evaluations focused on satisfaction, future training in the area of outcome-oriented course design and assessment, and the extent to which the academic goals were achieved with the faculty development component funded by the grant. In general, the findings from this portion of the evaluation were consistent 14 with other qualitative findings in this study and illustrated a clear message from participating faculty regarding three issues: the value they perceived in sustaining the particular curriculum offered through the program, the resources assigned to its design and delivery, and the structured collaboration they had with colleagues as members of the cohort. Of the 106 individuals who completed faculty evaluations of the program, 99 commented either on what they felt should be kept from the program and why, and/or offered suggestions for future changes to the program, creating 297 total data points to analyze. None of the comments suggested the program should not be sustained, nor did any comment suggest that the program was inappropriate for faculty development or a waste of time. In fact, with the exception of comments coded as “Obstacles to application” and a few that were irrelevant because the suggestions were already part of the program or impractical to administer, the ∆ comments provided helpful suggestions that could improve an otherwise worthwhile and sustainable program. (see Appendix P for summary of Plus/∆ findings) “Plus” Feedback – (207 data points) These data represent 70% of the total feedback provided by faculty participants. They reveal what faculty felt was valuable in the training program and what they wanted more of, what they believed should be sustained, and insight into why it should be sustained. What emerged from this feedback fell into three major categories: 1. Feedback on sustaining the curriculum and delivery of the program, including a resource capable of designing and implementing the type of curriculum in the program (112 data points, 38% of total feedback) 2. Feedback on the program’s application to teaching practice (55 data points, 19% of total feedback) 3. Feedback on the importance of sustaining the program’s structured opportunities for faculty collaboration (40 data points, 13% of total feedback) It could be said that categories 2 & 3 are embedded in category 1, but the sub-categories that emerged were varied enough to warrant these three as distinct, with related subcategories. The sub-categories, in turn, help understand what faculty felt was valuable within each of the major categories and provide insight as to why those things are worth sustaining. 15 “∆” Feedback – (90 data points) These data represent 30% of the total feedback provided by faculty participants. They reveal what faculty felt could be changed in the training program to improve it for future value. Some specific comments were based more on personal preference than on the value added to teaching and learning practices (i.e. online v. face-to-face delivery; the way the textbook was written). However, some of the participants’ detailed suggestions provided a valuable perspective with regard to scheduling future development sessions, structuring more collaborative opportunities, and providing more opportunities for practice with new concepts on their own course. What emerged from this feedback fell into three major categories: 1. Suggestions for the program’s curriculum/delivery, most of which built upon or suggested more depth of existing components of the program (51 data points, 17% of total feedback) 2. Suggestions on how the program could incorporate even more application to teaching practice, including suggestions on how to incorporate the program into existing institutional systems (28 data points, 9% of total feedback) 3. Suggestions on how to structure more opportunities for faculty collaboration (11 data points, 4% of total feedback) These data points do not critically disclaim the program nor indicate disapproval of its design. Because of this it may be that instructors who put their comments in the “∆” column did so because they saw a specific suggestion as a “change” related to furthering the design and improving the program. Therefore, even the data points in the “∆” column can be said to support the sustainability and institutionalization of the program and its recognized value. The summary of findings in Appendix P breaks down these major categories into more specific areas on which faculty found important enough to provide feedback for sustaining and improving the program. Final Pilot Report Feedback (see Appendix Q) Sections 2 and 3 of the final pilot reports comprise a significant portion of the qualitative data elements in this study. Discourse analysis was used to compile and summarize findings from the final reports. Coding structure emerged from a deductive process and yielded the following major themes that the cohort faculty found important enough to include in their final pilot reports. 16 1. 2. 3. 4. 5. 6. 7. Major Theme Observations of student responses to changes in teaching/learning Instructor recommendations based on learning instructional design Changes in teaching and learning in the classroom Learning about instructional design or teaching/learning Attitudes and thoughts about the program Obstacles to implementing effective teaching methods Recommendations for specific activities and assessments Occurrences 369 331 210 198 124 73 9 From these data we can see that the primary theme of import to the pilot faculty had to do with their conscious reflections and understanding about student learning, through the experimental process of observing through the pilot. The experience of the pilot, therefore, can be said to have generated reflections important enough for instructors to record them in a final report to share with others. Within each major theme, instructors further reflected on what they had learned through the pilot process. The following secondary themes were the most frequently noted within each major theme. Secondary Theme of Greatest Import Within Each Major Theme 1.1. Things that promote student learning 2.1. More collaboration 3.1. Assessment 4.1. Terms/Language 5.1. Improvement in instruction 6.1. Lack of time 7.1. Student-designed assignments Occurrences 146 105 57 72 31 53 2 In this secondary level of data, faculty reveal what they learned through the training and pilot process, as well as their reflections on the process and the products of what they learned. The data reveal strong evidence of learning about things that promote student learning, about assessment in general, of basic terms and new language. They communicated the benefits of—and indeed recommended more opportunities to—engage in collaboration with each other in order to learn from each other’s experiences and share practice. Also significant was the importance of time and resources in order to learn these new skills and to apply them. Ultimately, however, they saw positive results in the process which, according to their final reports, could be seen in an improvement in instruction, and many specific methods and techniques, most notably, student-designed assignments. 17 These findings suggest achievement of several overarching goals in the Title III Academic Activity that are designed to be institutionalized as the College sees appropriate: 1. The incorporation of learning-centered/outcome-oriented learning theory in the classroom 2. The use of train-the-trainer or collaborative methods of faculty development 3. The increased use and understanding of assessment methods in the classroom Other Areas Impacted by Services and Activities (see Appendix R) During the course of the academic activity, other areas of the College called upon the services of and instigated activities related to those provided by the Learning Outcomes Specialist. This activity has been recorded in a matrix that matches the functional areas and/or special programs of the College with the various services, expertise, and skills afforded the College through the Title III funding for a trained Learning Outcomes Specialist. Areas requesting services and support provided through Title III funding: Faculty/Staff Development Distance Learning/Instructional Design Learning Community Program Service Learning Program VC Student Affairs and Areas Within The Chancellor’s Office Information Technology Allied Health & Nursing The Quality Enhancement Plan Implementation LCT Linkage Grant for Social Entrepreneurship Massage Therapy Program ADOT Program English As a Second Language Program Grant through Early Childhood Education Program Math Department Individual instructors from academic programs Convocation Planning Professional Development Academy Online Professional Development Program Program Review The Leadership Connection Specific services, functions, and skills: Outcomes/Assessment Consulting Documentation/Guidelines Design Best Practice Research/Consulting Training Design and Delivery Curriculum Design for Workforce Training Instructional Design Methodology Consulting Competency Model Design Program Development and Review Facilitation services Business Process Mapping Data-based Planning, Training, and Support 18 This activity indicates not only an institutional need for resources such as those funded by the Title III grant, but an interest in and willingness to learn more about outcomes/assessment and planning and using data for decisions, instructional design, training, and best practices research and consulting. In addition, the Title III grant presupposed the need to institutionalize processes, practices, and methods instigated by the grant activity. In fact, it is the College’s agreement in accepting this Federal funding that certain functions will be appropriately absorbed into the natural operations of the College and sustained in a way that makes sense, given the organizational needs, culture, and environment. Findings generated from faculty feedback, the challenges to stating and assessing SLOs that were incurred during the grant Activity, the history of planning and assessment at Delgado, and the level of activity recorded above by other areas of the College led to the following conclusion: the need to institutionalize the support and special resources such as those provided through the Title III funding for a Learning Outcomes Specialist is not only reasonable but critical for ongoing organizational effectiveness. As the College is also bound to the contractual conditions of the Federal Title III funding, it would be sensible to institutionalize the functions assumed by the Learning Outcomes/Assessment Specialist. How, when, and by whom remain to be discussed. Examining Research & Best Practice The Role of Student Learning Outcomes and Assessment Coordinators The California Community College system faced similar challenges to those Delgado has faced in terms of accreditation standards that have required outcomes-based assessment for all areas of the institution. Requirements such as these, followed by the swift implementation and obligatory use of comprehensive information systems to support these directives, have left an enormous gap in capacity building for the complying institutions to stumble into, flounder about in, and eventually fill as best they can with the resources at hand. Consequently, the quality and purpose of the assessment process is understandably compromised by a conspicuous lack of adequate preparation, training, guidance, and support by a core of capable resources in the areas of outcomes and assessment. The CCC system conducted a study examining the “Role of Student Learning Outcomes and Assessment Coordinators in California Community Colleges” (The Accreditation and Student Learning Outcome Committee [ASLOC], 2007). The expectations and function of these LOA Coordinators is comparable to the LO Specialist funded through the Title III Grant and found to be a necessary component for understanding and achieving the requirements mandated by accrediting agencies for meaningful learning outcomes design and uses of assessment. The “Accreditation and Student Learning Outcome Committee” conducted the study with 80 19 participating schools in the system in the effort to address the challenges institutions have with institutionalizing LOA Coordinators and to make recommendations for addressing those challenges. The study culminated with an overarching conclusion similar to that which has been found in this report: that “without adequate resources, organization and training, outcomes assessment will not achieve its goal of improving teaching and learning” (ASLOC, pg. 23). They emphasize the necessity for institutions to communicate, organize, and develop processes and systems to adequately support the function of SLO Coordinators, which looks differently in different institutions. Of the participating institutions, many had assigned several SLO Coordinators rather than one administrative staff to assume the support for the entire organization. This seems to have its benefits, particularly when the roles have been carefully designed, job descriptions and accountabilities clarified, reporting structures determined, and compensation/release times/appointment durations clearly and appropriately determined. Here is a list of recommendations resulting from this study. They can be referenced in detail in Appendix S. Table 9: Summary of Recommendations for SLO Coordinator Positions (ASLOC, pg. 22) 1. In order for the SLO Coordinator position to be effective, its placement within the college organizational structure must be adequately defined and carefully considered. 2. The college must determine how it will assign responsibility for the different major areas of assessment: student services, library, and instruction (courses, programs, General Education and degrees). 3. A clear job description for the SLO Coordinator position is essential. 4. A clear selection process for the SLO Coordinator with a specified length of service will assist in making the position viable. 5. Clear lines of reporting and accountability make the position more successful. 6. The SLO Coordinator should be fairly compensated in some way for this work. 7. The process will not be successful without other significant dedicated resources. 8. SLO Coordinators need ongoing training in various aspects of their assignment. 9. SLO Coordinators and the assessment processes should be regularly evaluated. Findings from impact data elements Course evaluations revealed again the value in structured collaborative opportunities for faculty, key to continual improvements in instruction Course evaluations also revealed the need and desire for ongoing training opportunities in learning theory, action research, and applied methods of instruction and assessment Faculty feedback confirmed again the need to keep a resource for SLO design and assessment, to keep the quality of curriculum and delivery methods implemented through the Title III training program, and to plan more opportunities for faculty collaboration 20 Summary of Results The Title III Academic Activity achieved the following objectives through process: Revise and/or develop measurable student learning outcomes for 68 GenEd courses and freshman seminar over four-year period (achieved 71 + additional program courses) Pilot learning outcomes and class assessment methods in these GenEd courses and freshman seminar in cycles over four-year period; (piloted methodology and processes with five faculty cohorts) product: and Instigate comprehensive faculty development in creating learning outcomes, active/collaborative learning strategies, instructional design techniques, and classroom assessment strategies; (designed course, methodology, ancillary workshops, and consultations on demand) Institutionalize outcomes and active learning strategies for all GenEd courses and academic programs (achieved approval for revised Master Syllabus template, guidelines and policy as a result of faculty pilot) impact: Sustain faculty development over a four-year plan and beyond through mentorships and faculty-led workshops on creating learning outcomes, active/collaborative learning, instructional design techniques, and classroom assessment strategies; (plans to use faculty development program as ongoing part of Delgado’s faculty curriculum) 21 Overall findings Implications for institutional effectiveness Data reveal and best practice confirms that establishing organization-wide expertise and skill in Outcomes/Assessment is difficult; sustaining its effective application is even more difficult. Designing SLOs and assessments for courses, service and functional areas should not occur in isolation from the greater outcomes/goals/expectations of a program, discipline, GenEd core, community, accreditation bodies, student and the workforce. Understanding and designing SLOs and assessments for courses, programs, services, and functional areas requires a clear institutional direction and coordination of purpose Institutionalizing outcomes/assessment requires a set of principles for decisionmaking; systems that support, reward, and sustain continual improvement efforts; and a network of resources and ongoing development opportunities that facilitate effective practice. Implications for effective teaching and learning Faculty learned and were able to apply concepts related to instructional design, outcomes, and assessment as a result of the Title III Academic Activity. Faculty expressed understanding of the value of knowledge, skill, and attitudes related to effective instructional design, learning outcomes, and assessment. Data reveal that structured opportunities for faculty collaboration facilitate learning and increase application of best practices in teaching and learning. Faculty collaboration increases understanding of outcomes/assessment methods and practice in the classroom. Faculty expressed understanding of the value of collaboration and knowledge sharing in improving instruction. 22 Implications for effective faculty development One-shot training in outcomes/assessment does not guarantee understanding nor effective application of concepts taught. Faculty expressed and other data reveal that the curriculum offered through the Title III Academic Activity is worthwhile and important to sustain. Faculty expressed and other data reveal that resources capable of designing, delivering, assessing, and serving as a trusted resource in the area of outcomes/assessment is critical to institutionalizing the levels of competence required for effectiveness in teaching and student learning. Faculty expressed and other data reveal that the more applicable faculty development curriculum is to what they are doing in the classrooms, the more satisfied faculty are with development opportunities and the more effective they are in applying best practices in the classroom. Faculty expressed and other data reveal that they would rather learn from each other or from an “in-house” expert than from an external consultant or administrative staff who does not experience teaching in the classroom. 23 Quality Improvement & Sustainable Impact for Discussion Considerations Based on Findings Big picture – systems overhaul, building connections An internally developed organizational model, based upon Delgado’s traditional strengths and mission that would provide the rationale for overall College strategy and connect all functional, academic, and student service areas to a common purpose: o Plan and work out a schedule to continue work on defining stages, outcomes, and principles of the Student Life Cycle (Arnel’s work, Valencia, Title III SA Activity) and o Ensure all departments and functional areas contribute to and understand the model. Use of organizational model to implement planning and assessment methodology and to align all College initiatives to a common set of outcomes with a common set of guiding principles for decision-making o Coordinate a unified assessment plan, structure and schedule, beginning immediately Process outlined by Brett Heintz and Elizabeth Land, based on an assessment of prior processes and perceived needs for various areas Initiative from Debbie Lea involving assessment, key assessment points of contact Best practices/California model of implementation with Learning Outcome Coordinators Comprehensive review and revision of all committees and councils to align with the organizational model and to determine effectiveness measures based on accountabilities o Design a process based upon principles defined in the student life cycle o Use principles defined in the student life cycle to determine committee actions, planning, and assessment Comprehensive review and revision of current performance appraisal system so that it clearly reflects the organizational emphasis on evidence-based planning and assessment, learning-centered methodologies, and scholarly-practice through performance expectations, methods of appraisal, and the forms o Analyze student life cycle in terms of planning/assessment/outcomes written by each area o Determine organizational needs in order to succeed, define competencies, measures, and expectations o Research best practices Comprehensive review and revision of General Education mission, purpose, components, and assessments to give cohesion to SLO design o Best practice – Valencia’s process o Working papers – rationale 24 On the ground – communication/coordination Multiple forms of College-wide communication from leaders acknowledging and explaining the direction of the College o Announcement of Learning Outcomes Specialist transition plan (faculty’s work) o Announcement of program review changes (Brett/Liz/Debbie/Yvette’s work) o Announcement of planning and assessment changes, process (Brett/Liz/Debbie/Yvette’s work) o Announcement of data workshops (Chancellor’s charge, Tim/Cathy/Angela/Arnel’s work w/ Brett/Liz/Debbie/Yvette’s work) o Organizational structure – With outcomes/assessment coordinators (Debbie’s work) o Distribute endorsed “Outcomes Assessment Culture” statement for the College (Liz’s work) o Distribute principles of teaching, instructional design, and a College-wide definition of learning-centeredness (Cindy/Angela/Missy/Patrick’s work) o Define and document institutionalization process resulting from Title III Activity (Faculty’s work) o Distribute Title III results College-wide (Faculty/Staff’s work) Assessment of Convocation, use of data to revise purpose, structure, timing and duration o Research best practices o Review recommendations based on cohort data o Design a mission, purpose, and outcomes for Convocation that can be assessed Support for College-wide development of Planning and Assessment expertise in the form of training and compensated resources (Debbie’s work) o Review of best practices o Review recommendations based on cohort data o Design job description, reporting structure, accountabilities/evaluation, and expectations for Learning Outcomes Coordinators o Implement baseline survey and begin longitudinal study of burden: benefit o Document baseline data for establishing a “culture of evidence” Systems and processes in place to recognize, reward, share, and collaborate on planning and assessment projects o Review recommendation based on cohort data o Document baseline data for longitudinal study on use of data for decisions, planning and assessment o Establish a system for collaboration and knowledge management 25 Institutionalizing the Title III Learning Outcomes Specialist Function The institutionalization of the Title III Activity is broad, requires thoughtful consideration and examination of need and results of the program. However, it is clear that the College would benefit from many of the functions set in place with this Federal funding. Indeed, several proposals for improvements are directly related to the functions and services provided by Title III. Therefore, it would behoove the College to use the evidence available to support those proposals as institutionalized processes for those Title III functions we are required to institutionalize in some fashion. This could illustrate the extent to which the College capitalized upon the grant objectives for ongoing improvement efforts and would position the College as a viable candidate for future funding. In considering the institutionalization of the Title III Learning Outcomes Specialist Function, it should coordinate with existing structures and fulfill an existing need for those skills and services. The following options are suggested based upon the findings of this study as well as recent proposals for fulfilling needs in the area of outcomes/assessment support and organizational coordination/planning. Options for discussion: One-year transition of Learning Outcomes Specialist function from single source to multiple sources throughout the College Based upon Title III Activity results, best practices, and support for existing plans 1. Maintain current job description* until end of year 1 a. Continue to support faculty/staff in present projects DLIT – Faculty Development Data Driven Decision Initiative QEP – Assessment Process MATH – Faculty Development Assessment ADOT – Program Analysis Early Childhood Grant – Internal Evaluation Student Services – Student Life Map Service Learning – Assessment Plan Learning Communities – Program Development/Assessment Multiple Course-level Faculty Initiatives Leadership Connection Professional Development Academy b. Transition out - work w/ College on plan to distribute function Reporting structure Compensation/Duration Job description/Expectations (see Appendix T) Training/Development Accountabilities/Performance Appraisal 26 2. Assume new job description (see Appendix U) by the end of year 1 a. Continue to serve the College as SME and training source b. Transition in – work w/ College on organizational development *As described in the U.S. Department of Education’s Title III Grant for Delgado 27 References Accreditation and Student Learning Outcome Committee. 2007. Agents of Change: Examining the role of student learning outcomes and assessment coordinators in California Community Colleges. The Academic Senate for California Community Colleges. Fall. Angelo, T. A. and Cross, K. P. 1993. Classroom Assessment Techniques, A Handbook for College Teachers, 2nd Ed. Jossey-Bass: San Francisco. Fink, L. D. 2003. Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses. Jossey-Bass: San Francisco. Henscheid, J. M. 2006. Using active learning strategies in the classroom: Practices in the United States. Workshop presented at Japanese Annual Conference on General Education, Kanazawa, Japan. Huba, M. E. and Freed, J. E. 2000. Learner-Centered Assessment on College Campuses: Shifting the Focus from Teaching to Learning. Allyn & Bacon: Boston, MA. Leamnson, R. 1998. Thinking About Teaching and Learning: Developing Habits of Learning with First Year College and University Students. Stylus Publishing: Sterling, VA Michaelsen, L. K., Knight, A. B., Fink, L. D., 2002. Team-based learning: a transformative use of small groups. Westport, CT: Praeger Publishers. Millis, B. J., Cohen, M. W., O’Brien, J. G., and Diamond, R. M. 2008. The Course Syllabus: A Learning-Centered Approach. Jossey-Bass San Francisco. Millis, B. J. and Cottell, P. G. 1998. Cooperative learning for higher education faculty. Oryx Press :Santa Barbara, CA. Stiehl, R., & Lewchuk, L. 2002. The OUTCOMES Primer: Reconstructing the College Curriculum (2nd ed.). Corvallis, OR: The Learning Organization. Stiehl, R., & Lewchuk, L. 2005. The MAPPING Primer: Tools for Reconstructing the College Curriculum. Corvallis, OR: The Learning Organization. U.S. Department of Education. 2004. Application for Grants Under the Strengthening Institutions Programs CFDA #84.031A. Focusing on Learning to Increase Learner Success. Delgado Community College. Weimer, M. E. 2002. Learner-Centered Teaching: Five Key Changes to Practice. Jossey-Bass: San Francisco. 28 Appendices Appendix A: Implementation Recap ......................................................................................................... 29 Appendix B: Faculty Cohorts and Courses .............................................................................................. 34 Appendix C: SLO/LA Overview and Pilot ................................................................................................. 37 Appendix D: Outcomes/Assessment Methodology and Guidelines .............................................. 41 Appendix E: A General Education Rationale (excerpt)....................................................................... 47 Appendix F: Final Pilot Report Template Example .............................................................................. 58 Appendix G: Faculty Development Course Syllabus ............................................................................ 60 Appendix H: Syllabus Checklist for Major Instructional Components ......................................... 63 Appendix I: The SLO/LA Database Interface Pages from Slide Presentation ............................ 65 Appendix J: Revised, Approved Master Syllabus Policy/Template ............................................... 66 Appendix K: DCC GenEd Core Competency/SLO Matrix – Excerpt ............................................... 74 Appendix L: Sample Slides from Instructional Design Workshops ............................................... 77 Appendix M: Cohort Course Evaluation Form – Sample.................................................................... 82 Appendix N: Data Results from Likert Scale Survey Items ............................................................... 83 Appendix O: Raw Data Plus/Delta Feedback, Course Evaluation .................................................. 87 Appendix P: Summary Results – Plus/Delta........................................................................................... 95 Appendix Q: Data Results From Section III, Title III Final Pilot Report ..................................... 101 Appendix R: Functional Areas Impacted by Title III with Related Services and Activities 107 Appendix S: Distributing Student Learning Outcomes Coordinators ......................................... 109 Appendix T: Learning Outcomes Coordinator Position ................................................................... 112 Appendix U: Director of Learning Outcomes Assessment .............................................................. 114 29 Appendix A Implementation Recap Summer/Fall2006 *4 consultants hired for summer workshop series, focused on instructional design, active and collaborative learning, using learning theory in the classroom, and learning assessment *Met one on one with each cohort to review expectations, revised syllabi, and pilot plans. *Interviews become data elements Spring/Fall2007 *Title III pilot faculty present results of their work to colleagues at spring Convocation and 25 faculty are recruited for the 2nd cohort, extending beyond GenEd to include faculty and courses from several academic programs. *Faculty participate in a modified, online version of the previous summer’s series on instructional design, active/collaborative learning, learning theory, and learning assessment. *Online discussion forum becomes data element *Met one on one with each cohort instructor Fall2007-Spring2008 *As 2nd cohort pilots in the fall, the 3rd cohort begins the course component of the program with newly upgraded materials, self-assessments, guidelines, and small group presentations of topics from the text book *3rd cohort pilots 17 courses in the spring *Meet one on one with each cohort instructor *Note the need for deeper understanding of terms and application in writing SLOs – some of the previous cohort did not get it *Other instructors report transformations in their teaching and student learning Spring2008-Fall2008 *As 3rd cohort pilots in the spring, the 4th cohort begins course component of the program *4th cohort pilots ___ *Meet one on one with each cohort instructor *Note the need for Collegewide statement of assessment principles, methods, and process, as per confusion relayed by faculty and staff Fall2008-Spring2009 *As 4th cohort pilots in the fall, the 5th cohort begins course component of the program *5th cohort pilots ___ *Meet one on one with each cohort instructor *Note the need to define course SLOs within a context of the bigger picture, not in isolation 30 Summer/Fall2006 *Defined and implemented pilot plans and studentlearning outcomes for 15 GenEd courses, including CCSS, in the Fall. *Defined and implemented SLOs for Orientation *Note confusion as to what courses are actually GenEd courses, determine that our developmental and ESL courses counts as GenEd courses for the purposes of implementing the grant objectives Spring/Fall2007 *Defined and implemented sample master syllabi, using outcomes-oriented methodology for 25 courses *Presented rationale for involving T3 Task Force in institutionalization of course SLOs, Assessments. Task force does not meet again. *Focused cohort on revising and piloting outcomes-oriented master syllabus template as a purpose for their work Fall2007-Spring2008 *Cohort further modifies the master syllabus template *Pat Roux and I present to the deans for approval and are told to pilot the template further *SLO work done on some courses still not included on new courses going through curriculum, still using old format *As per faculty feedback and to use in implementing their pilots, proposed and was approved for 3 classrooms to be outfitted as seminar rooms, in conjunction with learning centered principles of small class size. The rooms provided for this were not appropriate for the purpose; some teachers did not want Spring2008-Fall2008 *Cohort further modifies master syllabus template *Instructors ask for assistance on rubrics *Note the need to re-assess outcomes for Orientation and assessment methods *Request to serve on DLIT planning committee, specifically to build a competency model for online faculty that can be used to align all faculty development along defined streams of knowledge, however, some confusion by team members on terms and how to use the model. Suggests, again, the need for a consistent set of principles and guidelines for determining the impact of an initiative and the reasons for doing it. Fall2008-Spring2009 *Final cohort product is presented to curriculum for use in all future master syllabus revisions, results in policy change and first documented institutionalization of cohort work *Request for assistance with assessment piece for LTC linkage *Request for assistance with PDA outcomes/assessment plan *Request for assistance with Service Learning Outcomes and Assessment *Request for assistance with QEP assessment plan *Request for assistance with individual instructors’ 31 Summer/Fall2006 *Created database to relate courses and learning outcomes with levels of GenEd core competencies *Designed SLO/GenEd Competency matrix to use in building a model for assessing GenEd Competencies *Designed recommendation for GenEd portion of course catalog *Began filming Spring/Fall2007 Fall2007-Spring2008 tables. Conclusion: determine 3 rooms that can be used as seminar rooms and booked only with classes that require that set-up (i.e. Honors courses, developmental courses, collaborative learning) *Facilitated SLO design *Institutionalization of with Student Services piloted syllabi still not groups and Learning regular nor planned by Communities respective disciplines *Confirmed impact of and programs cohort work on GenEd *Assessments piloted assessment: course-level by cohorts also not assessment would help extending beyond define discipline- and individual instructor in program- level outcomes T3, except on rare and and assessment informal occasions *Confirmed between individual commonalities between faculty T3 and QEP principles *Faculty begin to for standardizing request participation in SLOs/Assessments and the program, less Teaching methods at the recruiting necessary course level *English faculty reach *Feedback on impasse on 061/062 Spring2008-Fall2008 *Request by VC Lea to assist with LSD outcomes *Request by IT to assist with process mapping and strategic planning *Request by Chancellor’s office to facilitate process mapping *Request by nursing staff to analyze instructional design of syllabi, note the absence of outcomes and integrated learning activities Fall2008-Spring2009 courses: Culinary, Biology, Massage Therapy, English, ESL, Reading, History, Philosophy, Business *Request by nursing faculty development to deliver all-day workshop on course design *Created and delivered 2 hour workshops on integrated course design at all campuses *Request for assistance to prepare for “TracDat” entry: Physical Therapy *Note the need for guidelines and process for instructors who want to revise a course syllabus and present to curriculum (must be collaboration and outcomes examined within the context of other 32 Summer/Fall2006 short piece to help students, faculty, and the community understand the purpose of GenEd Spring/Fall2007 inconsistencies between the principles behind promotion/performance appraisal and those behind the learningcentered mission *Wrote communication plan, web content, and marketing plan for faculty recruitment *Created marketing brochures, presented Title III program at departmental meetings and on 3 campuses *Assessed first cohort experience and dramatically modified the development program and pilot process Fall2007-Spring2008 outcomes, divisions between City Park/West Bank *Modified online course to be offered as an alternative to face-toface workshops *Brought in MaryEllen Weimer to discuss learning-centered teaching *Assessed second cohort *Assessed third cohort experience, revised experience, revised course content, went course texts, binder back to face to face material, and created meetings as per PowerPoint feedback on the benefits presentation to of collaboration. *Course accompany workshop evaluations and final pilot sessions reports become data *Met with 3rd cohort elements mid-way through pilot, as per feedback on the necessity for support during the pilot. Grant Spring2008-Fall2008 *Begin to work with Learning Community Faculty on integrated course outcomes *Assessed forth cohort experience, revised binder material, structure of the workshop sessions, times and locations *Met with 4th cohort mid-way through their pilot to discuss assessments and support they might need. Feedback focused on the challenges of implementing a course design after a week of hurricane evacuation. Fall2008-Spring2009 courses and the program/discipline) *Many instructors voice concern at the possibility of not having a support person for outcomes/assessment *Note the need for institutionalization support, perhaps in committee charters 33 Summer/Fall2006 *Created manual, course materials, Blackboard site, presentation materials, evaluations, and templates for pilot syllabi and final reports Spring/Fall2007 *Distributed final reports to respective Department Chairs for institutionalization within their areas, offered assistance *Math department designs College-wide institutionalization process and assessment analysis *English department analyzes rubrics and exit exam *Very little faculty collaboration on pilot courses within respective departments, outcomes still not used when presenting new master syllabi to Curriculum committee Fall2007-Spring2008 states that department chairs, task force, and deans implement the pilot and learning assessment *Distributed final reports to respective Department Chairs for institutionalization within their areas, offered assistance *2nd and 3rd cohort populate SLO/GenEd Competency Assessment matrix *Feedback from faculty on the difficulty in getting colleagues together to discuss student learning and their work. Recommend convocation focus on examining results of learning assessments Spring2008-Fall2008 Discussed alternatives to starting semester when we do in order to ensure better student learning *Distributed final reports to respective Department Chairs for institutionalization within their areas, offered assistance *4th cohort populates SLOs/GenEd Competency Assessment matrix *Feedback from faculty on the need to revise GenEd Core competencies/characteristics to more appropriate and needed workforce skills Fall2008-Spring2009 34 Appendix B Faculty Cohorts and Courses # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 RUNNING TOTAL: 88 Discipline READ CCSS ENGL ENGL ENGL ENGL ENGL PHIL HUMA MATH MATH MATH MATH MATH MATH ACCT ACCT ADOT ADOT ADOT ADOT ADOT ADOT CADD CADD CULA HORT BIOL BIOL GEOL PSYC ENGL ENGL ESLR MUSC MATH MATH BUSG ARCH MATH Faculty FAVRET DEFFENDAL MCARTHUR FELTEY MITCHELL DIAZ GORVINE FINDON AUTHEMENT VILA SANTOLUCITO FRICKEY WILLIAMS ROUX ALBANO GATZKE LASKEY DUCLOS HOLLIDAY LAWRENCE LOGAN TOLLIVER GREEN ZERINGUE KRIEGER BURBACK ABBATE ROSENZWEIG RATARD WOOD HEINTZ BRYANT COSPER TYLER EDWARDS JOHNSON ROME PEMBO MIRZAI BICKSLER Course 072 107 221 205 206 102 101 101 105 118 128 130 091 095 120 218 205 101 106 105 265 264 178 125 201 105 101 101 102 101 240 222 212 023/027 105 095 096 250 110 203 Cohort Group 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 Date 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2006, Fall 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Spring 2007, Fall GenEd X X X X X X X X X X X X X X X X X X X X X X X X X X X X X 35 # 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 RUNNING TOTAL: 88 Discipline MATH BIOL BIOL BIOL PHYS ASLS FNAR PSYC PSYC HUMA ENGL ENGL HIST HIST READ SOCI PHIL ADOT ARCH BIOL BIOL BIOL CHEM CMIN CRJU FREN GEOL INTD MATH NURS BIOL CHEM CMIN ECON ECON ENGL FNAR FNAR FNAR INTD PSYC PSYC SOCI SPCH Faculty DOYLE ARCEMENT ARNAUD-DAVIS HURRELL DUPLESSIS KNOWLES NIOLET PRINCE-MADISON RAY CHOULDHURY ROSEFELDT HOWARD ANSELMO FONTANA DUERSON DEEL CAMAILLE WILSON MIRZAI PERRER WORRELL ANSPAUGH DODSON GILYOT WHORTON LOFTON WOOD SANDERS SAM WARNER VARNADO DODSON LATIOLAIS HILL GONZALES GAMBLE BREWSTER DAWES NIOLET SCANLAN BERRY DENNIS WYLLIE CLANTON Course 131 141 161 251 101 101 126 127 225 150 211 062 101 260 101 151 175 102 180 211 252 254 141 201 298 101 102 242 120 112 212 101 201 201 202 061 103 120 125 125 226 250 155 130 Cohort Group 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 Date 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2007, Fall 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Spring 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall 2008, Fall GenEd X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X 36 RUNNING TOTAL: 88 Cohort # Discipline Course Faculty Group 85 SPCH 230 LOUIS 5 86 SOCI 155 HALL 5 87 THEA 101 SANTOS 5 88 VISC 131 WALLEY 5 88 records = 86 instructors (Niolet and Wood piloted twice) Date 2008, Fall 2008, Fall 2008, Fall 2008, Fall GenEd X X X 37 Appendix C SLO/LA Overview and Pilot Title III Designs For Learning and Course Pilot Strengthening Our Capabilities in Teaching and Learning In August, 2005, General Education Department Chairs selected 16 courses and instructors who taught them as the first cohort in a series of 5 pilots that began the Summer, 2006 and that would continue over the next 4 years. A stipend and opportunity for development travel is included for the collaborating faculty in the pilot. In addition, faculty participants will have a learning plan to build upon competencies in active learning, collaborative learning, and classroom assessment techniques. The plan includes professional development in this course but the real learning occurs as innovative instructional design and learning assessments are put into practice in the classrooms. The ongoing support network and online forum will provide an environment for dialogue on learning as well as a record of the process that the cohort can use at the end of Phase 1 for recommendations in curriculum, teaching methods, assessment methods, learning outcomes, and professional development. The train-thetrainer method set down in the Title III objectives for this project speaks to our "one college" mission (identity? The word mission suggests it should appear in the College’s Mission Statement.) and the natural inclination that so many of our instructors already have to collaborate and learn from one another. Pilot cohorts mentor incoming faculty in the next group of courses selected in the process (Phase II), cycling through the learning plan with a third, fourth and fifth cohort until all GenEd courses and sections have been covered and all faculty who teach them have completed the program. While the cohorts are selected from a pool of full-time GenEd faculty, any instructor interested in participating is encouraged to join at any time. Contact Pat Roux pbroux@dcc.edu, the Title III Activity Director for Academic Objectives or Angela Breckenridge abreck@dcc.edu, the Learning Outcome and Assessment Specialist. Title III GenEd Learning Outcomes/Classroom Assessment Project Plan 2006 2007 2008 2009 Define SLO/LA for 14 Pilot 2/3 of the sections of Institutionalize all sections of Institutionalize all sections of core courses 1st 14 courses with Active 1st 14 courses with Active 2nd 14 courses with Active Learning Strategies to Learning Strategies to Learning Strategies to achieve increase in student achieve increase in student achieve increase in student learning and satisfaction learning, satisfaction, and learning, satisfaction, and course completion course completion Study Learning Pilot 1/3 of 2nd 14 courses Pilot 2/3 of the sections of Pilot 2/3 of the sections of Communities with Active Learning 2nd 14 courses with Active 3rd 14 courses with Active Strategies to achieve Learning Strategies to Learning Strategies to increase in student learning achieve increase in student achieve increase in student learning and satisfaction learning and satisfaction Pilot 1/3 of 1st 14 Define SLO/LA for 3rd 14 Pilot 1/3 of 3rd 14 courses Pilot 1/3 of 4th 14 courses courses with Active courses with Active Learning with Active Learning Learning Strategies Strategies to achieve Strategies to achieve to achieve increase increase in student learning increase in student learning in student learning Define SLO/LA for Design Learning Define SLO/LA for 4th 14 Define SLO/LA for 5th 14 2nd 14 courses Communities for more core courses courses courses 38 First Cohort and Course Selection Summer 2006 PHIL 101 - Charlie Findon HUMA 105- Rebecca Authement ENGL 221 - Janet McArthur ENGL 205 - Elizabeth Feltey ENGL 206 - Chris Mitchell ENGL 102 - Missy Diaz ENGL 101 - Cathy Gorvine READ 072 - Kathy Favret MATH 118 - Janet Vila MATH 120 - John Albano MATH 128 - Susan Santolucito MATH 130 - Dena Frickey MATH 095 - Pat Roux MATH 091 - Darlene Williams CCSS 107 - Melanie Deffendall Second Cohort and Course Selection Spring 2007 ACCT 205 – Beth Lasky ACCT 218 – Kim Gatzke ADOT 101 – Elaine Tolliver ADOT 105 – Krista Lawrence ADOT 106 – Ruby Holliday ADOT 178 – Cheryl Green ADOT 264 – Warren Duclos ADOT 265 – Mary Logan ARCH 110 – Victor Mirzai BIOL 101 – Amanda Rosenzweig BIOL 102 – Marceau Ratard BUSL 250 – Tara Pembo CADD 125 – Lorraine Zeringue CADD 201 – Ken Krieger CULA 105 – Nancy Burback ENGL 212 – Emily Cosper ENGL 221 – Janet McArthur ENGL 222 – Brenda Bryant ESLR 023/027 – Shelley Tyler GEOL 101 – Jacqueline Wood HORT 101 – Bettie Abbate MATH 095 – Jennifer Johnson MATH 096 – Pat Rome MUSC 105 – Steven Edwards PSYC 240 – Brett Hein Third Cohort and Course Selection Fall 2007 MATH 203 – Laila Bicksler MATH 131 – Chris Doyle BIOL 141 – Rashad Arcement BIOL 251 – Jim Hurrell PHYS 101 – Raymond Duplessis ASLS 101 – Leslie Knowles FNAR 126 – Mae Niolet PSYC 127 – Sandra Prince-Madison PSYC 225 – Sadhana Ray HUMA 150 – Shameem Choudhury ENGL 211 – Paul Rosefeldt ENGL 062 – Joe Howard HIST 101 – Sal Anselmo HIST 260 – Treg Fontana READ 101 – Linda Duerson SOCI 151 – Kellye Deel PHIL 175 – Angela Camaille 39 4th Title III Pilot Cohort – 2008 Timeline By Jan. 16, 2007 – new pilot cohort will. . . Be enrolled in the “Designs for Learning” Blackboard course for Title III Receive preliminary readings and initial assignment for 1st Cohort Session Receive contract and contract conditions to review and complete for 1st Cohort Session Designs for Learning 1st Cohort Session, Jan. 25 – new pilot cohort will. . . Submit contract and course registration forms Receive administrative orientation and materials (Lillie Fleury) Receive pilot orientation and materials (Angela) Discuss preliminary readings, complete self-assessment Present ideas and visions for student learning Designs for Learning 2nd – 6th Cohort Sessions 2hr sessions, times dependent upon instructor consensus 2nd session – Friday, Feb. 8 3rd session – Friday, Feb. 22 4th session – Friday, Mar. 14 5th session – Friday, Apr. 4 6th session – Friday, Apr. 25 Designs for Learning, Spring 2008 – pilot cohort will be responsible for. . . Facilitating one of several collaborative discussions based on the assigned textbook Actively contributing to the dialogue, using topics from the textbook and readings Bringing personal learning theory into classroom practice Designing Student Learning Outcomes/Classroom Learning Assessments for their course and communicating appropriately to others teaching that course (The Pilot Plan) Outcomes by end of Spring 2008 – pilot faculty will. . . Present a solid draft syllabus for pilot course to Deans, faculty Present their pilot timelines to Deans, faculty (either Summer or Fall course) Receive Stipend During course pilot, Summer or Fall 2008 – pilot faculty will. . . Implement their syllabus (Designs for Learning) Record observations and continue dialogue with cohort Report observations, reflections on process Communicate as necessary with colleagues who teach the same course, department chairs, etc. Observe one another’s classes, as appropriate, or request observation of their own class by 3 rd party Administer formative assessment methods and respond to feedback Week after Finals, Summer or Fall 2008 – pilot faculty will. . . Prepare final pilot conclusions, observations, results, and recommendations Meet with other pilot faculty to review reports, synthesize results, make recommendations to be presented to Vice Chancellor Outcomes by end of Fall, 2008 – pilot faculty will. . . Present final pilot conclusions, observations, results, and recommendations Receive stipend 40 Title III Pilot Cohort – Administrative Processes I. Getting Paid – What it is and How you get it a. What it is i. You will get a stipend for participating in this pilot and helping to implement the results ii. The stipend will come in two checks, one at the end of the 1st semester of the program, one at the end of the 2nd semester of the program iii. As of today, the checks will be in the amounts of $1,500 and $1,300 b. How you get it i. Complete the DCC Part-Time Contract COMPLETELY ii. Have everything signed as necessary iii. Turn it in to Lillie Fleury (Bldg. 1, 220E, CP, 671-6010) on or BEFORE 1/31/08 II. Professional Travel – What it is and How you get it a. What it is i. For each cohort, Title III will reimburse 3 professional travel leaves for conferences, seminars, or site visits that relate to Student Learning Outcomes/Assessment, Active/Collaborative Learning Strategies, and/or Applying Learning Theory in the Classroom. This means three people out of every cohort can choose an appropriate conference, seminar, or site visit to supplement the development part of this project. b. How you get it i. Send an email to Angela abreck@dcc.edu with the type of event and its relevance to the Title III project ii. Complete all standard travel requests and reimbursement forms as per Delgado’s operational guidelines and procedures iii. Have everything signed as necessary iv. Turn in to Lillie Fleury (Bldg. 1, 220E, CP, 671-6010) 41 Appendix D Outcomes/Assessment Methodology and Guidelines Strengthening Capabilities in Teaching Title III Professional Development Series Title III Learning Outcomes Resource Center Title III Learning Outcomes Specialist Professional Development Based on the results of a baselining survey conducted by Delgado’s Institutional Effectiveness department in September, 2005, faculty identified among other things the need for specific professional development in areas of learning-centered instruction. The Title III grant objectives directly respond to this feedback, focusing specifically on professional development in creating student learning outcomes, active and collaborative learning, instructional design techniques, classroom learning assessment strategies, and the relationship of scholarship and practice. Again, any faculty member is invited to participate in the development program, even though several select faculty have been chosen to participate in the first cohort and receive a stipend. Professional Development workshops and learning activities will strengthen our capabilities as a learning-centered college by pinpointing the following competencies and indicators: Creating Outcomes-Oriented Courses o Define meaningful, measurable student learning outcomes o Align course outcomes with GenEd core competencies o Align learning activities with course outcomes o Align classroom assessment strategies with learning activities and course outcomes o Adopt flexibility in learning activities that responds to assessment feedback o Use assessment feedback in collaboration with colleagues to assure and demonstrate progression of student learning across courses Applying Learning Theory to Instructional Design o Foster social connections in classroom, library, counseling environments o Design learning experiences that address students’ unique strengths o Design learning experiences that address students’ unique needs o Develop reciprocity and cooperation among students (interdependence and teamwork) o Include content well-suited to Delgado’s diverse* student population o Establish connections among students in and out of the classroom (learning communities) o Vary assessment measures and techniques to engage cognitive diversity o Create learning atmospheres that encourage all students to share view points o Use diverse* perspectives to engage and deepen critical thinking (diversity* as a learning resource) 42 o Develop student self-awareness (learning styles, personality types, assumptions) *Note: diversity has many dimensions, including age, culture, gender, race/ethnicity, socio-economic circumstances, learning style, education background, skill level, etc. that contribute to the many ways that individuals perceive and contribute to the world. Applying Active Learning Strategies o Employ techniques such as engaging lectures, discussions, experiential learning, scenarios, role-play, case study, problem-based learning, etc. o Employ collaborative and cooperative learning techniques o Encourage students to challenge ideas with reason o Integrate concrete, real-life situations into learning strategies o Invite student input on course outcomes (goals to achieve course outcomes, choice among assignment topics; in-progress student feedback. . .) Using Assessment Tools for Learning o Employ formative feedback loops early and often (both to and from students) o Provide students with written or face-to-face comments on strengths and weaknesses o Give timely feedback on class activities, exams, and papers o Design activities to help students refine their abilities to self-assess learning o Integrate self-assessment into course processes o Align summative evaluations with course outcomes and learning activities (appropriate to level of thinking; appropriate levels of performance) o Make assessment criteria public to students and colleagues o Evaluate effectiveness of assessment strategies and grading practices o Vary assessment measures and techniques to form a more complete picture of learning Engaging in the Scholarship of Teaching and Learning o Produce professional work that meets the Standards of Teaching Excellence* (course designs, action research projects, publications, etc.) o Build upon the work of others (consult literature, peers, self, students) o Be open to constructive critique (by peers, students, self) o Make work public to College and broader audiences o Demonstrate relationship between scholarship and improved teaching and learning processes o Expand College capabilities in learning-centeredness by sharing expertise with others (mentorships, leadership programs, faculty-led workshops, etc.) 43 Designs for Learning – Online Pilot Prep Course While initially focused for the GenEd pilot cohort, all faculty are invited to enroll in this Blackboard course and to participate in the dialogue, contribute, or browse this space focused on scholarly practice in learning and assessing learning. Your suggestions and feedback will make it your own, so be vocal with ideas on how to make it a meaningful space for knowledge sharing and collaboration with colleagues. Contact Angela Breckenridge if you need assistance in enrolling. abreck@dcc.edu Some ideas for the content of this site are: 1. Discussion Forums a. For specific disciplines, current programs b. For learning outcomes and assessment methods tried in the classroom c. Forums on specific areas of interest regarding teaching and learning d. General forum – The Social Café e. Focus Groups on Special Topics – The “Expert” Forum f. Assessment feedback results and recommendations 2. Resource Areas a. “Been Around the Block” – our valuable experience and learning b. Who is doing what that works? c. Travelers: meaningful things to share from off-site conferences and development d. Reference Links i. Active Learning 1. http://www.tlc.eku.edu/tips/ 2. http://vccslitonline.cc.va.us/MRCTE/active.htm 3. http://www.med.jhu.edu/medcenter/quiz/home.cgi?SMSESSION=NO 4. http://edweb.sdsu.edu/people/bdodge/Active/ActiveLearning.html 5. http://www1.umn.edu/ohr/teachlearn/tutorials/active/index.html 6. http://www.iub.edu/~teaching/faqdisc.shtml 7. http://www.criticalthinking.org/ 8. ii. Collaborative Learning 1. http://www.clcrc.com/ 2. http://www.dartmouth.edu/~collab/index.html 3. http://www.wcer.wisc.edu/archive/cl1/CL/doingcl/DCL1.asp 4. http://www.co-operation.org/pages/cl-methods.html 5. http://home.capecod.net/~tpanitz/ 6. http://www.sci.ccny.cuny.edu/~chemwksp/index.html iii. Instructional Design 1. http://www.ou.edu/idp/tips/ideas/diagram.html 44 iv. Classroom Learning Assessment and Other Assessment Links 1. http://faculty.mdc.edu/jmcnair/EME2040/behaviorist_lesson_plan1.ht m 2. http://www.siue.edu/~deder/assess/catmain.html 3. http://www.uleth.ca/edu/runte/tests/ 4. http://www.udel.edu/pbl/ 5. http://www2.acs.ncsu.edu/UPA/assmt/resource.htm#course_assmt 6. http://www.niu.edu/assessment/_resourc/gloss.shtml#5 7. http://people.jmu.edu/yangsx/ 8. http://people.jmu.edu/yangsx/ 9. http://www.uwstout.edu/soe/profdev/assess.shtml v. Rubrics 1. http://mh034.k12.sd.us/classroom_debate_rubric.htm 2. http://landmark-project.com/classweb/tools/rubric_builder.php3 3. http://projects.edtech.sandi.net/staffdev/tpss99/rubrics/rubrics.html 4. http://www.rubrics.com/rubric_examples.html 5. http://its.monmouth.edu/facultyresourcecenter/rubrics.htm 6. http://its.monmouth.edu/facultyresourcecenter/rubrics.htm vi. Student Services 1. http://css.rpgroup.org/ vii. Discipline-specific resources 1. Science, Math, Technology a. http://ublib.buffalo.edu/libraries/projects/cases/case.html b. http://www.flaguide.org/ c. 2. Developmental a. http://www.league.org/league/projects/remedial/index.htm 3. English a. http://www.siue.edu/ECPP/Statements/assessment.html viii. General Teaching Reference a. http://ericae.net/ b. http://adulted.about.com/od/icebreakers/ c. http://www.tltgroup.org/programs/Teach/Smart_Classrooms.ht m d. http://www.iub.edu/~teaching/names.shtml e. http://www.wku.edu/Dept/Support/AcadAffairs/CTL/db/quotes /index.html f. http://www.wvu.edu/~lawfac/jelkins/orientation/socratic.html g. http://ject.lib.muohio.edu/contents/contents.php?vol=13&num =2 45 h. http://its.monmouth.edu/facultyresourcecenter/FRCTrainingSeriesList.htm i. http://www.clt.cornell.edu/campus/teach/faculty/faculty.html j. http://www.lovedungeon.net/humor/college/index.html k. ix. Community College Reference 1. http://www.aacc.nche.edu/ 2. http://css.rpgroup.org/ 3. http://www.ncspod.org/ 4. http://www.diversityweb.org/ e. Manual on Learning Outcomes/Assessment f. How to interpret and respond to assessment feedback 3. Scholarship and Practice a. Pedagogy b. Teaching and Instructional Theory/Methodology c. CATT d. ELearning e. Discipline-Specific – i.e. DOLL, etc. Learning Outcomes Specialist Support for Individual SLO/LA Projects In addition to facilitating the outcomes/assessment projects, the Learning Outcomes Specialist Angela Breckenridge is a resource, an advocate, and a voice for faculty in all aspects of strengthening capabilities in teaching and learning. Specifically, the faculty Learning Outcomes Specialist can: Act as an "information center" for faculty Facilitate reviews of student learning outcomes in courses Support classroom learning assessment projects with individual faculty Serve as coach for classroom learning assessment projects with individual faculty Sponsor workshops for innovative teaching methods and other topics of critical interest Provide newspapers, magazines, books, and journals about professional development Survey faculty and staff to address professional needs regarding learning outcomes and classroom assessment Research and continually improve the faculty resources for learning outcomes and classroom assessment 46 Strengthening Capabilities in Learning Meaningful Rationale for General Education Introduction Defining General Education Criteria Of An Effective GenEd Program Characteristics Of Strong GenEd Programs Standard GenEd Disciplines and Outcomes Standard Competencies and Their Outcomes Best Practice Framework for Stages of GenEd Competency Development Guidelines, Examples, Influences for Consideration DCC’s Present GenEd Purpose and Assessment Plan Core Values and Campus-wide Learning Initiatives A Launching Point For New Dialogue: Suggested Framework for a Competency-Based, Holistic GenEd Program Student Learning Plan General Education Competency Model Terms and Definitions GenEd Core Competencies GenEd Characteristics GenEd Student Learning Outcomes GenEd Competency/Learning Outcomes Matrix GenEd Class Objectives GenEd Classroom Learning Assessment Learning Plans/Eportfolios GenEd Student Learning Outcomes/Learning Assessment Pilot 47 Appendix E A General Education Rationale (excerpt) General Education: a definition to work with (Adapted from LCTCS, Fall 2004) Two Types of GenEd GenEd Designed for Transfer includes courses listed in the catalog under GenEd Courses and may or may not be included in the College’s GenEd Core Applied GenEd Designed For Skills-Based, Non-Transfer Credit – includes skills, knowledge, and attitudes required to fulfill GenEd competencies, which may be included in the College’s GenEd Course offerings or in the learning outcomes stated for courses within major programs of study. General Education Core Program “General Education is a program and not just a collection of unrelated courses in different disciplines. It must be defined so that students think of it as a connective pathway to intellectual growth, not episodic potholes within a degree program.” General Education provides higher-level critical thinking skills, discovery, and problem-solving. More specifically, it provides literacy and fluency in various methods of communication in various languages. It promotes understanding of the existing scientific and mathematical views of the world and their impact on our global culture. It emphasizes the ethical demands of our common lives, demonstrates the importance of skills and knowledge of social and behavioral sciences to understand our contemporary world and to succeed with others. GenEd Core courses should have catalog definitions that explain how they: 1. have a recognized place in history: survey scientific paradigms, social patterns, development of reason and moral action; 2. reflect commonality of all humans in a multicultural world: universal concerns, human rights and liberties, value diversity within global interests; 3. prepare individuals for engaged citizenship, to deliberate and self-direct, calculate and reason logically. Criteria Of An Effective GenEd Program (Adapted from SACS) Institutions are required to: 1. Publish GenEd program requirements and provide a rationale with which to determine, “Why must I take this core general education course at this college?” This rationale should state how the GenEd program is linked to: College and program missions Student learning goals Student professional identity and occupational value Self-directed lifelong learning; 2. Offer a GenEd program that is a substantial component of each undergraduate degree; 48 3. Ensure breadth of knowledge (i.e. courses do not narrowly focus upon those skills, techniques and procedures specific to a particular occupation or profession) and provide an explanation of how this is designed; and, 4. Clearly identify competencies within GenEd core and provide evidence that graduates have attained those college-level competencies. Characteristics Of Strong GenEd Programs Give an explicit answer to the question, “What is the point of general education?;” Are based upon well-articulated competency paths and/or learning goals and outcomes; Strive for educational coherence; Relate to the major field of study; Reach beyond the classroom; Include assessment to monitor learning; Embody institutional mission and reflect its culture; Spring from, require and foster sense of community; Have strong faculty renewal and administrative leadership; Ensure continuing support for faculty; and Are designed carefully to permit continued evolution. Standard GenEd Disciplines and Outcomes (Adapted from LCTCS Guidelines for General Education Programs, Fall 2004) I) English Comp/Communication a) Analyze and evaluate oral and/or written expression by listening and reading critically for elements that reflect an awareness of situation, audience, purpose, and diverse points of view. b) Distill a primary purpose into a single, compelling statement and order and develop major points in a reasonable and convincing manner based upon that purpose. c) Demonstrate that the writing and/or speaking processes include procedures such as planning, organizing, composing, revising, and editing. d) Make written and/or oral presentations employing correct diction, syntax, usage, grammar and mechanics. e) Develop appropriate rhetorical patterns (i.e. narration, exemplification, process, comparison/contrast, classification, cause/effect, definition, and argumentation) and other special functions (i.e. analysis or research), while demonstrating writing and/or speaking skills from process to product. f) Manage and coordinate basic information gathered from multiple sources for the purposes of problem solving and decision making. 49 g) Recognize the use of evidence, analysis and persuasive strategies including basic distinctions among opinions, facts, and inferences. II) Mathematics a) Build upon (not replicate) the competencies gained through the study of high school algebra, geometry, and higher levels of mathematics. b) Utilize mathematics to solve problems and determine if the solutions are reasonable. c) Utilize mathematics to model real world behaviors and apply mathematical concepts to the solution of real-life problems. d) Make meaningful connections between mathematics and other disciplines. e) Utilize technology for mathematical reasoning and problem solving. f) Apply mathematics and/or basic statistical reasoning to analyze data and graphs. III) Natural Sciences a) Conduct an experiment, collect and analyze data, and interpret results in a laboratory setting. b) Analyze, evaluate and test scientific hypotheses. c) Utilize basic scientific language and processes and be able to distinguish between scientific and non-scientific explanations. d) Identify unifying principles and patterns in nature, and apply them to problems or issues of a scientific nature, recognizing the values of nature’s diversity. e) Analyze and discuss the impact of scientific discovery on human thought and behavior. f) Exhibit ethical behavior. IV) Humanities and/or Fine Arts a) Analyze significant primary texts and works of art (ancient, pre-modern, modern , and postmodern) as forms of cultural and creative expression. b) Explain the ways in which humanistic and/or artistic expression throughout the ages expresses the culture and values of its time and place. c) Explore global cultural diversity. d) Frame a comparative context through which they can critically assess the ideas, forces and values that have created the modern world. e) Recognize the ways in which both change and continuity have affected human history. f) Practice the critical and analytical methodologies of the Humanities and/or Fine Arts. g) Explore the ethical implications of cultural identity/cultural development/cultural integrity. h) Analyze historical fact and interpretations. i) Analyze and compare political, geographic, economic, social, cultural, religious and intellectual institutions, structures, and processes across a range of historical periods and cultures. j) Recognize and articulate the diversity of human experience across a range of historical periods and the complexities of global culture and society. 50 V) Social/Behavioral Sciences a) Recognize, describe, and explain social institutions, structures, and processes and the complexities in a global culture and diverse society. b) Think critically about how individuals are influenced by political, geographic, economic, cultural and familial institutions in their own and other diverse cultures and explain how one’s own belief system may differ from others. c) Explore the relationship between the individual and society as it affects the personal behavior, social development and quality of life of the individuals, the family and the community. d) Examine the impact of behavioral and social scientific research on major contemporary issues and their disciplines’ effects on individuals and societies. e) Using the most appropriate principles, methods, and technologies, perceptively and objectively gather, analyze, and present social and behavioral science research data, draw logical conclusions, and apply those conclusions to one’s life and society. f) Take ethical stands based upon appropriate research in the social band behavioral sciences. g) Analyze and communicate the values and processes that are used to formulate theories regarding the social context of individual human behavior in the social and behavioral sciences. h) Analyze historical facts and interpretations. i) Analyze and compare political, geographic, economic, social, cultural, religious, and intellectual institutions, structures, and processes across a range of historical periods and cultures. j) Recognize and articulate the diversity of human experience across a range of historical periods and the complexities of a global culture and society. k) Draw on historical perspective to evaluate contemporary problems/issues. l) Analyze the contributions of past cultures/societies to the contemporary world. Standard Competencies and GenEd Outcomes (Adapted from Institutional Effectiveness Associates’ Presentation, 2005) I) Basic Skills Competencies a) Reading b) Writing c) Speaking d) Listening e) Performing mathematical calculations f) Demonstrating basic computer skills II) Knowledge/Understandin g Competencies a) Historical perspective b) Literary styles c) Culture d) Meaning of numerical data e) Global perspective f) Impact of technology III) High Order Thinking Skills a) Critical thinking b) Logical reasoning c) Scientific/Abstract inquiry d) Concept integration IV) Values Development a) Commitment to democratic foundations 51 b) Respect for cultural diversity/cultural integrity c) Appreciation for aesthetics d) Awareness of self within local and global concerns Best Practice Framework for Stages of Competency (Adapted from Institutional Effectiveness Associates presentation material, 2005 and Anderson, 2005)) I) Novice – Level 1 a) Bloom’s Knowledge Cognitive Domain i) Observing and recalling information, knowing dates, events, places, major ideas and mastery of subject matter ii) Verbs for behavioral outcomes: list, define, tell, describe, identify, show, label, collect, examine, tabulate, quote, name who-when-what-where b) Krathwohl, Bloom, and Masia’s Receiving Affective Domain i) Passive but attentive II) Beginner – Level 2 a) Bloom’s Comprehension Cognitive Domain i) Understanding information, grasping meaning, translating knowledge into new context, interpreting facts, comparing, contrasting, ordering, grouping, inferring causes, predicting consequences ii) Verbs for behavioral outcomes: summarize, describe, interpret, contrast, predict, associate, distinguish, estimate, differentiate, discuss, extend b) Krathwohl, Bloom, and Masia’s Responding Affective Domain i) Complying and aware III) Competent – Level 3 a) Bloom’s Application Cognitive Domain i) Using information, using methods, concepts, theories in new situations, solving problems using required skills or knowledge ii) Verbs for behavioral outcomes: apply, demonstrate, calculate, complete, illustrate, show, solve, examine, modify, relate, change, classify, experiment, discover b) Krathwohl, Bloom, and Masia’ Valuing Affective Domain i) Behavior consistent with attitude IV) Proficient – Level 4 a) Bloom’s Analysis Cognitive Domain i) Seeing patterns, organizing parts, recognizing hidden meanings, identifying components ii) Verbs for behavioral outcomes: analyze, separate, order, explain, connect, classify, arrange, divide, compare, select, explain, infer 52 b) Krathwohl, Bloom, and Masia’s Organization Affective Domain i) Bringing together different values and building internally consistent value system V) Expert – Level 5 a) Bloom’s Synthesis Cognitive Domain i) Using old ideas to create new ones, generalizing from given facts, relating knowledge from several areas, predicting and drawing conclusions ii) Verbs for behavioral outcomes: combine, integrate, modify, rearrange, substitute, plan, create, design, invent, compose, formulate, prepare, generalize, rewrite b) Krathwohl, Bloom, and Masia’s Characterization Affective Domain i) Behaving according to moral “life style” and maintaining a consistent philosophy regardless of coercive surroundings c) Bloom’s Evaluation (integrated within competency stages) Cognitive Domain i) Comparing and discriminating between ideas, assessing value of theories, presentations, making choices based on reasoned argument, verifying value of evidence, recognizing subjectivity ii) Verbs for behavioral outcomes: assess, decide, rank, grade, test, measure, recommend, convince, select, judge, explain, discriminate, support, conclude, compare, summarize Guidelines, Examples, Influences for Consideration These guidelines, examples and internal/external influences are presented for comparison with the standards and expectations of other institutions and stakeholders to frame a valued public General Education program. While these sources may represent a different demographic and location from that of DCC, the message is consistent: that higher education, particularly the general education component, plays a broad and vital role in the success of students, professionals, and communities. We can learn from this feedback and make decisions to clarify the rationale for our own program, an action which will allow us to determine accurate learning outcomes for GenEd courses and more efficient assessment measures. I) LCTCS Guideline for GenEd Statement of Purpose: The purpose of a core GenEd is to “ensure that college students have the broad knowledge and skills to become lifelong learners in a constantly changing global community . . . general education core provides for students (and citizens as a whole) the means to a . . . more fulfilled and rewarding life. . . they are foundations of history and culture, serving to bridge the diverse heritages of a multicultural world, thus focusing upon the commonality of all humans and preparing individuals for their roles as socially responsible and engaged citizens.” (excerpted from LCTCS, 2004) II) Examples of GenEd Statements of Purpose South Louisiana Community College: . . .”general education” requirements represent a conviction on the part of the faculty that all students need to reason logically, solve problems, communicate effectively, and relate to the world around them. General education courses not 53 only enhance awareness of the world and the people in it but also foster an appreciation of the arts and humanities, encourage insight into the social and behavioral sciences, and provide a basic understanding of mathematical and scientific principles. The realistic expectations of a general education program are to empower the student with a reliable set of skills and understanding that move a lifelong learner forward in academia or the workforce.” Parish Community College: . . . considers “general education” to be a common body of skills, knowledge, and values to which all graduates (Associate and Academic Certificates) must be exposed, and for which the College shall determine certain levels of competency. These skills, knowledge, and values are representative of a common body of educational experiences that the College views as vital for enabling its graduates to be successful in today’s complex society.” What do all of these examples have in common? Do they seem to give an answer to “Why General Education here?” How does our own GenEd mission compare? Internal and External Influences on GenEd Statement of Purpose These few examples from the business community, public at large, and undergraduate students as well as our present circumstances in the wake of natural disasters offer some valuable input into the kinds of competence expected and needed from our graduates. Most of this competence can be gained through successful guidance and achievement of GenEd learning outcomes, but only if they clearly correspond to a focused program rationale. a) Business Community Expectations In a study done in Omaha, Nebraska, Metropolitan Community College leaders found that “employability skills” and “a strong work ethic” are traits most desired by business representatives, and the traits seen as most lacking among new hires. (Van Wagoner, 2004) In addition, they found that young people coming through the educational pipeline are woefully unaware of the nature of work, workplace expectations, and the demands of success. Most notably, employer’s regarded the following top ten skills necessary for, but lacking in, the workforce: Attendance and punctuality Follow-through Customer-service skills Professionalism Desire to learn Respect for diversity Ability to work as part of a team Time-management skills Oral communication skills Problem solving skill b) Public’s Expectations Ranking of “absolutely essential” goals of general education from NCPPHE poll: 54 Sense of maturity and [ability to] manage on [one’s] own (71%) Ability to get along with people different from self (68%) Problem solving and thinking ability (63%) High-technology skills (61%) Specific expertise and knowledge in chosen career (60%) Top-notch writing and speaking ability (57%) [but note that this is also a success factor for “getting along with people”] Responsibilities of citizenship (44%) [but note that this is also a success factor for “sense of maturity”] c) Students’ Expectations Ranking of Importance of GenEd Goals by U Mass. Amherst: % agreeing “somewhat or very important” (April, 1998). Learn on my own (98%) Write clearly and effectively (92%) Understand the relationship between ideas (92%) Think analytically and logically (91%) Awareness of other societies and cultures (90%) Awareness of American society (88%) d) Katrina Hurricane Disaster to impact internal/external needs in higher education in New Orleans Unknown student demographic and population Unknown specific needs from business community Unknown specific student needs for learning and professional growth Potential need for transparent critical information, leadership, coordination and followthrough on community development plans Potential need for partnerships with public school system, other institutions, local businesses, city planning commissions, all levels of government and neighborhood special interest groups Potential need for competencies in collaborative decision-making and action, high-level communication skills, self-direction, technology, customer success, critical thinking, creativity, and leadership Potential need for urban planning, civil and marine engineering, economic development, teaching, quality public information What broad-based, general competencies apply here? How does our program demonstrate commitment to these expectations?” How have we designed our program to respond to these needs? 55 DCC’s Present GenEd Purpose and Assessment Plan Mission Statement General Education at Delgado Community College prepares students to think critically, demonstrate leadership and be productive citizens through course offerings, core requirements and college-wide activities associated with the GenEd characteristics and learning outcomes. What would help communicate the College’s approach to General Education in relation to individual lifelong learning and community value? What roles do you see the graduates of the program taking? GenEd Requirements and Rationale as per Catalog [Within “Types of Degrees,” Delgado has a flexible, student-centered Associate of General Studies, which usually follows an articulation path with a four-year college and has to be worked out with an advisor.] [Within “Categories of Requirements,” Delgado’s GenEd courses are in English, mathematics, fine arts, humanities, natural science, and social science – required for most degree programs and include a few electives beyond the specific categories which the student can choose within the course list. Requirements Associate Degrees in Arts, Science, and Applied Science, Associate, and Certificate of Applied Science requirements for GenEd: GenEd course hours must be “successfully” completed; A proficiency exam in writing must be passed to earn credit in English 101; Students should be particularly careful about adhering to the catalog and curriculum in effect at time of admission (or change of major). Substantial Component of Degree Offerings [This should be confirmed. General Education Committee documents state that within DCC’s Degree Programs, 3 Degree/Certificates require 27 hours; one requires 15 and one requires 9. Course guide also stipulates that “students may not use a course in their major to fulfill their degree requirements,”. . .which is taken to mean that the general education requirement can only be fulfilled by successful completion of general education courses and cannot be fulfilled by successful completion of a specialized course within the major program. The chart below is from the 2006 catalog and reflects the different degree programs in which many Divisional GenEd requirements vary. If the Degrees had set levels of competency rather than course requirements across Divisions, there could be more flexibility on how students developed that core across courses.] 56 Degree GenEd Hours English Math Fine Arts Humanities Natural Science Social Science Associate of General Studies 30 9 6 3 3 6 6 Associate of Arts 27 “or” 21 “or” 24 6 3 3 6 “or” 3 6 3 “or” 6 Associate of Science 27 “or” 26 “or” 24 “or” 18 6 6 “or’ 7 3 “or” zero 3 6 “or” 4 3 Associate of Applied Science 19 “or” 18 “or” 15 “or” 12 “or” 9 “or” 6 3 “or” 6 3 3 “or” zero 3 “or” zero 3 “or” zero “or” 4 in PHYS 3 “or” zero “or” 3 in PSYC Certificate of Technical Studies 3 “or” zero 3 “Or” 3 “or” 3 in TECH Certificate of Applied Science 9 3 3 Curriculum Option for LPN – RN 18 6 6 3 “or” 3 3 3 Rationale for General Education [General Education at Delgado is not a program. It is rationalized by course requirements for majors and defined by general education characteristics. These characteristics explain specific outcomes the student “will have” as a result of completing the course requirements.] What rationale could be given for these requirements to justify their value and purpose in a student’s learning journey? What is needed to demonstrate how GenEd as a whole contributes to the College’s core values, the student’s lifelong development, and continual improvement to those ends? 57 Design To Ensure Breadth Of Knowledge Without Narrow Focus [The College’s GenEd design is partly demonstrated by DCC GenEd Characteristics listed below, but some feedback has suggested that a more defined notion of “breadth” related to the College mission would help to clarify how Certification programs benefit from GenEd as well as how GenEd prepares students to succeed in the Technical Competency Development with their Divisions.] How could the design of GenEd demonstrate the College’s learning-centered mission and attention to a variety of educational needs? 58 Appendix F Final Pilot Report Template Example Delgado Community College Student Learning Outcomes Pilot Report, Fall 2008 The purpose of this report is to communicate to departments and colleagues the work completed by Title III Faculty Cohort in learning about, designing, and piloting Student Learning Outcomes and Learning Assessment methods for courses involved in the Title III Grant’s Academic Objectives. The Title III Grant has provided resources for Delgado to pilot and incorporate Student Learning Outcomes and Assessment Methods on Master Syllabi as well as to examine and make recommendations for strengthening our Master Syllabi. The following conclusions and recommendations have been made by a Title III Instructor who has completed a course in Student Learning Outcome/Assessment Design and who has piloted a Master Syllabus Design with those objectives in mind. The following conclusions and recommendations are based upon the instructor’s findings from action research in the classroom and should provide a valuable source for building consensus on course-level Student Learning Outcomes and piloted assessment methods to be incorporated into the Master Syllabus. Faculty Cohort Name: Campus: Division/Department/Program: Course: Dean: Semester(s) Piloted: Part 1: Student Learning Outcomes What are students able to do as a result of learning in this course, and how does that align with broader program and institutional goals? This may or may not have changed due to your pilot. Update here as necessary. Student Learning Outcomes - (What are students able to do as a GenEd Competency/Characteristic - (From the Competency Matrix -- What result of learning in this course?) competency and level does this SLO help develop?) 59 Part 2: Assessment of Student Learning What methods did you use to determine whether students were learning/had learned? Student Learning Outcomes - (What are Learning Assessment Method - (What Explanation of Use - (How did students demonstrate students able to do as a result of assessment methods were used to the learning outcome? Was this an effective measure learning in this course – same as in the monitor learning and determine of student learning? Why/why not?) above table) achievement of this SLO?) Part 3: Conclusions and Recommendations for Master Syllabus – To Department and Colleagues who also teach the course. (What are your conclusions from the pilot and recommendations for the Master Syllabus of this course? What do you feel the next step should be in building consensus on SLOs/Assessments for this course? In piloting the Master Syllabus SLOs/Assessments with colleagues?) 60 Appendix G Faculty Development Course Syllabus Workforce Development & Education Non-Credit/Continuing Education Master Course Syllabus A. LECTURE/ LAB/CREDIT/CONTACT HOURS: 8 B. MAXIMUM COURSE ENROLLMENT: 40 C. SPECIAL FACILTY OR EQUIPMENT NEEDS/SAFETY RULES AND ISSUES: Orientation session for this course takes place on each campus as per available location. Concluding session for this course takes place at completion, time and location TBD. D. LAB FEE: N/A 1. COURSE NAME: Designs For Learning – Title III Student Learning Outcomes/Learning Assessment Pilot Preparation Course 2. COURSE PREFIX AND NUMBER: 3. COURSE DESCRIPTION: This collaborative online/face-to-face forum prepares instructors for piloting course designs based on the principles of Active Learning, Learner-Centered Assessment, and Outcome-Oriented Learning. Participants formulate a constructive and meaningful grading system, syllabus and course evaluation that provides valid guidance and feedback for continual improvement 4. PRE-AND CO-REQUISITES: Pre-reading material distributed prior to course. Full-time teaching experience at Delgado, familiarity with Blackboard and an online learning environment. 5. COURSE GOAL: To increase Delgado student learning by building upon existing strengths in learning-centered instructional design. 6. LEARNING OUTCOMES: As a result of learning in this course, individuals should be able to . . . 1. 2. 3. 4. 5. Create an outcome-oriented course Apply learning theory to Instructional Design Apply Active Learning Strategies Use Assessment Tools For Learning Engage in the Scholarship of Teaching and Learning 7. Unit Outcomes: Individuals achieve the learning outcomes by using the course content in the following ways. . . In Creating Outcome-Oriented Courses, course content will help: o Define meaningful, measurable student learning outcomes 61 o Align course outcomes with GenEd core competencies o Align learning activities with course outcomes o Align classroom assessment strategies with learning activities and course outcomes o Use assessment feedback in collaboration with colleagues to assure student learning align across courses In Applying Learning Theory to Instructional Design, course content will help: o Create ways to foster social connections in classroom, library, counseling environments o Design learning experiences that address students’ unique strengths, needs o Develop reciprocity and cooperation among students (interdependence and teamwork) o Include content well-suited to Delgado’s diverse student population o Establish connections among students in and out of the classroom (learning communities) o Vary assessment measures and techniques to engage cognitive diversity o Create learning atmospheres that encourage all students to share view points o Use alternative methods to engage and deepen critical thinking o Develop student self-awareness (learning styles, personality types, assumptions) In Applying Active Learning Strategies, course content will help: o Employ techniques such as engaging lectures, discussions, experiential learning, scenarios, roleplay, case study, problem-based learning, etc. o Employ collaborative and cooperative learning techniques o Encourage students to challenge ideas with reason o Integrate concrete, real-life situations into learning strategies o Invite student input on course outcomes and assessments (goals to achieve course outcomes, choice among assignment topics; in-progress student feedback. . .) In Using Assessment Tools for Learning, course content will help: o Employ formative feedback loops early and often (both to and from students) o Provide students with written or face-to-face comments on strengths and weaknesses o Give timely feedback on class activities, exams, and papers o Design activities to help students self-assess their learning o Align summative evaluations with course outcomes and learning activities (appropriate to level of thinking; appropriate levels of performance) o Make assessment criteria public to students and colleagues o Evaluate effectiveness of assessment strategies and grading practices In Engaging in the Scholarship of Teaching and Learning, course content will help: o Produce professional work that meets the Standards of Teaching Excellence (course designs, action research projects, publications, etc.) o Build upon the work of others (consult literature, peers, self, students) o Be open to constructive critique (by peers, students, self) o Make work public to College and broader audiences o Demonstrate relationship between scholarship and improved teaching and learning processes o Expand College capabilities in learning-centeredness by sharing expertise with others (mentorships, leadership programs, faculty-led workshops, etc.) 7. COURSE CONTENT: The course content includes a standard textbook, background reading material, short instructional handouts, and articles on learning-centered practices. It also includes your own course syllabus, the Delgado course guide, and a General Education Core Competency Matrix to use in developing Student Learning Outcomes. Topics follow those in the Table of Contents of the textbook, but the forum is not 62 limited to those topics. Finally, the discussions that emerge from your own ideas, reflections on teaching and learning, and shared observations provide a source of invaluable, collaborative content in this type of format. 8. TEXT: “Learner-Centered Assessment on College Campuses” by Mary Huba and Jann Freed; selections from “The Outcomes Primer” and “The Assessment Primer” by Ruth Stiehl and Les Lewchuk. 9. ASSESSMENT: Definition: Assessment is the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning. Self-Assessment of Learning-Centered Practice – completed at the beginning and the end of the course to determine any strengthening of these capabilities; provides the participant with important information for reflective teaching as well as for achieving the learning outcomes of this course. Informal Feedback – through the online discussion, you will receive informal feedback from peers through the process of your conversation and as you talk about how you apply learner-centered concepts in your classes. In addition, the lead facilitator will provide one on one feedback related to the online discussion, individual work, classroom observation, or in one-on-one consultation as requested by the participant. Formal Feedback – rubrics will be used to examine SLOs formulated in the course. Participants will receive formal feedback on SLOs, Course Designs, and Pilot Plans or classroom observations in the form of peer review/responses and rubrics used by participants as well as the lead facilitator. Facilitator Summaries – participants will be divided into small work-teams and select one of the textbook chapters as a topic of focus for the group. When the group’s chapter is scheduled for discussion, the group is responsible for facilitating the discussion, engaging all participants in the dialogue, and following up the topic discussion with a summary. This summary is a synthesis of the main themes found in the discussion as well as reflections on what those themes mean, what valuable ideas were shared, and any action taken by any participant that emerged from the topic of discussion. These will be posted for feedback from participants as well as from the lead facilitator. Deliverables – participants will design a Learner-Centered syllabus and a pilot plan for use in an initial course pilot. Participants will have the opportunity to revise drafts and collect feedback on these items prior to their final presentation to department heads and other colleagues for their information and comment, if appropriate. These assessments should provide ample results to continue or to improve learning in this course. The intention is to “set everyone up for success.” 63 Appendix H Syllabus Checklist for Major Instructional Components Student Learning Outcomes Checklist Yes No Yes No 1. Do the SLOs begin with an active verb? 2. Are the SLOs written as outcomes rather than as objectives? Language indicates an important overarching concept versus small lesson or chapter objectives Outcomes address what a student will be able to do at the completion of the course, instead of what they will be doing in the course SLOs address student competency rather than content coverage. 3. Do the SLOs address the expected level of learning for the course using Bloom’s Revised Taxonomy as a guideline? (i.e. remembering, understanding, applying, analyzing, evaluating, creating) 4. Will students understand the SLOs and find them meaningful? 5. Are the SLOs appropriate for the course? Consistent with the curriculum document of record Represents a fundamental result of the course Aligns with other courses in a sequence, if applicable Represents collegiate level work 6. Do the SLOs suggest or identify an assessment? Learning Assessment Checklist 1. Does the assessment adequately evaluate academic performance relevant to the desired outcome? (validity) 2. Does this assessment tool enable students with different learning styles or abilities to show you what they have learned and what they can do? 3. Does the content examined by the assessment align with the content from the course? (Content validity) 5. Will the assessment provide information at a level appropriate to the outcome? (Bloom’s) 6. Will the data accurately represent what the student can do in an authentic or real life situation? (Authentic assessment) 7. Is the grading scheme consistent; would a student receive the same grade for the same work on multiple evaluations? (Reliability) 8. Can multiple people use the scoring mechanism and come up with the same general score? (Reliability) 10. Is the assessment summative or formative - if formative does it generate diagnostic feedback to improve learning? 11. Is the assessment summative or formative - if summative, is the final evaluation built upon multiple sources of data? (AAHE Good practice) 12 If this is a summative assessment, have the students had ample opportunity for formative feedback and practice displaying what they know and can do? 13. Is the assessment unbiased or value-neutral, minimizing an attempt to give desirable responses and reducing any cultural misinterpretations? 14. Are the intended uses for the assessment clear? (Grading, program review, both) 64 15. Have other faculty provided feedback? 16. Has the assessment been pilot-tested? 17. Has the evaluation instrument been normed? 18. 19. 20. Will the information derived from the assessment help to improve teaching and learning? (AAHE Good Practice) Will you provide the students with a copy of the rubric or assignment grading criteria? Will you provide the students examples of model work? Teaching and Learning Activities Checklist 1. Do they give students opportunities to practice before evaluation? 2. Do they give students opportunities to learn in different ways? 3. Do they give students opportunities to DO something with the course content? 4. Do they give students opportunities to reflect upon their learning? 5. Is the student or the instructor using the course content? 6. Are students interacting with each other and with the instructor? Yes No 65 Appendix I The SLO/LA Database Interface Pages from Slide Presentation Developing Core General Education Competencies Increasing Student Success In The Job Market Overview Oral/Written Communication Computation/ Problem Solving Applied Technology Search Cultural Value & Awareness Citizenry & Social Values Development Leadership/ Professional Identity Logical Reasoning/ Critical Thinking Navigation Options: 1. Select a button above to see an Overview, visit a Competency Domain, or Search 2. Select a button to the right to access teaching/learning resources Sample SLO Teaching Methods Sample SLO Assessment Rubrics (Next Page is what user sees if he clicks on “Overview” above) Classroom Assessment Techniques Critical Thinking Across Domains Technology Across Domains Professionalism Across Domains Developing Core General Education Competencies Increasing Student Success In The Job Market Oral/Written Communication Computation Problem Solving Technology Diversity Awareness Citizenry Social Value Professional Identity Overview Search Navigation Options: 1. Select a button above to visit a Competency Domain 2. Select a button to the right to access teaching/learning resources 3. Select a link below to navigate this topic Overview (bullets Sample SLO Teaching Methods below are links) • Learning-Centered Framework for General Education Core Sample SLO Assessment Rubrics • Introduction to Competency-Based Learning • General Education Core Competency Domains Classroom Assessment Techniques • General Education Characteristics • General Education Student Learning Outcomes Critical Thinking Across Domains • Using the Framework • Learning Outcomes Project Request Form • General Education Student Learning Outcomes/Learning Assessment Pilot Technology Across Domains Professionalism Across Domains (Next Page is what user sees if he clicks on “Oral/Written Communication” above) 66 Appendix J Revised, Approved Master Syllabus Policy/Template Policy No. AA-1503.1A POLICY & PROCEDURES MEMORANDUM TITLE: MASTER SYLLABUS EFFECTIVE DATE: December 7, 2004* (*Title Updates: 1/30/08) CANCELLATION: AA-1503.1 (6/20/95) OFFICE: Academic Affairs (AA) POLICY STATEMENT It is the policy of Delgado Community College to have and maintain an up-to-date master syllabus for each credit course currently taught at the College. Master syllabi are developed and managed in accordance with Louisiana Community and Technical College System Policy I.1.034, Participatory Curriculum Development, as well as the guidelines and procedures outlined in this memorandum. PROCEDURES & SPECIFIC INFORMATION 1. Purpose To publish policy and procedures for the development and management of the master syllabus for each course carrying credit hours offered by Delgado Community College. 2. Scope and Applicability This policy and procedures memorandum applies to all faculty and academic units of the College. 3. Responsibilities Under the authority of the Board of Supervisors for the Louisiana Community and Technical College System and with the leadership of the Vice Chancellor for Learning and Student Development, Division Deans, and the Curriculum Committee of the 67 College, the faculty develops, maintains, and uses the appropriate master syllabus for teaching credit courses. 4. Components A syllabus is an approved plan for the content, instruction, assessment, and management of credit courses. A master syllabus is the approved syllabus for a course which must be used for all sections of the course, wherever offered and regardless of campus or delivery system. The master syllabus is the basis of all course syllabi. In developing the course syllabus--which the faculty member must give to students at the start of each semester--the faculty member must include the major components of the course as outlined in the master syllabus. Faculty members are encouraged to enhance and individualize their course syllabus. Faculty members are encouraged to include policies and procedures regarding attendance, discipline, etc., in their course syllabi. Each instructor's course syllabus must be submitted to the academic unit supervisor every semester for review at least once each academic year. 5. Development of Master Syllabi A master syllabus is developed by the faculty member(s) responsible for a particular course. After approval by the appropriate Division Dean(s), the syllabus is presented to the College Curriculum Committee for review. Sufficient justification should be provided to plan for the new course and for faculty/administration evaluation of the new course. The Committee can (1) recommend approval; (2) recommend approval with amendments; or (3) reject the proposal with or without recommendations. The Committee makes its recommendations to the Vice Chancellor for Learning and Student Development who gives final approval. 6. Management of Master Syllabi Full-time faculty members are expected to participate and contribute to the development of master course syllabi in their discipline. If major changes in a syllabus are necessary, the syllabus must be treated as a new course syllabus and must go through the curriculum review process. The Vice Chancellor for Learning and Student Development makes the final decision to accept a master course syllabus. Once the master syllabus is approved, it is entered into the official master syllabus file. The official master syllabus file, which is maintained by the Director of Curriculum and Program Development, contains the official documents used for inter- and intrainstitutional communication about course content. At least once every five (5) years faculty and academic administrators must review a master syllabus. 7. Academic Freedom In accordance with the College’s Policy on Academic Freedom, the guidelines and procedures outlined in this memorandum will not be used to interfere with the 68 academic freedom of instructors to add to and enhance their courses in content, teaching style, delivery, use of technology, or independent assessment of learning. The faculty is encouraged to be creative and innovative in setting high standards for courses. 8. Cancellation This policy and procedures memorandum cancels AA-1503.1, “Master Syllabus,” dated June 20, 1995. SIGNATURE Alex Johnson Chancellor Attachment: Attachment A – Master Syllabus and Course Syllabi: Guidelines and Formats Policy Reference: Principles of Accreditation, Southern Association of Colleges and Schools, Commission on Colleges Louisiana Community and Technical College System, Policy I.1.034 Participatory Curriculum Development Review Process: Curriculum Committee 10/22/04 Academic Affairs Council 10/28/04 College Council 12/7/04 Distribution: Distributed Electronically Via the College's Intranet and E-mail Systems 69 Attachment A (page 1 of 5) MASTER SYLLABUS AND COURSE SYLLABI: Guidelines and Formats Attached is the format to be used in developing the Master Syllabus and the suggested model for Course Syllabi. Definitions Master Syllabus: This syllabus contains the approved student learning outcomes, content, and assessment standards for a credit course at Delgado Community College, no matter when or where it is taught, who teaches it, and regardless of the delivery system. There is only one Master Syllabus for each course taught at the College. Each course must have a Master Syllabus on file in the Office of Curriculum and Program Development. Course Syllabus: A syllabus developed by an individual instructor based on the Master Syllabus. It is revised each semester, provided to the faculty member's supervisor, and given to all students, preferably on the first day of classes. The Course Syllabus provides each instructor the opportunity to individualize an approved course and to respond to the specific needs of the students and the circumstances of the class. Scope All courses have one Master Syllabus agreed to by the faculty who teach those courses. If possible, all Master Syllabi should be put in a common form using Microsoft Word, and both a hard copy and an electronic copy be submitted to the Office of Curriculum and Program Development. Master Syllabi for all courses must be filed in the Curriculum and Program Development Office. The normal administrative procedures are: Faculty members/lead instructors/department heads on all campuses agree on a Master Syllabus that follows the format and guidelines provided. - For divisions with many adjunct faculty or that have multiple courses building upon one another, instructors may decide to design a more detailed “model” syllabus from the master, which all faculty of a given course could use. The Master Syllabus is submitted to all Deans of divisions where the course is taught. The Master Syllabus is submitted to the Curriculum and Program Development Office. Approved: Curriculum Committee May 1, 2009 Vice Chancellor for Learning & Student Development May 26, 2009 70 Attachment A (page 2 of 5) Master Syllabus Format This is the required Master Syllabus format for all Delgado Community College credit courses. CURRENT (APPROVED) COURSE PREFIX, NUMBER AND NAME Course Lecture-Lab-Credit and/ Contact Hours: - - / _____ Course Maximum Enrollment: _______________________ Lab Fee: __________________ Special Facility or Equipment Needs/Safety Rules and Issues: ________________________ **** Course Description: Describes the type and level of course, including typical method of delivery. The description should be no more than five typed lines in length appropriate for the College Catalog. Each course description should begin with something other than a verb. The first portion is a half-sentence, followed by complete sentences. The sentences of the description should group similar concepts together. Include a brief statement about how this course contributes to the College mission for student learning and/or how it contributes to developing certain General Education competencies in the student. Pre- and/or Co-requisites: Specifies the appropriate pre-requisite and/or co-requisite courses based on skill and experience levels required for the course. Pre-requisite(s):_____________________________________________________________ Co-requisite(s): _____________________________________________________________ Course Goal: The goal should be a general statement of what the course is intended to accomplish, including the particular program outcomes or General Education competencies/characteristics the course is designed to develop. The goal should answer the question, "What does the course do for the student?" Student Learning Outcomes: Learning outcomes state what students will be able to do as a result of learning in the course. Use measurable or observable terms to define 1-5 71 overarching learning outcomes. They should begin with active verbs, such as those used in Bloom’s Taxonomy, and should suggest an assessment of some kind. Each outcome statement should complete the sentence, "As a result of learning in this course, the student will be able to...." Course Content: This is an outline or a narrative description and explanation of the major topics, concepts, or themes of study that students will learn about in the course. Any revisions of the major topics of the course content must be brought before the Curriculum Committee. Texts and Readings: The text(s) and/or reading(s) agreed upon by all concerned faculty should be listed in an appropriate format giving the title, author, edition and year of publication. If applicable, additional recommended readings, websites and/or software materials should be listed. Updates of the agreed-upon text(s) and/or reading(s) need only be revised with Division Dean’s approval. Assessment: This section should explain the ways students will demonstrate achievement of learning outcomes as well as the quality standards expected by the discipline. Types of assessment should be consistent with the language used to define learning outcomes and produce reliable measurements of student learning. For example, the statement for Assessment on Master Syllabi might read, “Student learning in this course will be monitored and measured using a variety of approved methods including but not limited to writing assignments, research projects, collaborative projects, presentations, portfolios, case studies, demonstrations and problemsolving activities.” Although not required for the Master Syllabus, each individual instructor's Course Syllabus MUST specify exact expectations of students, including rubrics that explain what different levels of performance look like and how grades are assigned to them. Statement for Disability Services: The policy statement regarding disability services is included in all master and course syllabi and should cite the College Catalog as a source of more information. Approved: Curriculum Committee May 1, 2009 Vice Chancellor for Learning & Student Development May 26, 2009 72 Attachment A (page 4 of 5) Course Syllabus Format This is a suggested format for Course Syllabi which instructors must present to division deans and give to students at the beginning of every semester for each course. The individual course syllabus with regard to description, content, etc., can elaborate on the master syllabus, but must contain all major components described in the master syllabus for that course. COURSE NAME Instructor: _________________________Course Section(s): _________________ Office (place): _______________________Meeting Place(s): __________________ Office Hours: _________________________ Safety Issues & Rules: __________________________________________________________ Course Name (complete): _______________________________________________________ Course Prefix and Number: ___ Course Lecture-Lab-Credit and/ Contact Hours: /__ Course Description: Includes catalog description but may be more detailed. Pre- and/or Co-requisites:______________________________________________________ Instructors may include a brief statement describing what is expected that students already know or are able to do coming into the course. Course Goal: General statement of the purpose of the course; may indicate how consistent the course is with the College Mission and the needs of Delgado students; if appropriate, indicates program goals met by the course and the types of General Education Competencies/Characteristics developed as a result of learning in the course. Student Learning Outcomes: These are the overarching course outcomes in the format, "As a result of learning in this course, you will be able to . . ." The instructor may also add outcomes in oral communication, writing, critical thinking, and problem solving skills. Course Content and Unit or Topic Outcomes: This section provides students with a weekly or daily schedule of classes. It lists the assignments and learning activities for each Unit or Topic of the course, including specific outcomes expected from learning in each area and exams or assessments 73 of learning in each area. These specific outcomes may be an elaboration of course level SLOs listed on the Master Syllabus, providing greater detail at the Unit or Topic level. The following sample format may be used for this section: Date or Week # Unit/Topic and Learning outcome. “As a result of learning in this section of the course, you will be able to…” Learning Activities and Assignments Learning Assessment, Tests or Exams Texts, Readings and Other Educational Resources: Required texts for purchase, supplementary texts, library readings, websites, audio-visual/computer materials, supplies or special equipment, and educational resources (e.g., availability of tutoring, learning resource centers, and computer labs). Assessment: Explain how student learning will be monitored and measured. Specify expectations of students. Include assessment objectives (student knowledge, skills, attitude, and behavior); specify outcomes criteria and include grading rubrics to clarify expectations and levels of quality. Explain the grading rationale. Teaching/Learning Methods: Instructors may choose to inform students of the basic structure of the course, i.e. lecture, PowerPoint, group discussion, collaborative learning, one-on-one coaching, etc. Statement for Disability Services: The policy statement regarding disability services must be included in the course syllabus and should cite the College Catalog as a source of more information. College and Classroom Policies: Instructors are encouraged to include policies and procedures regarding attendance, discipline, make-up exams, etc., in their individual course syllabi. Approved: Curriculum Committee May 1, 2009 Vice Chancellor for Learning & Student Development May 26, 2009 74 Appendix K DCC GenEd Core Competency/SLO Matrix – Excerpt This document is in development. Down the first column are Delgado’s 9 General Education core competencies and characteristics we hope to develop in our students. The next three columns divide each core competency into stages that correspond to Bloom’s taxonomy of learning. Plotted according to the competency and stage to which Title III Pilot instructors have determined they contribute are course level student learning outcomes for courses to be piloted for Title III. GenEd Core Competency – A set of skills, knowledge, attitudes, and behaviors developed and consistently applied in the workforce and in life. Stage 1 – Foundation (Remembering, Understanding) Student may be passive and compliant, but should be attentive and aware. Much observing and recalling information at this stage. Basic recall includes dates, events, places, major ideas and mastery of subject matter or basic skill. This stage begins the student’s ability to understand information, grasp meaning, translate knowledge into new context, interpret facts, compare, contrast, order, infer causes, and predict consequences. Stage 2 – Inquiry (Applying, Analyzing) The student is now using information, methods, concepts, theories in new situations, solving problems using learned skills or knowledge. Student is bringing together different values and building internally a consistent value system that can be see in his/her behavior. He/she sees patterns, organizes parts, recognizes hidden meanings, and identifies components. Stage 3 – Meaning-Making (Evaluating, Creating) Student uses old ideas to create new ones, generalizes from given facts, relates knowledge from several areas, predicts and draws conclusions. He/she compares and discriminates between ideas, assesses value of theories, makes choices based on reasoned argument, verifies the value of evidence, recognizes subjectivity. Student begins to behave according to a moral “life style” and maintains a consistent philosophy regardless of coercive surroundings. I. Writing and Critical Thinking – ability to read satisfactorily in an organized and critical manner. Demonstrates a general understanding of the English language. Characteristics: 1) The ability to describe, report, order and analyze facts and opinions 2) The ability to distinguish between facts and opinions, to synthesize facts and opinions, and to think critically. 3) The ability to compose and express a series of related ENGL 061 – Developmental English I Apply the fundamentals of grammar to his/her own essay Identify and correct specific elements of grammar ENGL 101 – English Composition I: Edit to correct errors in grammar, punctuation, spelling, and usage ENGL 221/222 – British Lit. I & II: Independently synthesize and analyze FREN 101 – Elementary French I CCSS 107 – College Success Skills: Analyze what’s required for entering the professional workforce ENGL 062 – Developmental Composition II: Generate and organize their own ideas into an essay written in predominantly error free standard English prose. ENGL 101 – English Composition I: Accurately quote, summarize, and paraphrase ENGL 102 – English Composition II: Read with comprehension Analyze ideas, opinions, patterns, and themes Apply critical thinking skills to real world experiences including other disciplines BIOL 212 – Microbiology Laboratory: Demonstrate improved critical thinking skills through intricate problem-solving analysis ENGL 061 – Developmental English I Generate and organize his/her own ideas into a paragraph Generate and organize his/her own ideas into an essay ENGL 062 – Developmental Composition II: Generate and organize their own ideas into an essay written in predominantly error free standard English prose. ENGL 101 – English Composition I: Develop and organize ideas to support Writing - Construct cohesive compositions using the themes covered in 75 GenEd Core Competency – A set of skills, knowledge, attitudes, and behaviors developed and consistently applied in the workforce and in life. Stage 1 – Foundation (Remembering, Understanding) Student may be passive and compliant, but should be attentive and aware. Much observing and recalling information at this stage. Basic recall includes dates, events, places, major ideas and mastery of subject matter or basic skill. This stage begins the student’s ability to understand information, grasp meaning, translate knowledge into new context, interpret facts, compare, contrast, order, infer causes, and predict consequences. Stage 2 – Inquiry (Applying, Analyzing) The student is now using information, methods, concepts, theories in new situations, solving problems using learned skills or knowledge. Student is bringing together different values and building internally a consistent value system that can be see in his/her behavior. He/she sees patterns, organizes parts, recognizes hidden meanings, and identifies components. Stage 3 – Meaning-Making (Evaluating, Creating) Student uses old ideas to create new ones, generalizes from given facts, relates knowledge from several areas, predicts and draws conclusions. He/she compares and discriminates between ideas, assesses value of theories, makes choices based on reasoned argument, verifies the value of evidence, recognizes subjectivity. Student begins to behave according to a moral “life style” and maintains a consistent philosophy regardless of coercive surroundings. thoughts, unified in content and coherent in language. French 101 Reading/Listening Comprehension: - Translate French 101 level passages written in non-technical French ENGL 205 – Intro.to Short Story & Novel: Relate literary works to personal experience Analyze/interpret literary works Analyze relationships among selected elements of literary form and thematic content (e.g. setting and characterization, or symbol and theme) ENGL 206 – Intro.to Poetry and Drama: Analyze poetry and drama by using interpretative tools Write a literary analysis using proper MLA documentation ENGL 212 – American Literature II: Analyze and interpret a text in writing FREN 101 – Elementary French I and explicit thesis statement ENGL 102 – English Composition II: Synthesize ideas on a variety of topics Integrate current, accurate, and valid information from various sources, without incident of plagiarism ENGL 212 – American Literature II: Use critical thinking to plan and implement a project Evaluate primary and/or secondary sources and integrate into an original response ENGL 221/222 – British Lit. I & II: Demonstrate respect for others in written responses PHIL 175 – Social Ethics: Evaluate arguments and identify faulty arguments in reason and logic SPCH 230 – Public Speaking: Assess your audience, occasion, and communication context for an oral presentation Research and Select appropriate subject (Need to accommodate foreign languages and sign language here. Also need to consider extracting “Critical Thinking” and define that as a skill Assessments to include: Write a well-organized paper using appropriate documentation Read a document and demonstrate a comprehensive response Identify the theme and READ 072 – Dev. Reading II: Identify and use context clues to understand the meaning of unfamiliar vocabulary Identify the stated main idea in a paragraph Recognize the supporting details in a paragraph and state how they relate to the main idea (pattern of organization) Speaking/Listening Comprehension: - Answer and ask basic questions about identity and origin - Describe themselves and others - Explain schedules and daily activities - Communicate preferences, likes, dislikes, and plans in the near future 76 GenEd Core Competency – A set of skills, knowledge, attitudes, and behaviors developed and consistently applied in the workforce and in life. setting of a written document) Stage 1 – Foundation (Remembering, Understanding) Student may be passive and compliant, but should be attentive and aware. Much observing and recalling information at this stage. Basic recall includes dates, events, places, major ideas and mastery of subject matter or basic skill. This stage begins the student’s ability to understand information, grasp meaning, translate knowledge into new context, interpret facts, compare, contrast, order, infer causes, and predict consequences. Stage 2 – Inquiry (Applying, Analyzing) The student is now using information, methods, concepts, theories in new situations, solving problems using learned skills or knowledge. Student is bringing together different values and building internally a consistent value system that can be see in his/her behavior. He/she sees patterns, organizes parts, recognizes hidden meanings, and identifies components. Stage 3 – Meaning-Making (Evaluating, Creating) Student uses old ideas to create new ones, generalizes from given facts, relates knowledge from several areas, predicts and draws conclusions. He/she compares and discriminates between ideas, assesses value of theories, makes choices based on reasoned argument, verifies the value of evidence, recognizes subjectivity. Student begins to behave according to a moral “life style” and maintains a consistent philosophy regardless of coercive surroundings. Writing - Construct cohesive compositions using the themes covered in French 101 matter to be organized and presented in an informative speech HIST 101/102 – Early Western Civilization/Modern Western Civilization: I. Writing and Critical Thinking, (cont.) Use academic skills necessary for success in any college-level history class READ 072 – Dev. Reading II: Use decoding skills in order to read independently at 8-12 grade level READ 101 – Analytical Reading Demonstrate the ability to read college-level materials critically and analytically Identify and analyze the organizational parts of both fiction and nonfiction 77 Appendix L Sample Slides from Instructional Design Workshops Integrated Course Design Planning for learning Today’s Objectives • Identify a strategy for addressing resistance • Explain a developmental learning strategy • List examples of active learning • List teaching/learning activities based on brain research • Name the basic components of course design • Analyze a course design for effective integration Integration Example #1 Student Learning Outcomes Teaching and Learning Activities Feedback and Assessment 78 Part 2: Writing Student Learning Outcomes Try it • • • • Look at the big picture Consider the taxonomy Determine results of learning Determine performance criteria Critique it Are these well written student learning outcomes? • Know more about personal communication style • Understand the role of gender and cultural differences in communication • Complete the “Feedback on Verbal Communications” to identify my strengths and weakness • Identify and list my communication strengths and weaknesses, design an action plan, and evaluate the effectiveness of learning three months later • Interpret the logical consistency of a report. 79 Consider the course you teach Part 2 – Determining Assessment Learning Outcomes: - Distinguish between various types of assessment - Choose an assessment that aligns with course outcomes Learning Assessment Process – Course Level Define Student Learning Outcomes Discuss and use assessment results to improve learning We Are Here Develop or select assessment measures Create experiences leading to outcomes Try It Learning Outcome: Distinguish between various types of assessment • Examine a student learning outcome – How would I know if the student was learning how to do this? – How would I determine whether or not a student had achieved this? Learning Outcome: Choose an assessment that aligns with course outcomes 80 Assessment Tool Checklist –Do they provide feedback to students for improvement? –Do students perform the outcome in some way in the assessment? –Are they in alignment with the teaching methods and learning activities? –Do they provide information instructors can use to facilitate learning or improve instruction? Learning Outcome: Choose an assessment that aligns with course outcomes Consider the course you teach Part 3 – Achieving Learning Outcomes Try It Learning Outcome: Choose learning activities that align with course outcomes • Examine a student learning outcome – From what kind of an experience would a student learn how to do this? – What kind of activity would set a student up for success on the final evaluation? Learning Outcome: Choose learning activities that align with course outcomes 81 Activity Checklist – Do they give students opportunities to practice before evaluation? – Do they give students opportunities to learn in different ways? – Do they give students opportunities to DO something with the course content? – Do they give students opportunities to reflect upon their learning? – Is the student or the instructor using the course content? – Are students interacting with each other and with the instructor? Learning Outcome: Choose learning activities that align with course outcomes • • • Peer Review Outcome statements – Do they begin with an active verb? – Are there less than 6 of them? – Do they suggest an assessment? – Will the chosen assessments measure them? – Do they align with our GenEd core competencies? Learning assessment methods – Do they provide feedback to students for improvement? – Do students perform the outcome in some way in the assessment? – Are they in alignment with the teaching methods and learning activities? – Do they provide information instructors can use to facilitate learning or improve instruction? Teaching and learning activities – Do they give students opportunities to practice before evaluation? – Do they give students opportunities to learn in different ways? – Do they give students opportunities to DO something with the course content? – Do they give students opportunities to reflect upon their learning? – Is the student or the instructor using the course content? – Are students interacting with each other and with the instructor? 82 Appendix M Cohort Course Evaluation Form – Sample Cohort # – Date Impact of Title III Program Goals: Please use this scale to respond to the following statements: 1 Yes, due to this program 2 Mostly due to this program 3 Somewhat due to this program 4 Not due to this program 5 1. I have an understanding of what student learning outcomes are. 2. I feel confident in creating an outcome-oriented course. 3. I have an understanding of how active and collaborative methods increase learning. 4. I will use active and collaborative learning techniques in my course. 5. I learned something valuable from the stories/experience of my colleagues. 6. I see the value that good course design can have on student learning. 7. I am able to apply learning theory to instructional design. Feedback for Continual Improvement On the plus side, list those things that were valuable in this program. On the delta side, list those things that you would change to increase the value of this program. + 83 Appendix N Data Results from Likert Scale Survey Items “Designs for Learning” Course and Workshop I have an understanding of Student Learning Outcomes 3 4 Completely due to this program 19 Mostly due to this program Somewhat due to this program 79 Not due to this program 93% agree that the program helped them understand student learning outcomes 6% agree that something other than the program helped with their understanding of student learning outcomes I feel confident in creating an outcomeoriented course 2 Completely due to this program 6 Mostly due to this program 35 62 Somewhat due to this program Not due to this program 92% agree that the program helped them feel confident in creating an outcome-oriented course 7% agree that something other than the program helped them feel confident in creating an outcome-oriented course. 84 I have an understanding of how active and collaborative methods increase learning 7 2 Completely due to this program Mostly due to this program 34 60 Somewhat due to this program Not due to this program 91% agree that the program helped them understand how active/collaborative methods increase learning 8% agree that something other than the program helped them understand how active/collaborative methods increase learning I will use active and collaborative learning techniques in my course 5 Completely due to this program 5 Mostly due to this program 13 38 Somewhat due to this program Not due to this program 84% agree that the program helped them use active and collaborative learning techniques in their course 16% agree that something other than the program helped them use active and collaborative learning techniques in their course 85 I learned something valuable from the stories/experiences of my colleagues 2 2 Completely due to this program 32 Mostly due to this program 67 Somewhat due to this program Not due to this program 96% agree that the program helped them learn something valuable from their colleagues 4% agree that something other than the program helped them learn something valuable from their colleagues I see the value that good course design can have on student learning 3 2 Completely due to this program 14 Mostly due to this program Somewhat due to this program 45 Not due to this program 92% agree that the program helped them see the value that good course design can have on student learning 8% agree that something else helped them see the value that good course design can have on student learning 86 I am able to apply learning theory to instructional design as a result of this program 1 Completely due to this program 2 Mostly due to this program 18 25 Somewhat due to this program Not due to this program 93% agree that the program helped them apply learning theory to instructional design 6% agree that something other than the program helped them apply learning theory to instructional design 87 Appendix O Raw Data Plus/Delta Feedback, Course Evaluation Plus (The Keepers) – 207 data points Sub-Categories Meta Category - Curriculum/Delivery (112 data points) Instructor-39 Structure/Delivery method-38 Resources/Content-23 General value added-12 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. Presents knowledge well Models what she teaches Passion for topic Angela-good teacher Angela-great at keeping the discussions flowing without lecturing. The instructor's patience Gives clean instructions Presenter tried to work with the group Working one on one Working with Angela Angela and Pat Angela's actually working with us, not just directions. Feedback Also, one on one meetings to outline my course and program outcomes Angela is an excellent facilitator Very helpful and open The facilitation ability of the teacher to coordinate the class Facilitation by the team leader to clarify "grey" issues Angela This has helped me so much. Thank you for your help. Angela Breckenridge Angela's patient guidance Also I felt Angela worked hard to give me abstractions/outcomes, "Goals" Excellent Speaker Angela Breckenridge in general Angela as an instructor Excellent teacher Angela's expertise on the subject Her ability to keep discussions on track and positive Well done Angela Availability of Instructor. Encouragement Helpfulness of Instructor Angela Breckenridge Continuous enthusiastic engagement/constructive feedback from Angela Mandatory one on one time with instructor Still need help in writing outcomes for our course Great Instruction Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor Instructor 88 Plus (The Keepers) – 207 data points 1. 2. 3. 4. Sub-Categories Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method 33. 34. 35. 36. 37. 38. Ability to work around class schedule On time Presentation The flow of information and the order in which it was given helped me to the ultimate goal of dissecting my course to discover what the learning outcomes were Understanding of the Nursing program design before the presentation. You always have a lot of detailed well-groomed info in your presenting. Group Activities Relaxed atmosphere Design Presentation was excellent, pacing good PowerPoint presentations Good PowerPoint slides Able to work in course groups Group Activities Writing Peer reviews Paired group activities Smaller consistent groups/members Large group intro & wrap-up Very organized On point Guest Lecturer PowerPoint presentations Dialogue Coincide manner in which information was presented Well organized The readings when combined w/discussion Work in groups Group project & interaction In the first semester Discussion of SLO's Exercises Awesome how material from Nursing course were incorporated into w/s Active Participation Stimulated the thought process Activities Presentation by each participant Exercises Individual Presentations 1. 2. 3. 4. 5. 6. 7. Excellent topic Examples used by Angela Handouts Resources Online Excellent Book The readings The readings Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method 89 Plus (The Keepers) – 207 data points 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. Text book Booklet Textbooks The PowerPoint video that showed the evolution of technology etc. (Think about it?) Reading assignments & discussions Binder material Textbooks Idea paper I really enjoyed the presentations each session Information Gave great examples "2" Scores indicate where felt abstract nature of concepts, not presentation made some ideas difficult to catch-otherwise. Harder Edges Basic research Information provided Handouts 1. 2. 3. Sub-Categories Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Resources/Content Presentations were excellent This is a very good workshop. You have managed to put so many "essential truths" in a comprehensible 2 hours. 4. Pauses to think 5. Workshops 6. Definitely 7. This was very valuable 8. Knowledge 9. Overall really enjoyed the course 10. Learned a lot 11. I would change nothing 12. All of the above were valuable General value added General value added General value added Meta Category – Application to Teaching Practice (55 data points) Core Competencies for Faculty-29 Application in practice-16 Awareness/Perspective Shift-10 1. Core Competencies for Faculty Recognizing important concepts to focus on for Teacher / student education 2. Creating outcomes & objectives 3. Creating outcomes & objectives 4. S.L. outcomes, Objectives, Assessments, How they affect each other 5. Outcomes - SLO 6. State learning outcomes in terms of what you want the students to "own" 7. Especially with new / younger instructor 8. Various learning styles 9. Student Learning Outcomes & Student Learning Objectives 10. A better understanding of active/collaborative learning 11. Finally get what learning outcomes are 12. Configuring of outcomes & assessments General value added General value added General value added General value added General value added General value added General value added General value added General value added Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty 90 Plus (The Keepers) – 207 data points 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. Sub-Categories Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty 28. 29. Better understanding of assessment linked to improve learning Learning outcomes Stimulated participants to think about SLO's "Integrated Course Design" which utilizes Learning Goals Teaching / Learning Activities, and feedback & Assessment A deeper understanding of the terms Learning objectives Integrated definition assessment and outcomes Learned about "Integrated" course design How it relates to older terminology of goals/outcomes & objectives Identify ways to evaluate student learning using methods to provide feedback in an ongoing manner and to help them retain content learned Using the active learning principles giving me tools to enhance student learning Justify what a student should learn in class How students learn material & How assessment improves your classroom experience Active Learning Hold myself accountable for enabling learning 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. Always in context of actual courses Using the actual master & course syllabi as models - double plus. Used our Nursing SLO and course Outcomes so it was real world Allowed us to apply your "content" right away Models of SLO's/assessments already being used at DCC I plan to utilize this in my particular course. Samples from our curriculum One minute paper for feedback Practicality Student learning outcome development in a course syllabus Methods for feedback Methods for assessment Practice exercises Idea's on different learning techniques Teaching strategies Practicality Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice 24. 25. 26. 27. 1. It helped me have the confidence to break traditional Science teacher roles 2. Re-thinking current methods 3. Re-thinking current strategies 4. I really needed to analyze my courses 5. Stress using an actual course I'm teaching 6. The way I impart my information. 7. Started a thought process 8. If we as educators embrace that notion that notion, then education should prove meaningful for the students we teach without ever having to articulate why learning any discipline is important. 9. Took a bit before I felt I understood how to write a better syllabus 10. Freedom to experiment w/ teaching and learning Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Awareness/Perspective Shift Awareness/Perspective Shift Awareness/Perspective Shift Awareness/Perspective Shift Awareness/Perspective Shift Awareness/Perspective Shift Awareness/Perception Shift Awareness/Perspective Shift Awareness/Perspective Shift Awareness/Perspective Shift 91 Plus (The Keepers) – 207 data points Sub-Categories Meta Category – Collaboration (40 data points) Structured dialogue -20 Why it’s value added-20 1. This was a great course for the instructor to collaborate with each other Feedback from other faculty Group Discussions Exchange of Ideas Ability to interact with colleagues from diverse populations and disciplines. Loved group sessions where we got to share ideas from difference disciplines. Experiences from other instructors. Group Discussions Discussion among other colleagues Discussion with faculty of other disciplines Group Discussions Group Activities with other colleagues Group Discussions Opportunities to share positive/negative teaching experiences. Discussion Being with other teachers Discussing our student Made us collaborate with our own group and other groups. - asked thought- provoking questions Sharing of the "Learning Journey" with Title III faculty at each meeting. Meetings with other faculty to share what others are doing in other disciplines Structured dialogue All having the same problem It didn't matter whether it was culinary arts or math To learn what others are doing apply similar things Good collegial collaboration Listening to others ideas Sharing interaction with colleagues on designing learning Discussion of the ways students learn Discussion of shared experiences by faculty from variety disciplines Communication about learning outcomes with colleagues An appreciation that education does not occur in a vacuum and that all disciplines are inter related. Models of SLO's assessments already being used at DCC. Discussion of the techniques used by other instructors. Discussions of our examples Made us collaborate with our own group and other groups. - asked thought- provoking questions Sharing of the "Learning Journey" with Title III faculty at each meeting. Meetings with other faculty to share what others are doing in other disciplines Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added Why it’s value added 92 Plus (The Keepers) – 207 data points 18. Collegial nature of group 19. Opportunity to discuss course to colleagues in a less stressful atmosphere 20. Faculty interaction Delta (Changes) – 90 data points Sub-Categories Why it’s value added Why it’s value added Why it’s value added Sub-Categories Meta Category – Curriculum/Delivery (51 data points) Structure/Delivery method-27 Timing/Venue-21 Resources/Content-3 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. More discussion This could be done online More role-playing of how a particular strategy works Could cluster instructions Less face to face meetings More face to face meetings For me online takes too much time better designed discussions on blackboard Focus on most important concept(s) Instead of 20 page details Assessment of teaching style in the beginning of the program More reflective discussion per session Terminology was hard to grasp/sort at first the difference between Outcome and Objective Subject-focused How to satisfactorily evaluate the assessments Condense Some of this was too basic Make packet bound together Number the pages for easy cooperation with audience Give us a reading assignment (link) ahead of time Practiced what you taught What do I want STDS to remember/incorporate/practice in 2-3 years More intense work session A little disorganized Decrease size of groups Handouts to accommodate note taking Having 2 cohorts cross all disciplines Too short Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Temperature All dates were not connected Hold session at the beginning of semester when we are fresher More time Timeframe workshop for 4 hours lunch then dismissed More time or a multi-part workshop Not so long At the end of the semester, it would be good to do a face to face Increase number of meetings to perhaps 2 per mentor to keep information fresh Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 1. 2. 3. 4. 5. 6. 7. 8. 9. Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method Structure/Delivery method 93 Delta (Changes) – 90 data points Sub-Categories 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. Shorter Length of program Offer two part Program on 2 consecutive Fridays A break Temp audience seating arrangement. Configuration of rooms Group work lends itself better to circular configuration Lighting in the room Room temperature Room more conducive to learning Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue Timing/Venue 1. 2. Perhaps a better text Discuss author's own acknowledgements of limitations of the "new paradigm" The textbook makes these concepts sound more complicated than they really are Resources/Content Resources/Content 3. Resources/Content Meta Category – Application to Teaching Practice (28 data points) Application in Practice-15 Obstacles to application-7 Core Competencies for Faculty-6 1. 2. More time to apply to each of our course Tweak or expound on what they're already doing. In other words, we are highlighting that which they're already doing while reinforcing the need of the project to enhance their further development Tell us to bring our course objectives with us to the program Make seminar specific to nursing program Do some readings that fit the profile of DCC students More emphasis on the pilot Need more examples of how this really works in classroom I would like to spend more time on the actual try-its. Make it a true try-it. More application to Nursing Courses individually Maybe have us physically work on the syllabus a little more First session, bring syllabus Second session, add your learning outcomes. So that we can discuss within the group A few more examples of appropriate assessments Examples (models) from this faculty of their SLO's/assessments syllabi As with all initiatives, recognize that ANY data collected is valuable…what DOES NOT work is as invaluable as what DOES! Application in practice Application in practice In general an odd semester Meetings disturbed by Gustav (hurricane) Snow day in December. The conflict I encountered when I presented my Title III Syllabi to Curriculum Committee So I guess what I'm saying is that it is still difficult to reach consensus in how to apply the information to wide student learning objective & Obstacles to application Obstacles to application Obstacles to application Obstacles to application 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 1. 2. 3. 4. 5. Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Application in practice Obstacles to application 94 Delta (Changes) – 90 data points 6. 7. 1. 2. 3. 4. 5. 6. outcomes Give released times for pilots Using a delta symbol on an evaluation form I would like to take another course like this. Part II? Others would need training before possibly Should continue for years until all have participated Consider special cohort to address issues peculiar to online learning Should include successful completion for promotion Continue it in the future to allow interested faculty to repeat process for other classes Sub-Categories Obstacles to application Obstacles to application Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Core Competencies for Faculty Meta Category – Collaboration (11 data points) Structured dialogue-9 Why it’s value added-2 1. 2. I think meeting as a group during the spring Visits from faculty members who have already gone through this process Stories of successes/challenges in the various departments while developing their syllabus, etc. Meetings at those conducting pilots to discuss problems/successes Needed more time as a group More designated time working with other instructors on this process Maybe a blackboard site with database to share with all faculty in program to share With all faculty in program to draw everyone in During the semester after the course when we pilot our courses, I would appreciate some way to share with other faculty in the pilot about experiences Structured dialogue Structured dialogue Using previous pilot experiences more to learn from them Sharing experiences would be helpful Why it’s value added Why it’s value added 3. 4. 5. 6. 7. 8. 9. 1. 2. Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue Structured dialogue 95 Appendix P Summary Results – Plus/Delta “Plus” Feedback Program components to keep for future success Curriculum/Delivery - Keeper Categories 12 39 23 Instructor Structure/Delivery Method Resources/Content General Value Added 38 Structure/Delivery Method 38 comments regarding the value of the program’s structure/delivery (34% of the data points in this category) Instructor 36 comments regarding the value of a capable, full-time instructor, someone who knows how to create the kind of experience they say is worth keeping (32% of the data points in this category) Resources/Content 22 comments regarding the value of resources/content (20% of the data points in this category) General Value Added 11 general comments about the value of keeping the program (10% of the data points in this category) 96 Application to Teaching Practice How and Why 10 Core Competencies for Faculty 29 16 Application in practice Awareness/Perspective Shift Core Competencies for Faculty 29 comments regarding the applicability of the program to developing key knowledge, skills, and attitudes for successful teaching (53% of the data points in this category) Application in practice 16 comments affirming the applicability of program outcomes to teaching practice and providing examples of the program’s influence on instructional design (29% of the data points in this category) Awareness/Perspective Shift (The importance of reflective practice on continual improvement) 10 comments reflecting on the program’s influence in creating awareness and new perspective on instructional design (18% of the data points in this category) 97 Structured Collaboration - Importance of Sustaining 20 20 Structured Dialogue Why it's value added Structured Dialogue 20 comments regarding the importance of having structured opportunities for dialogue among faculty within and across disciplines (50% of the data points in this category) Why it’s value added 20 comments regarding what is learned from and why collaboration is important to facilitate and to structure formally for some purposes (50% of the data points in this category) 98 “∆” Feedback Program components to modify for sustainable value Curriculum/Delivery - Suggestions for Improvement 3 Structure/Delivery Method 21 27 Timing/Venue Resources/Content Structure/Delivery Method 27 comments indicating what and how to improve the program’s curriculum and delivery for the future, including suggestions that build upon or complement the existing structure (53% of the data points in this category) Timing/Venue 21 suggestions to help schedule and organize the program for the future (41% of the data points in this category) Resources/Content 3 opinions about the authors and textbook, statistically unremarkable in comparison to the number of comments that valued the resources and content of the program (6% of the data points in this category) 99 Application to Teaching Practice - How to, why it's hard, where the College needs it 6 Application in Practice 15 7 Obstacles to application Core Competencies for Faculty Application in Practice 15 suggestions that build upon (add more of the same to) or complement existing components of the curriculum believed to be practical for teaching (54% of the data points in this category) Obstacles to Application 7 comments indicating why applying some of the concepts in the program is difficult (33% of the data points in this category) Core Competencies for Faculty 6 comments that recommend the program be institutionalized into existing developmental and HR systems (29% of the data points in this category) 100 Collaboration - More of the Same 2 Structured Dialogue Why it's value added 9 Structured Dialogue 9 suggestions for structuring more faculty collaboration (81% of the data points in this category) Why it’s value added 2 comments indicating the value of collaboration, statistically unremarkable in comparison to the number of similar comments in the “Plus” column. However, these two data points support the “Plus” column and could be included there in the findings concerning the value of collaboration (18% of the data points in this category) 101 Appendix Q Data Results From Section III, Title III Final Pilot Report Self-report: Learning about instructional design or teaching/learning – 198 Instructor recommendations based on learning instructional design – 331 Meta Category Terms/Language FQ 72 Instructional effectiveness 61 Assessment 65 More Collaboration 105 More development like Title III 61 Sub-Category Outcomes v. Objectives FQ 18 Refining Awareness Syllabus Covering Material Learning styles Awareness Already doing it Impact on student Framework for learning Awareness TracDat Methods Already doing it Competency assessment Across disciplines 9 20 25 19 12 6 5 10 9 29 17 8 2 9 13 Share what we’ve learned Use Convocation For Program goals Consensus on course level SLOs Consensus on quality standards No consultants, learn from within For course syllabi W/ students on SLOs Online Support/Resource for SLO/Assessment For Understanding Assessment For managing group learning W/ faculty led workshops and sharing 19 28 11 10 11 14 9 3 9 17 11 6 18 102 Meta Category Changes in teaching and learning in the classroom – 210 Revise GenEd FQ 70 Syllabus revision throughout curriculum 80 Walk the walk of learning centeredness 3 Setting expectations for students 12 Setting expectations for students 38 Revise course design 31 Integrating new techniques 51 Student involvement Assessment 8 57 More use of library resource 9 Sub-Category Emphasis on writing and critical thinking Clarify Fine Arts v. Humanities Critical analysis of most important content Common rubrics Divide – Skills & Content Areas Define more clearly Include variety of assessments in MS Include variety of activities in MS Consistency Online/FTF versions same Critical analysis of curriculum Seminar classrooms Scheduling classes Building labels (Names v. numbers) Type of teaching method, explaining why Explaining SLOs to students Academic SLOs for Orientation Rubrics as learning tool FQ 28 5 6 9 4 18 9 5 27 26 13 1 1 1 1 5 6 16 Modify presentation of SLOs Produce competency checklist Reiterate presentation of SLOs Forward looking assessments Clarify SLOS, more measurable More student-focused Using SLOs for everything in the class Peer mentoring Learning in chunks Summarizing as a learning tool Generate test questions Pre-post-testing of student competency Incorporating critical thinking, problemsolving, and group efforts Keep trying with rubrics Portfolio Students learning databases for research 8 5 9 8 23 29 9 5 4 4 8 6 30 6 15 4 103 Meta Category Observations of student responses to changes in teaching/learning – 369 Time management for incorporating LC principles Positive response to teaching change Negative response to teaching change Things that promote student learning FQ 16 Sub-Category Field trips to library Library SLOs Worth taking the time, but takes a lot of time FQ 3 2 16 141 Use of “real world” problems 10 collaborative learning/social networking 13 daily in-class writing activity students have more active role in learning process work on exams or assignments as a group 9 19 Active learning and learning-centered activities Discussing the syllabus and the importance of SLOs Active learning and learning-centered activities Student responsibility for learning Discussion board Working in groups (lack of motivation) Need chance to make mistakes before getting a grade Just getting a test back instead of learning from mistakes Daily in-class writing activity Open discussion method Active learning and learning-centered activities Rubrics as teaching tool Work on exams or assignments as a group Active involvement in learning process Case studies Pre-tests 11 8 146 20 11 18 9 21 8 12 5 11 15 12 12 9 7 16 14 104 Meta Category Assessments that reveal learning 20 Theory, scholarship, or methods that have impacted teaching changes 13 Sub-Category Presentations A value put upon the learning and the activity “Ill-defined” group problem Low levels of required skills for succeeding in the course Resistance to change No explanation for use of rubric Me explaining everything instead of the students explaining to each other Low expectations of students Students have no hope Moving writing out into the world changes the way that they perceive writing Group work is not effective for a skills-based class Reflective writing 1-minute papers Short assignments on reading Brain research 53 Evidence of active learning/experiential learning working Blackboard Extra course loads 3 8 5 4 18 Volume of material Shortened calendar (registration, weather, absences) Release time for development Additional institutional demands Teachers Outcomes v. objectives Writing Fear of “wrong” answer Eager to use what was learned Things that inhibit student learning Obstacles to implementing effective teaching methods – 73 Attitudes and thoughts about the program – 124 FQ 26 Beliefs about students 10 Beliefs about teaching methods 5 Lack of time Resistance to change New language Lack of required skills in students Lack of confidence in students Excitement in teaching FQ 18 14 13 9 5 4 8 6 4 2 3 9 5 6 3 4 6 18 9 6 14 6 3 8 5 4 6 105 Meta Category FQ Awareness/perspective shift 21 Improvement in instruction 31 Critical reflection on teaching methods 10 Observations of others Beliefs about teaching 5 16 Lack of excitement 2 Beliefs about organization 14 Sub-Category Sharing experiences with colleagues Due to the pilot Already using some Due to collaboration with other disciplines Lean toward methods that were used on us Takes a while to get Interactions with students and colleagues Inspired to do more Due to the quality of program Need time to focus on classroom learning to improve Assessments, peer review Methods and process, creative and effective Some resistant Helping students overcome obstacles to education Teaching, not methods, is where teachers learn to teach Students need to know they matter More than content, need to teach them what’s being asked of them in the workforce I want them to love reading I want them to be more confident I want them to be able to make decisions about their well being TrackDat boxes and jargon Conforming to whatever May need to change our rules about general education Skepticism that what’s being asked to do will not matter No one believes this will change the master syllabus Some faculty will never change New initiatives mean new buzz words, that’s all FQ 8 4 5 8 3 5 7 8 5 11 4 6 5 2 3 2 3 1 3 2 1 1 1 3 1 3 2 106 Meta Category Recommendations for specific activities and assessments – 9 FQ Sub-Category Department meetings talk about administration, not learning Someone cares about what I’m doing Talking with colleagues FQ 4 Appreciation 7 2 5 Student-designed assignments 2 Visual diagrams Oral reports Group assignments followed by individual assignments on same thing Kinesthetic – building models Take-home practice exams Monthly faculty collaboration meetings Build time in course for critical reflection and discussion 1 1 1 Interconnections of things Demonstration of learning to others 1 1 1 1 1 1 For explaining concepts 1 107 Appendix R Functional Areas Impacted by Title III with Related Services and Activities Faculty/Staff Development Online Learning Learning Communities Classroom Learning Service Learning Academic Affairs Student Affairs Chancellor's Office (Larissa & DDD) DLIT IT Allied Health & Nursing QEP LCT Linkage Documentation Guidelines Best Practice Research Consult Training Curric Design Course Design Method Compet. Model Design Program Devel. X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Title III Obj Outcome Assesmt Consult X X X Business Process Mapping X X X X X X X X X X X X X X Facilitation X X X X 108 Social Entrepreneurs hip Massage Therapy ADOT ESL Early Childhood Math Dept Individual Instructors Convocation Planning Professional Development Academy Online Professional Development Program Program Review Leadership Connection Title III X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X 109 Appendix S Distributing Student Learning Outcomes Coordinators Considerations and Recommendations (ASLOC, pg. 20-22) The research undertaken by the Ad Hoc Accreditation and Student Learning Outcomes Committee reveals an emerging leadership group at California community colleges, Student Learning Outcomes and Assessment Coordinators. These faculty are attempting to undertake a huge task: designing and implementing a sustainable, learning-centered, institution-wide assessment process that addresses the expectations in the new ACCJC-WASC Accreditation Standards. The enormous nature of the task is complicated by the variety of assignments, working conditions, training and preparation of those serving in this position. These variations are not surprising because they speak to the unique cultures, governance policies, and organizational structures of California community colleges. The Accreditation Standards state that no one method or organizational strategy be used; colleges must create ones that suit their institutions and mission. In spite of the local differences, a careful examination of the data gathered through surveys and in the regional meetings, reveals a commonalities and areas of concern that local senates should use when considering the role of the SLO Coordinator and assessment oriented towards processes, not products. The recommendations, in bold, are followed by considerations that include questions and statements intended to help guide you in a manner appropriate for your institution. 1. In order for the SLO Coordinator position to be effective, its placement within the college organizational structure must be adequately defined and carefully considered. One way to begin this analysis is to define a college’s assessment tasks and the best way to organize them. Institutions beginning SLO and assessment work should scrutinize the strategies used by other colleges across the state that may be further along. The SLO listserv created by the Academic Senate, the Academic Senate website, and the RP Group website, Center for Student Success (http://css.rpgroup.org/) are sources to gather information, data and examples. Begin by considering existing committees on your campus. Some colleges connect Program Review committees with program outcomes; others connect Curriculum Committee work with course outcomes, and still others create new committees that interface with existing college governance committees. As always, individual college culture is the key. What will work at your college? 2. The college must determine how it will assign responsibility for the major areas of assessment: student services, library, and instruction (courses, programs, General Education and degrees). Will it work best for your college to have only one person as SLO Coordinator, or two, or a team? Should there be a committee with equal responsibilities? Who will guide and direct student services areas? Will it be the same person coordinating instructional outcomes? Will your campus include administrative services in creating and assessing outcomes? Is one person able to bridge all these areas? What will be most effective considering your college dynamics? 110 3. A clear job description with expectations for the SLO Coordinator position is essential. The local senate should be the principal author for the SLO Coordinator description. What level of authority will the SLO Coordinator will have? Will he or she act as mentor or manager, as SLO czar or outcomes facilitator? Does participatory governance flourish if the coordinator is more a manager than a mentor or coordinator? Will he or she be the keeper of the data or trainer of faculty chairs who instead will keep the data? Determine the importance of qualifications and knowledge for the job, as well as scope of institutional knowledge and experience. Carefully define the expectations for this position and then prudently determine how much time is required to meet the task. Work cooperatively with administration to establish this important faculty role. 4. A clear selection process for the SLO Coordinator with a specified length of service will assist in making the position viable. In most cases, this position should be selected through a process which involves the local senate in order to establish early faculty buy-in. Since reassigned time and research resources are essential, focus group members suggested that cooperation between the local senate and the administration is very helpful. Who will review the applications? Will this process include an interview? How will the administration be involved in the selection process? 5. Clear lines of reporting and accountability make the position more successful. There are many questions that must be clearly answered if the SLO Coordinator position is to work effectively. To whom does the SLO and Assessment Coordinator report? Should there be standing reports to the local senate, the college Vice President, the Board of Trustees? When? How often? If difficult suggestions or decisions have to be made (such as to meet accreditation standards or recommendations) is it better coming from an individual or a committee or the senate? 6. The SLO Coordinator should be fairly compensated in some way for this work. Outcomes and assessment tasks cannot be accomplished by a faculty member in addition to a full teaching load. Some colleges have advanced in limited areas without the reassigned position; however, to cover the breadth of the accreditation requirements for assessment, some type of reassigned time, equivalent to the job assignment, is essential. Local senates should help determine these parameters in conjunction with the administration in a method similar to the reassigned time for other faculty positions related to local senate and legally mandated faculty tasks such as curriculum and program review. 7. The process will not be successful without other significant dedicated resources. The implementation of student learning outcomes is not a trivial task. It cannot proceed without the allocation of significant resources. It is necessary to determine accessibility and use of research data and research staff and to clarify the administrative and clerical support that will be available (e.g. to document evidence). 111 How are institutional research data made available? SLO Coordinators need to be able to work with faculty in departments on an individual level to write and develop SLOs and assessment. What resources are available for faculty training and staff development? Is there a budget for outside speakers? Will stipends be given to faculty doing pilot projects or special work? Is there funding to attend conferences? SLO Coordinators must also work with and educate administrators about outcomes and assessment; most administrators have very little background in the process other than accountability reporting. Working cooperatively with administrators is essential to assure resources and authority. It is imperative that the SLO Coordinator be a liaison to the local academic senate as a whole. However, other considerations should also be examined, such as the SLO Coordinator participating in or reporting to other operational and governance committees such as the curriculum, program review, the institutional effectiveness, planning, or the budget committee. The relationship and responsibility for work with the accreditation team and Institutional Researcher should be understood by all. 8. SLO Coordinators need ongoing training in various aspects of their assignment. The literature review provided in this paper describes the importance of ongoing training for SLO and assessment leaders. The Ad Hoc Accreditation and Student Learning Outcomes Committee, in cooperation with the Research and Planning Group, has held trainings for SLO Coordinators and will continue to provide more. A plan is being considered to develop a statewide training process for SLO Coordinators, perhaps with certification, to identify and provide instruction in the core skills and issues necessary to accomplish and sustain this task locally. After a literature review we believe this may be the first statewide attempt to train faculty-leaders that are coordinating student learning outcomes and assessment efforts. 9. SLO Coordinators and the assessment processes should be regularly evaluated. Consistent with the intent of regular assessment and evaluation that leads to improved teaching and learning, institutions should develop a regular evaluation process for the SLO Coordinator position coupled with an evaluation of the effectiveness of the institutional processes. 112 Appendix T Learning Outcomes Coordinator Position Sample Job Description, Accountabilities, & Comp. Job Announcement: Modesto Junior College Student Learning Outcomes Facilitator The MJC Academic Senate seeks applications for the Student Learning Outcomes Facilitator position. To apply, submit a letter of application (1-2 pages) describing your interest and how your experience fulfills the list of desirable characteristics below. Also, please provide a current resume. Submit all materials to Geri Wend in the MJC Academic Senate Office prior to 3:00 p.m. on Thursday, May 5, 2005. Applications will be reviewed by the Senate SLO Committee and finalists will be invited to an interview. The position is open to all current Modesto Junior College faculty and staff members who meet the minimum qualifications necessary to teach at a California Community College. Initial funding will allow compensation at the equivalent of 40% reassigned time. SLO Facilitator Job Description The Student Learning Outcomes Facilitator will provide leadership and support to faculty and staff as the college develops the infrastructure, policies and procedures to implement an assessment plan. The initial assignment will be for two years with an option for a third year. The facilitator, position description and committee activities will be evaluated by the Academic Senate after the first year. Compensation will be reviewed each year and may be increased as the college develops its approach to assessment. The position shall sunset after (6) full semesters pending renewal by the Academic Senate. Facilitator Tasks Chair the SLO Committee, schedule the SLO Committee meetings and coordinate committee activities with the Academic Senate, Office of Instruction, Director of College Research and Planning and the Curriculum Committee. Recommend policies and procedures to the Curriculum Committee for the development, assessment and improvement of course, program and general education level objectives and measurable outcomes AND in cooperation with the Academic Senate President-Elect, advocate those policies and procedures to the Academic Senate. Collaborate with the Office of Instruction, the Instructional Administrators Council, and the Curriculum Committee to integrate the development, assessment and improvement of measurable program level SLO’s with the current program review process. Present information at meetings including Academic Senate, Curriculum Committee, Instructional Administrators Council and other appropriate venues. Develop, schedule and promote training opportunities for assessment and measurable SLO development including on campus workshops and the Summer SLO Institute. Provide training for staff on development of measurable SLO’s and assessment instruments. Attend local, state and national conferences related to assessment and measurable SLO development. Recommend activities and resource requests to the Academic Senate President for inclusion in the college Educational Master Plan. Assist in the preparation of SLO reports for future accreditation processes. Desirable Characteristics Demonstrated enthusiasm and commitment to the improvement of teaching and learning at Modesto Junior College. Knowledge of the curriculum development process and guidelines at Modesto Junior College. Knowledge of the college program review plan and associated procedures. Possess skills in persuasive communication and facilitating dialogue. Evidence of success in writing for college programs. Knowledge of qualitative and quantitative educational research methodologies. Demonstrated success in the application of Classroom Assessment Techniques. Willingness to travel to state and national conferences. 113 Results of California Community College Study: Release Time and Compensation for SLO Coordinator Table 6: How Much Reassigned Time Does Your SLO Position Provide? N=80 Compensated by reassignment as a part of load 20-30% = 17 40-60% = 16 80% = 1 100% = 3 Compensated by reassignment by hours per week 3-5 = 3 6-10 = 8 11-20 = 1 Compensated by stipends $6,000 = 1 10,000 = 1 *some have combined a stipend with reassigned time Compensation subsumed into other reassigned tasks Accreditation = 1 Research/Institutional Effectiveness = 2 Curriculum = 3 No reassignment or compensation Assigned but no compensation = 17 No coordinator position = 5 No response = 1 Table 7: Colleges Reporting Multiple Reassigned SLO and Assessment Coordinators College Allan Hancock College Cerritos College Chaffey College Diablo Valley College East LA College El Camino College LA City College Mount San Antonio College San Diego City College Southwestern College (ASLOC, pg. 17, 18) Number of Reassigned Roles Two Student Services Two Instruction (Total of four coordinators reassigned) Two Coordinators One Vocational Coordinator One Instructional non-vocational Two Coordinators One Coordinator Three SLO facilitators Two Coordinators One Coordinator Six faculty SLO team (Total of seven reassigned) One Coordinator Two Facilitators Two Coordinators One Student Services Coordinator One Instructional Coordinator Reassign Time 40-60% 40% each 40% each 25% each 60% 40% each Total 180% 33% each 100% 20% for each for six faculty 100% for implementation phase 60% each 8 hours each 20% each 114 Appendix U Director of Learning Outcomes Assessment Sample Job Description & Accountabilities Job Title: DIRECTOR OF LEARNING OUTCOMES ASSESSMENT & INSTRUCTIONAL DEVELOPMENT - Carroll Community College Westminster, MD Deadline Date: Open until filled. For best consideration, please apply by June 5, 2005 Starting Date: Institution: Located approximately 30 miles northwest of Baltimore, Carroll Community College is Maryland’s newest community college with an enrollment of about 3,200 students. For more information, please visit our Web site. Job Duties: The position of DIRECTOR OF LEARNING OUTCOMES ASSESSMENT AND INSTRUCTIONAL DEVELOPMENT is available in the Academic Affairs Division. This position is on a full-time, 12-month basis, working under the direction of the Vice President of Academic and Student Affairs. FUNCTION: Responsible for managing processes associated with instructional development and learning assessment/enhancement in collaboration with Deans, faculty, program directors, department chairs, student affairs directors, and personnel in the Office of Planning, Marketing and Assessment. Specific responsibilities include: Meeting individually with and supporting faculty in creating effective curriculum design and learning outcomes assessment practices and overseeing processes for conducting assessment at the course, program and General Education Core Competency levels; working with Academic and Student Affairs Deans in developing and overseeing a comprehensive plan for outcomes assessment/enhancement processes for all academic and Student Affairs units; consulting with Continuing Education and Training Directors on the development of outcomes assessment processes for non-credit programs and courses; supporting faculty and student affairs staff in the design and execution of surveys and other evaluation instruments in support of ongoing assessment and effectiveness evaluation projects; supporting faculty in the design of learning enhancement projects associated with faculty promotion criteria; developing and presenting instructional improvement/outcomes assessment workshops; working with the office of Planning, Marketing and Assessment to design data gathering and analyses reports related to outcomes assessment projects and participating in college wide strategic planning and assessment initiatives; overseeing the activities and processes of the Learning Outcomes Assessment Committee; promoting and maintaining external network connections with instructional design/outcomes assessment at peer institutions; preparing reports to communicate learning outcomes activities and finds to Maryland Higher Education Commission (MHEC); and assisting with periodic review and self-study processes for regional accreditation via Middle States Commission on Higher Education. 115 Qualifications: Masters Degree in higher education, educational research, social sciences, curriculum design or other discipline requiring coursework in curriculum design and assessment, research and quantitative/qualitative analysis or statistics. A Doctorate in one of these areas is strongly preferred. Director will have a minimum of three years experience in learning outcomes assessment and instructional enhancement processes; will have had experience developing applied research projects in higher education settings; demonstrated competency in the application of curriculum design and learning outcomes assessment theories and methods; proficiency in survey development and the use of statistical analysis tools such as SPSS, SNAP or SAS; demonstrated experience with database systems and data retrieval using query-based software (Datatel’s Colleague or SQL preferred); proficiency in use of office applications software; ability to handle multiple short and long-term projects; ability to maintain strict confidentiality; ability to maintain effective working relationships with diverse faculty and staff; demonstrated leadership and interpersonal skills; ability to communicate effectively, both verbally and in writing including proficiency in preparing written, oral and graphical reports and presentations about complex projects, data analyses, and use of results for internal and external audiences. Two years teaching experience and personal experience conducting learning outcomes assessment practices within courses and/or programs at the college level preferred.