UNCW
Quality Enhancement Plan
eTEAL: experiencing Transformative Education through Applied Learning
NOVEMBER 2012 version PLUS AUGUST 2013 Response
1
Table of Contents
Section
I. Executive Summary
II. University of North Carolina Wilmington
a. UNCW Mission
b. UNCW Learning Goals
III. Process Used to Develop the QEP
a. Values in Action
i. Collaboration
ii. Continuous Stakeholder Engagement
b. Pilot Program
i. Pilot Program Assessment
ii. Pilot Program Outcomes
c. eTEAL Preliminary Implementation (2012/13)
IV. Topic Selection
a. Evidence Based Topic Selection
i. Results for Critical Thinking
ii. Results for Inquiry
iii. Results for Thoughtful Expression (Written Communication)
iv. Results for Information Literacy
v. Results for Diversity
vi. Results for Global Citizenship
vii. Results for Foundational Knowledge
viii. Results for Second Language
b. Narrowing the Topic: Focusing on Applied Learning
c. Forums and Focus Groups and Lead to Final Topic Selection
V. Student Learning First and Foremost
VI. Literature Review
a. Experiential Education and Experiential Learning Theory
b. Critical Reflection
c. Bloom’s Taxonomy
VII. Timeline and Actions to be Implemented
VIII. Organizational Structure
IX. Resources
a. Detailed Budget to Support eTEAL
b. UNCW In-Kind Support for Applied Learning
X. Assessment
a. Assessment of Student Learning Outcomes
b. Assessment of Program Outcomes
c. Assessment Timeline
d. Use of Assessment Findings
e. Assessment Matrix
XI. References
XII. Appendices
a. QEP Topic Development Task Force Members
b. QEP Topic Development Events
c. QEP Topic Development Campus Outreach/Inclusion Events
Page No.
4
6
6
6
7
7
8
9
10
12
16
19
23
23
24
25
25
26
27
27
28
28
29
33
35
38
39
41
44
46
48
51
52
55
56
57
58
61
61
62
66
68
68
70
71
2
d. eTEAL: Expanding the Professoriate Pilot Activities
e. Assessment Rubrics
f. 2011-2012 eTEAL Pilot Budget
g. eTEAL Preliminary Implementation Budget (2012-2013)
h. eTEAL Preliminary Implementation Events (ALTC)
i. 2012-2013 Pilot Implementation RFP for eTEAL – Supported Initiatives
Response
73
74
80
81
83
84
86
3
Executive Summary
The primary purpose of the University of North Carolina Wilmington’s (UNCW) proposed
Quality Enhancement Plan (QEP) is to impact positively student learning through improved applied
learning experiences in three areas: critical thinking, thoughtful expression, and inquiry. A secondary
purpose is to enrich the environment supporting student applied learning. The proposed QEP—eTEAL:
experiencing Transformative Education through Applied Learning—promises not only to meet, but
also to exceed goals in these three areas of student learning. The name eTEAL was inspired by UNCW’s
official school colors—gold, blue, and teal—with teal being the color most associated with the University
and its marketing and branding campaigns.
Within UNCW, applied learning is understood to be a pedagogical model that places students in
experiences requiring them to integrate theories, ideas, and skills they have learned in new contexts,
thereby extending their learning. UNCW’s long standing commitment to applied learning distinguishes it
among its University of North Carolina (UNC) sister institutions. Today’s UNCW students enjoy a rich
environment where thousands of applied learning opportunities are made available during the academic
year. For instance, a QEP Task Force survey identified more than 7,700 applied learning experiences at
UNCW. All UNCW colleges, schools, and academic majors require “applied learning” for enrolled
students , and the general education curriculum (University Studies) mandates in a component area
entitled “Explorations Beyond the Classroom” at least one experience. To date, however, there has not
been a systematic University-wide effort to either capture evidence of student learning through applied
experiences, or to provide appropriate professional development support for faculty and staff instructors
interested in offering applied learning experiences.
In recognition of the widespread opportunities for applied learning at UNCW, the QEP
intentionally focuses on improving the quality of applied learning experiences. eTEAL addresses three
goals:
1.
To improve student learning in applied learning experience through enriching the
environment supporting student learning.
2. To provide faculty and staff with information and resources about high-impact
practices in applied learning pedagogy as appropriate to their discipline and
responsibilities.
3. To promote the incorporation and implementation of high-impact practices of applied
learning throughout UNCW.
4
In pursuit of these goals, eTEAL proposes a three-part model to both measure the impact of
applied learning on student learning in the three aforementioned areas of critical thinking, thoughtful
expression, and inquiry; and to improve the quality of applied learning writ large. Components of this
model are:
1. Applied Learning Summer Institute, open to all instructors.
2. Applied Learning and Teaching Community, open to all instructors and students.
3. eTEAL-supported Pedagogy Initiatives, awarded based on proposals meeting specific
criteria for implementing applied learning techniques and mechanisms for deepening
and documenting student learning.
5
The University of North Carolina Wilmington (UNCW)
UNCW Mission Statement
“The University of North Carolina Wilmington, the state’s coastal university, is dedicated to
learning through the integration of teaching and mentoring with research and service. A powerful
academic experience stimulates creative inquiry, critical thinking, thoughtful expression and
responsible citizenship in an array of high-quality programs at the baccalaureate and master’s
levels, and in doctoral programs in marine biology and educational leadership. Substantial
research activity, combined with our hallmark teaching excellence and moderate size, advances
distinctive student involvement in faculty scholarship. UNCW is committed to diversity and
inclusion, affordable access, global perspectives, and enriching the quality of life through
scholarly community engagement in such areas as health, education, the economy, the
environment, marine and coastal issues, and the arts.”
UNCW Learning Goals
In March 2009, the Faculty Senate through motion 09-07-24 endorsed the recommendations of the
task force report Revising General Education at UNCW. Later, through resolution 11-03-10, the Senate
modified the adopted learning goals for the University Studies program to those currently listed in the
Undergraduate Catalogue:
Goal 1. Students will acquire foundational knowledge, theories, and perspectives in a variety of
disciplines (Foundational Knowledge);
Goal 2. Students will engage in rigorous, open-minded, and imaginative inquiry (Inquiry);
Goal 3. Students will locate, evaluate, and effectively use information by applying a variety of
academic and technological skills (Information Literacy);
Goal 4. Students will integrate multiple methods and perspectives to critically examine complex
problems (Critical Thinking);
Goal 5. Students will effectively express meaningful ideas in speech and writing (Thoughtful
Expression);
Goal 6. Students will demonstrate basic proficiency in speaking, listening, writing, and reading in
a language in addition to English (Foreign Language);
Goal 7. Students will describe and examine the importance and implications of human diversity
(Diversity); and
Goal 8. Students will describe and examine the intellectual and ethical responsibilities of active
global citizenship (Global Citizenship).
6
These learning goals went into effect immediately after adoption, although the University Studies
curriculum phased in new core requirements beginning in fall 2011, to form a common thread linking all
courses approved to satisfy the general education requirement.
eTEAL will directly relate to these university-wide learning goals by positively impacting student
learning in three areas - - critical thinking, thoughtful expression and inquiry- -through improved applied
learning experiences and critical reflection.
Process Used to Develop the QEP
Values in Action
QEP development was powered, from the beginning, by strong values - a commitment to
representative leadership, collaboration, and a continuous process of stakeholder engagement. QEP
development also included the launch and assessment of a pilot program in 2011-12.This section includes
discussion of that component, with special emphasis on the impact of pilot program outcomes in honing
the QEP focus. Finally, QEP development included a preliminary implementation in 2012-13, which
provided myriad opportunities to refine the QEP, also described in this section. Topic selection is
7
inherently part and parcel of any discussion of the process used to develop a QEP, and this subject is
discussed in different detail in a subsequent section of the report.
To begin planning for the development of UNCW’s QEP, former Provost Cathy Barlow
appointed a QEP Topic Development Coordinator, and then charged a QEP Topic Development Task
Force to lead the campus through the QEP process beginning in 2010-11. The Task Force is
representative of all major units on campus, serves as the QEP’s primary planning body, and is composed
of faculty, staff, and student members (a list of members by year is available in Appendix A). Their work
focused on four goals: raising awareness across campus of the QEP and its importance, analyzing
evidence from assessment of student learning, collecting feedback from all constituents on their areas of
need and potential focus, and developing an acceptable QEP for campus consideration and later review.
Collaboration
Diverse strategies to accomplish these goals targeted and engaged different stakeholders and
included the following:

QEP Kick-Off celebration- -66 faculty and staff attended and filled out an awareness
and interest survey.

QEP Topic Development Forums - - five events offered across campus.

Presentations to key stakeholders, including Faculty Senate, Staff Senate, Student
Government Association, Graduate Student Association, and Deans’ Council, to
solicit input.

Focus Groups -- volunteer group QEP Ambassadors (67) formed to participate in
Focus Groups to analyze the relative advantages and disadvantages of potential
topics.
o
Ambassadors volunteered as representatives of their units to participate in
QEP-related events broad-based participation from all units in topic
selection.
o
Five focus groups with 87 members further explored what topics would best
suit UNCW.
o
The topic “Applied Learning” was embraced widely as the “most powerful
learning experience” due to its potential for broad appeal across campus.

QEP Town Hall - - QEP’s Applied Learning focus was announced and additional
feedback solicited.

QEP Pilot Program-- launch and assessment in 2011-12.
8

Initial program structure proposal-- March 2012 with feedback solicited prompting
significant revisions.

Preliminary implementation-- began in 2012-13.
Continuous Stakeholder Engagement
On November 1, 2010, the campus community was invited to an event celebrating the launch of
the QEP topic development process. QEP Kick-Off components were intentionally chosen to facilitate
and further stakeholder engagement. The QEP project and process was described and enthusiasm was
generated for developing a QEP for UNCW. During the QEP Kick-Off celebration, sixty-six faculty and
staff responded to a survey capturing their views on UNCW strengths, growth areas, and rewarding
experiences. Attendees were also asked for their opinions on what academic skill is most important for
enabling student learning (see Figures 1 and 2 below).
Figure 1. Identified Strengths, Growth Areas, and Rewarding Experiences (data collected at QEP
Kick-Off event)
9
Figure 2. What Academic Skill is Most Important to Enable Student Learning? (data collected at
QEP Kick-Off event)
The Kick-Off celebration also included a presentation from University Assessment highlighting
findings from General Education Assessment, National Survey of Student Engagement, Applied Learning
in the College of Arts and Sciences, among other reports. The QEP Topic Development Task Force
studied these findings to better understand the UNCW academic landscape and the importance of this
information for stakeholder education and evidence-based topic selection. Subsequent campus-wide
forums and focus groups explored more fully the advantages and opportunities the QEP’s potential topics
could provide for the UNCW community.
Pilot Program
A pilot program for UNCW’s applied learning QEP was launched in fall 2011. The program was
funded by the Chancellor’s office and included five pilot groups:

Expanding the Professoriate formulated a plan to implement applied learning activities across
campus by employing faculty fellows, post-doctoral fellows, and graduate students to focus and
guide the process. This new teaching and learning community would work in conjunction with
graduate students, as well as faculty and administrators, to expand and improve applied learning
10
at UNCW with regular workshops, themed discussion sessions, and celebrations. Particular
emphasis was on discussion and recognition of high-impact practices for UNCW.

Experience Research extended the work of the Center for Support of Undergraduate Research
and Fellowships (CSURF) staff to pursue faculty-mentored undergraduate research and creative
activity. These in-depth research opportunities extend to summer internships and grants and
informative outreach efforts to first year students, and offer development and skills training for
sophomores to seniors. CSURF also assists faculty in including more research experiences in
their courses and developing new courses explicitly designed to teach research methods earlier in
the students’ academic careers.

Explorations of the Natural World provided a unique opportunity for biology majors to engage in
applied learning outside of the traditional classroom environment through a collaborative directed
independent study (DIS) structure. Outdoor applied learning experiences were built into a
collaborative DIS effort, so that students interacted more often with multiple faculty members,
each other, and the abundant natural resource sites that the UNCW campus and environs have to
offer, including the Ev-Henwood Nature Preserve, the Bluethenthal Wildflower Preserve, and the
natural habitats in southeastern North Carolina. This collaborative DIS resulted in undergraduate
students, graduate assistants, and faculty mentors sharing knowledge, laboratory time/resources,
and research skills in achieving a common biological purpose.

Applied Learning Improving the Southeast focused on regional engagement by sending students
out into New Hanover, Columbus, and Brunswick counties to work with the area’s
underprivileged children. Drawing on one aspect of UNCW’s mission statement regarding
community engagement, students participated in poverty simulations and other campus-wide
awareness campaigns. This regional engagement component was designed to enhance UNCW
student learning while also enriching the region’s communities.

Standardized Evaluation of Applied Learning formulated and implemented a standardized
framework for evaluating the influence of applied learning experiences in service learning
experiences by monitoring a student’s academic skills. By using broad-based and general
measures that could potentially be applied in evaluating applied learning across multiple UNCW
schools, departments, and programs, students’ applied learning experiences and outcomes were
measured, evaluated, and assessed in an effort to determine if this approach would be an effective
learning technique for UNCW students.
A total of 521 students and 92 faculty and staff members participated in the five pilot projects.
Instructors and students in each pilot group were expected to articulate their “intentions” for the
11
learning experience and capture “reflections” on the student learning experiences. Multiple meetings
and discussions identified common threads for measurement and assessment purposes. Pilot team
instructors followed common student learning outcomes and prompts for students to reflect on the
learning experience.
Preliminary eTeal Student Learning Outcomes
While individual applied learning experiences addressed specific learning outcomes, students
participating in all eTeal experiences would:
o Discuss how subject-matter content and skills-in-practice were combined to shape
the experience or project, and
o Reflect on their learning by examining prior knowledge and assumptions, outcomes
of actions taken during the experience, and their impact on or significance for
Pilot Program Assessment
Instructors participating in the pilot program delivered their articulated intentions and
student reflections to the QEP Task Force Assessment subcommittee in December 2011.
Although all assignments except one used the same reflection prompts, the format in which
students were required to submit their reflections differed. For some projects, students responded
through an online survey to each point in the prompts, for others, students responded in three
brief essays, and for the rest, students were expected to create a holistic essay responding to the
prompts. Student reflections were scored using a modified version of the Association of
American Colleges and Universities (AAC&U) Valid Assessment of Learning in Undergraduate
Education (VALUE) Rubrics. The dimensions that were chosen were those that matched best
with the reflection prompts: analysis of knowledge, transfer of knowledge, reflection, selfassessment and evaluation of outcomes. Seven QEP Task Force members volunteered to score
the student reflections. Scorers attended a two-hour norming session on the rubric prior to
scoring 159 student work prompts on their own. Fourteen work products were either completely
blank or contained phrases such as “NA” or “don’t know” and were removed from the analysis.
Fifty-three work products were double scored, and the average score was used for the analysis.
12
Analysis of
Knowledge
1.64
1.14
2.00
1.07
0.81
0.68
0.59
0.86
NA
Transfer
2.32
1.04
2.25
1.04
0.95
0.81
0.75
0.85
0.93
0.91
0.92
1.15
All papers
0.99
1.07
Table 1. Means Scores and Standard Deviations
demonstrated, 4=capstone learning demonstrated)
Experience
Research*
Explorations of
Nat. World
Standardized
Eval. Of A.L.
Improving the
Southeast**
Expanding the
Professoriate
Reflection and
Reflection
Evaluate
Self-Assessment
Outcomes
2.36
2.23
1.90
1.18
1.06
1.37
2.00
2.13
2.20
1.31
1.25
1.09
1.00
1.10
0.38
0.89
0.89
0.52
0.82
1.18
0.88
1.05
1.03
1.04
1.63
1.51
0.67
0.73
0.77
0.76
1.39
1.48
0.99
1.12
1.03
1.09
by Pilot Project (159 student products scored: 1=weak learning
*Honors Projects, Hosier Fellowships, Travel Awards, Research Supplies Awards
**Includes students from UNI 101, EDN 200 & 348, PSY 247, NSG 486 & 503, and SOC 215, 320 & 390
It was noted that the form of response played an important role in the thoughtfulness of the
reflective responses. Two portions of the pilot, Improving the Southeast and Standardized Evaluation of
Applied Learning used survey software to collect reflections, and students were required to answer each
prompt separately. Students in two portions of the pilot, Experience Research and Explorations of the
Natural World, provided holistic responses more often in essay format. Students who responded
holistically to the list of prompts tended to provide more analytical and reflective responses.
The Evaluate Outcomes dimension was not originally part of the rubric when prompts were
written and disseminated. This dimension was used in the pilot to assess its value moving forward.
Results demonstrated that a minority of students (29.6%) addressed outcomes in their reflections,
although they were not directly prompted to do so.
In addition, scorers were asked to evaluate the rubric as it related to the student reflections.
Overall, scorers observed a paucity of depth in student reflections based on prior knowledge and transfer
of knowledge. Student reflections that were captured through holistic essays provided much more depth
and thoughtfulness than reflections captured through survey formats. When students are taught that
reflection is a process, not only a product, most seem better equipped to demonstrate their learning
through deeper reflection opportunities. Employing the rubric illuminated the need for a more refined and
targeted rubric for QEP goals.
13
Lessons Learned




“Intentionality” alone is insufficient for guiding applied learning experiences.
“Reflection” is a complex process and most instructors wish to utilize it more effectively to
enhance applied learning experiences for students.
eTEAL should support instructors in helping their students reflect on applied learning.
eTEAL should invest in “critical reflection” as a process and a product for capturing student
learning.
In December 2011 and January 2012, seven focus groups were conducted with students and
faculty who participated in the pilot project to gain more depth of understanding into the students’ applied
learning experiences. The student focus groups endorsed applied learning pedagogies and reported having
gained important academic skills in critical thinking, communication, technical skills in the discipline,
and more self-confidence as learners. These students clearly understood that applied learning, as a “highimpact” practice, requires considerable resource investment and desired to see this, in the words of one
participant, “promoted in the faculty, … in a way that faculty are able to feel excited as opposed to feeling
it as more of a burden.” Students participating in the focus groups enthusiastically reported that they
would participate in another enhanced applied learning experience if given the chance.
Pilot faculty focus group members reported some difficulties with “intentionality” and
“reflection” strategies and desired resources and support for improving those pedagogical skills. They
also noted that students were much more engaged in the learning process when applied learning
techniques were used, and they would definitely use these techniques again, with additional support from
the University through the QEP. Faculty members reported that enhanced applied learning was a very
rewarding teaching experience, which they would repeat with appropriate support. They requested a
workshop or series of workshops to generate ideas and learn what colleagues across campus and external
specialists recommend. Through this interaction, developing consistent support and resources to improve
student learning and applied learning pedagogy became the main goals of eTEAL.
The original pilot team was repurposed into a preliminary implementation group that served to
test specific aspects of the plan. Expanding the Professoriate provided the inspiration for an effort named
the Applied Learning and Teaching Community. Experience Research confirmed the importance of
14
undergraduate involvement in research as applied learning. Explorations of the Natural World inspired an
innovative approach to DIS work involving students in collaborative research teams with faculty pursuing
a common applied learning experience. Improving the Southeast and Standardized Evaluation of Applied
Learning both demonstrated the importance of critical reflection for community engagement and service
learning options in applied learning settings. The original pilot program for eTEAL generated valuable
lessons from the College of Arts and Sciences, the College of Health and Human Services, and the
Watson College of Education, and demonstrated the value of broad-based participation in testing the
original ideas and informing the design of this proposal.
What students said about their eTEAL-Pilot experiences:
Anytime you can go out and do what is in the book, it's a benefit.
Having to draw on the knowledge you learned in class and apply it to situations [outside] was rewarding.
As you are doing research or even when you finish, you have more questions than ever before.
[Critical thinking] is inherent in the project. You couldn’t just look at info on a website, you had to compare and judge the validity of the sources where you got your information and ask “is the information current, reliable?”
Every minute of it was enhancing critical thinking skills. It’s almost the objective of what we were doing every day.
It made me a better problem‐solver and critical thinker.
Figure 3. Student Assessments of eTEAL-Pilot Experiences
15
What faculty members said about their eTEAL-Pilot experiences:
Most important outcome is student and faculty interaction and engagement; I think this really increases critical thinking!
We can improve applied learning pedagogy by sharing it, and how it improves our courses, with others and then invite others to partner with us, i.e. showcase of campus for students to share their experience. [Then] faculty see the benefit and they may continue to use the methodology.
Proving spaces (literal/metaphorical) for faculty [and staff] to exchange ideas and work together [is important]. Faculty can inspire and resource each other but need places and events to help them gather and connect.
Figure 4. Faculty Assessments of eTEAL-Pilot Experiences
Pilot Program Outcomes
Pilot program findings positioned the QEP Task Force to hone the QEP focus. Specifically, faculty,
staff, and students stated a desire for the learning experience to be engaging and relevant to their
educational goals. Multifaceted enterprise-wide investment in professional development was identified as
a key to the enrichment of applied learning pedagogy. Workshops, travel, supplies, and time away from
classes to discuss and discover successful practices were further identified as core components of
professional development. The QEP Task Force recommended that UNCW’s QEP should therefore
provide the following resources and opportunities:

Regular opportunities to discuss, discover, disseminate and cultivate high-impact practices in
applied learning and improved reflection.

Professional development for faculty and staff to develop and design applied learning experiences
in eTEAL.

Adequate monetary support for implementation and reflection of applied learning, and to provide
materials and supplies for student learning, including travel to off-campus learning sites and
locations where students are involved in community engagement, expeditions off campus to
16
observe, explore, and interact with the subject of study, and travel to present findings of research
at scholarly meetings, among others.
These recommendations inspired a model utilizing a three-part approach. From January through
March 2012, two working groups were established to further develop this plan - the Design Team and the
Resource Team- both of which included members of the QEP Task Force and participants from the pilot
project. In March 2012, the QEP Task Force prepared and released a first draft of the proposed program
and solicited feedback and ideas for revisions/improvements in the plan from colleagues across campus.
eTEAL was refined during the summer of 2012 and organized around a three-part cycle.
Input that contributed to the refinement of eTEAL included the widely read original model
distributed to campus in March 2012, feedback from Faculty Senate and the pilot participants, as well as
faculty, staff, and students across campus. Input also prompted further revisions of the definition of
applied learning and the inclusion of critical reflection as cornerstones of the plan. Based on feedback
collected, the QEP Task Force conducted a further literature review for information regarding highimpact practices for applied learning and how best to promote student learning through reflection. UNCW
contracted with the National Society for Experiential Education to host a workshop on that organization’s
“good practices” and with PHC Ventures to host a workshop on “critical reflection” practices. As a result
of these workshops the definitions of applied learning and critical reflection were modified. The QEP
Task Force then continued to work on articulating the program goals, outcomes, and assessment methods.
The eTEAL three-part plan is designed to have a direct impact on student learning. High-impact
practices including intentionality, formative assessment, and critical reflection, helping students better
situate the applied learning experience within their educational progress. As such, the applied learning
experience becomes a vital link to richer learning. Students better connect their prior learning to the
experience, reinforcing the importance of their prior learning and improving learning from the experience.
Critical reflection also helps students extend applied learning to better understand their learning. While
each eTEAL opportunity incorporates its own mix of successful practices, critical reflection is essential
to all eTEAL experiences. The results obtained from assessments of the various experiences will be
shared across campus to stimulate discussion, including through the Applied Learning and Teaching
Community (ALTC), established to explore ways to enhance the applied learning experience and the
effects of pedagogy on student learning. The venues for applied learning are deliberately designed to
include but are not limited to:


Undergraduate Research
Directed Independent Studies (including collaborative DIS, such as in the pilot)
17




Community Engagement
Internships, Practica, and Service Learning
Leadership and non-curricular opportunities
Study Abroad
An Iterative Cycle Leads to a New Normal
The three part program for eTEAL is intentionally designed to produce iterative cycles of
continuous improvement in student learning and the environment supporting student learning.
Once captured, the evidence of student learning will be reported to campus and used to improve the
program as it develops over the course of five years. By continuously monitoring program
outcomes, the effectiveness of eTEAL will be measured so that improvements can be made. The
iterative cycle includes lessons from the Applied Learning Summer Institute described and
discussed in the Applied Learning and Teaching Community, communicated to the broader campus
community, and used to inspire enhanced applied learning pedagogy implementation throughout
the course of the program. Furthermore, eTEAL mentors will be available to work with instructors
who desire more specific support for their particular applied learning interests. When the program
concludes, the “new normal” at UNCW will feature the offering of consistently high-impact
applied learning experiences for students and a community of supportive colleagues promoting
applied learning across campus.
18
Applied Learning Summer Institute (developing and designing)
Participating Instructors and students Applied Learning
Applied Learning and Teaching Community (discussing, disseminating, and cultivating)
eTEAL‐Supported Initiatives (implementing and reflecting)
Figure 5. eTEAL Components and Processes
eTEAL Preliminary Implementation
For 2012-13, UNCW has continued preliminary implementation of eTEAL, focusing on testing
all three parts of the proposed model.
UNCW hosted the first Applied Learning Summer Institute in 2012. The workshop leaders were
Dr. Roseanna Ross who covered “good practices” in experiential education on behalf of the National
Society for Experiential Education, and Dr. Patti H. Clayton who covered high-impact approaches for
critical reflection. An announcement was distributed across campus inviting faculty and staff to
participate in the workshop on July 30, 31, and August 1, 2012. Forty people signed up to participate and
34 attended. Participants were invited to share what motivated their interest in participating and what they
hoped to gain from the Summer Institute and indicated a desire to learn more about applied learning
pedagogies and how to improve student learning through critical reflection. At the conclusion of the
Summer Institute, participants shared their reactions.
19
Pre‐workshop Reflections
Post‐workshop Reflections
I use various forms of reflection exercises in all my classes despite never having received any formal training. I want to learn about the practice as a whole and how I can improve upon what I’m doing now and add other exercises at the undergrad and grad levels.
I got more than I hoped for from the workshop. It was good to hear what other people are doing related to applied learning. It was a great opportunity to network and to learn from other individuals. The presenters brought great information and the participants were equally enriching.
I’m an avid fan of applied learning and have been incorporating it in my classes for some years now. But, my use of it has mostly been instinctual – I hadn’t read any of the pedagogical literature until I was asked to write chapters on my approach to teaching [as a book chapter]. … but, I know I am missing some pieces – e.g., the critical reflection component, so I am particularly interested in learning about that. I came in open to the workshop. Consequently, I was pleasantly surprised and pleased that not only was I validated in what I was already doing, but I also found ways to improve my teaching.
My motivation to attend this workshop stems from my interest and hope to: 1) deepen my current understanding of applied learning; 2) learn new experiential learning strategies for implementation, assessment, and reflection; 3) discuss current theory in this area and connect it to my practice; and 4) my desire to share what I learn with my students.
I'll be thinking through applied learning/experiential learning activities in all my courses to maximize their benefit in light of the concepts discussed here. Figure 6. 2012 Applied Learning Summer Institute Comments
20
Part of the preliminary implementation, the Applied Learning and Teaching Community (ALTC),
operates with the assistance of the Center for Teaching Excellence. Two Applied Learning Fellows are
identified each semester and earn either one course buyout or overload pay for their contributions. The
ALTC encourages and supports colleagues in using successful practices in applied learning, and compiles
and disseminates information (successful practices, pilot projects, etc.) about those efforts. Furthermore,
Fellows aim to cultivate support and enthusiasm for applied learning and eTEAL. A common goal for the
summer workshops, the ALTC, and the Applied Learning Fellows is to discuss and develop ideas related
to applied learning pedagogy and critical reflection and collaborating with the QEP Task Force. The
ALTC also distributes a newsletter to support publicizing eTEAL and successful practices, announces
events and opportunities, and disseminates lessons learned. Thus, the ALTC helps to maintain long-term
sustainability of eTEAL.
Companion to the ALTC are two continuing strands from the 2011-12 pilot programundergraduate research and collaborative DIS work. Both directly address the UNCW Learning Goal of
Inquiry and provide innovative approaches to teaching and undergraduate research through applied
learning. Continued development of undergraduate research helps the QEP Task Force to refine how such
research can be improved as one aspect of applied learning in multiple fields. By teaching lower-level
classes in research methods, using applied learning pedagogy and critical reflection, the QEP Task Force
is learning how best to implement undergraduate research as applied learning. This team is developing
regular courses exposing students to undergraduate research methods earlier in their academic career to
empower them with research skills earlier and longer. Course development stipends, materials, and
student project supplies have been provided to deliver five targeted sections of HON 292 (1cr) in spring
2013 and to support student travel. The courses will utilize successful practices in applied learning,
develop student learning outcomes, capture evidence of student learning, and participate in the ALTC.
The collaborative DIS pilot also continued from 2011-12, and key participants attended the 2012
Summer Institute. Helping to address goal three above (to promote the incorporation and implementation
of successful practices of applied learning throughout UNCW), the Summer Institute is designed to
inspire innovation with eTEAL-Supported Pedagogy Initiatives. This pilot group is developing an
improved applied learning experience through collaborative DIS work with a team of three faculty, eight
undergraduate students, and one graduate student researching linked biological questions. This group is
receiving support for materials/supplies and student travel. Duties include articulating student learning
outcomes and implementing successful practices in applied learning, documenting and capturing evidence
of student learning, participating in the ALTC, promoting eTEAL in their units and beyond, and
reporting to and collaborating with the QEP Task Force.
21
Finally, the preliminary implementation phase involved offering eTEAL-Supported Pedagogy
Initiatives for the first time. Solicited and funded initiatives emerging from the Applied Learning Summer
Institute participants (early adopters) aligned learning outcomes that were prepared by each instructor. In
the fall 2012, ten early adopters were supported:
Darwin Dennison (HAHS) HEA 207 Nutrition and Behavior
• This experiential, applied learning experience with continuous reflection provides students professional experience assisting participants with nutritionally preventable or mitigating conditions including overweight‐obesity (oo), diabetes (nidd), and/or hyperlipidemia (hyl).
Kristen DeVall (SOC/CRM) SOC/CRM 300 Research Methods
• Using Bloom’s Taxonomy as a framework, I will develop six in‐class activities and assignments that specifically address each of the learning objectives: Identify, Explain, Apply, Analyze, Synthesize, Evaluate. Kathleen Ennen (NUR) NUR 404 Health Policy
• Students wrote a “health policy issue paper” that includes going to a community to learn its implementation, its impact and effectiveness, and how the policy has been actualized by those who had to put it into action.
Jennifer Horan (PIA) and Kate Bruce (PSY) HON 110
• Students completed a service requirement, a cultural event requirement, and a “dream job” assignment in order to learn about and critically reflect on transitional learning opportunities. Michele Parker (WCE) EDN 642 Research II
• developed school teachers' research skills; analyzing, interpreting, and reporting research results ‐‐ including focused study and application of descriptive and inferential statistics. Colleen Reilly (ENG) ENG 313 Writing About Science
• to add a new service‐learning experience and enhance the current applied learning experience in this writing‐intensive course. Shannon Santana (SOC/CRM) CRM 490 Criminal Justice Practicum
• Criminal Justice Practicum, incorporates some of the eight principles of good practice for experiential learning into my practicum course.
Michelle Scatton‐Tessier (FLL and WST) WST 490 Women's Studies Internship
• Developed ways to incorporate better practices of critical reflection into the internship.
Rajni Shankar‐Brown (WCE) EDN 355 Language Arts Methods
• the overarching goal of this project is for students to refine, apply, and extend the foundation of knowledge, skills, and strategies developed in diverse settings that illuminate social inequality.
Patricia White (NUR) NUR 401 Pediatric & Community Nursing
• This is a group of 50 students who are required to incorporate concepts of nursing, humanities, and the fine arts to portray the needs of a specific patient they have selected. To achieve success with this assignment, the students are encouraged to stretch themselves through critical thinking, inquiry, and creative thinking.
Figure 7. Fall 2012 eTEAL -Supported Pedagogy Initiatives
22
The eTEAL-Supported Pedagogy Initiatives are at the heart of applying improved pedagogy in the
classroom and documenting student learning. The Initiative participants contribute to improving and
promoting two foundations of eTEAL (instructor-centered development, intentionality and critical
reflection in applied learning), and disseminate findings, and contribute to the documentation of student
learning through campus-wide events and other avenues for building on these experiences. Their applied
learning expertise will continue to inform the basis of the QEP over the course of the program and as it
evolves.
Topic Selection
Discussion of topic selection is, as is demonstrated in the preceding section, part and parcel of
discussion of the QEP development process. The development of the QEP topic included reviewing
existing evidence on student learning, taking stock of the applied learning environment at UNCW, and,
through forums and focus groups, continuously collecting feedback from across campus regarding areas
for improvement. Finally, reviewing the scholarly literature in specific areas also informed QEP topic
selection.
Evidence-Based Topic Selection
An important part of the overall process of defining the QEP topic was the collection of evidence
on student performance reflecting strengths and areas for potential improvement. Equally important was
the collection of evidence on the current applied learning landscape at UNCW.
Early in the topic development process, and continually thereafter, evidence was reviewed from a
variety of sources. These included assessment evidence of student performance on the eight UNCW
Learning Goals, surveys of faculty and staff perceptions, and review of evidence from the existing applied
learning programs on the campus.
UNCW assesses its Learning Goals at the University Studies level through the use of embedded
course assignments. A strength of this approach is that the student work products are an authentic part of
the curriculum, and hence there is a natural alignment directly to the student work in the course that is
often missing in standardized assessments. Students are more motivated to perform at their best because
the assignments are part of the course content and course grade. Furthermore, this method includes
faculty collaboration and full participation in both the selection of the assignments and the scoring of the
student work products.
23
The student work products are scored independently on a common rubric by trained scorers. The
results of this scoring provide quantitative estimates of students’ performance and qualitative descriptions
of what each performance level looks like, which provides valuable information for the process of
improvement.
Results for Critical Thinking
Results from student work scored on critical thinking were the lowest across all Learning Goals.
Within the dimensions of critical thinking, student performances were scored highest on explaining the
issues (CT1), with over a third of the students being able to clearly describe and clarify the issue to be
considered (scores of 3 and 4), and another third able to describe the issue, although with some
information omitted (scores of 2). Students had the most difficulty identifying context and assumptions
when presenting a position (CT3), and tying conclusions to a range of information, including opposing
viewpoints and identifying consequences and implications (CT5). With dimension 2 divided into two
components, the scores on the first part, interpreting the evidence and developing an analysis (CT2.1),
were the highest of all dimensions, and the scores on the second part, questioning the viewpoints of the
experts (CT 2.2), were the lowest . The information found from dissecting this dimension suggests the
need for more discussion about verifying the source of evidence and examining for potential biases. In
addition, when reviewing these findings, the QEP Task Force understood that framing UNCW’s QEP in
such a way as to impact critical thinking skills would be vital.
% of Work Products Meeting Benchmark
Dimension
Spring 2010
Fall 2010
CT1 Explanation of Issues
64.1%
73.7%
CT2 Evidence Year 1
65.0%
N/A
CT2 Evidence Year 2* Interpreting and Analyzing
N/A
72.8%
Questioning View Point
N/A
40.9%
CT3 Influence of Context and Assumptions
40.3%
59.2%
CT4 Student’s Position
51.1%
58.2%
CT5 Conclusions and Related Outcomes
37.0%
59.6%
N
183
114
Table 2. Critical Thinking Percent of Sample Meeting Benchmark
*In year 2, this dimension was divided based on scorer feedback.
Both the Collegiate Learning Assessment (CLA) and College Assessment of Academic
Proficiency (CAAP) results suggested an increase in student scores between freshman and senior years.
UNCW students compare favorably on the CLA with students from other participating universities. The
24
National Survey of Student Engagement (NSSE) results indicate that UNCW coursework compares
favorably with the coursework at other Carnegie-class institutions when it comes to student perceptions
about critical thinking being required.
Results for Inquiry
Student work from general education courses in both 2009-10 and 2011-12 demonstrated a high
level of achievement with regard to inquiry skills. In spring 2010, with student work sampled from 100and 200-level courses, scores were among the highest across the goals assessed. There were statistically
significant, though not large, increases in scores as credit hours completed increased for four of the
dimensions, demonstrating the positive impact of coursework on inquiry skills. The fall 2011 results were
from 100-level lab courses and were strong on three of the dimensions. The fact that IN1 and IN2 were
not applicable to these lab assignments is not necessarily problematic. The pedagogy of scientific inquiry
uses a guided approach, in which students are introduced to the inquiry process using well-established
laboratory questions. Scores for IN6 Limitations and Implications indicate that reflecting upon and
discussing limitations and implications of the inquiry process are areas for improvement.
% of Work Products Meeting Benchmark
Spring 2010
Fall 2011
71.5%
NA
91.4%
NA
83.7%
87.3%
82.6%
72.6%
70.4%
77.0%
62.3%
39.5%
98
339
Dimension
IN1 Topic Selection
IN2 Existing Knowledge
IN3 Design Process
IN4 Analysis
IN5 Conclusions
IN6 Limitations and Implications
N
Table 3. Inquiry Percent of Sample Meeting Benchmark
Results for Thoughtful Expression (Written Communication)
Results from the general education assessment process, which assesses student work from
representative general education courses, were strong in comparison to other UNCW Learning Goals.
Relative strengths were demonstrated in Context of and Purpose for Writing and Control of Syntax and
Mechanics.
25
% of Work Products Meeting Benchmark
Spring 2010
Spring 2011
200-level courses
200-level courses
84.4%
76.3%
76.8%
67.5%
74.1%
67.6%
64.7%
71.4%
76.7%
83.9%
116
80
Dimension
WC1 Context of and Purpose for Writing
WC2 Content Development
WC3 Genre and Disciplinary Conventions
WC4 Sources and Evidence
WC5 Control of Syntax and Mechanics
N
Table 4. Written Communication Percent of Sample Meeting Benchmark
The Collegiate Learning Assessment (CLA) was administered in a longitudinal study to
freshmen, sophomores, and seniors. Students assessed in the first and in the final years demonstrated an
increase in analytic writing abilities, with an effect size of 1.82, which was well above the expectations of
the test creators. However, the sample size for participation across the four years was small. Results from
the National Survey of Student Engagement (NSSE), specifically responses from seniors, suggest that
UNCW students prepare two or more drafts of a paper about as often as students at other institutions
within the Carnegie classification and write papers of 20 pages or more slightly less often. Although
written communication needs will be addressed through the new University Studies component Writing
Intensive courses, the University community determined that written communication skills would be a
logical part of any QEP topic.
Review of the student learning evidence on UNCW’s remaining Learning Goals demonstrates
that, in general, UNCW students perform as expected and thus are not specifically included in this Quality
Enhancement Plan. The findings are briefly summarized to validate this conclusion.
Results for Information Literacy
The benchmarks differed for the two courses, with a benchmark of 2 on the rubric from 200-level
courses, and a benchmark of 3 for the 300-level courses. The percent of student work meeting or
exceeding level 2 was about the same for the 300-level courses as the 200-level courses, indicating a lack
of gain between these course levels, although there were limitations of the 300-level assignments. The
implementation of the new University Studies component Information Literacy was deemed to be the
appropriate way to address this Learning Goal.
26
% of Work Products Meeting Benchmark
Spring 2011
Spring 2011
200-level courses
300-level courses
89.0%
52.5%
91.0%
70.0%
88.0%
22.9%
88.0%
76.9%
90.0%
42.5%
78
40
Dimension
IL1 Extent of Information Needed
IL2 Access Needed Information
IL3 Evaluate Information and Sources
IL4 Use Information for a Purpose
IL5 Access and Use Ethically and Legally
N
Table 5. Information Literacy Percent of Sample Meeting Benchmark
Results for Diversity
With the introduction of the Living in Our Diverse Nation component of the general education
curriculum in fall 2011, student factual knowledge of diversity issues is expected to improve. This and the
overall survey results are a good indication of the effects of this new component of the curriculum.
% of Work Products Meeting Benchmark
Fall 2010
Spring 2012
64.6%
85.5%
57.7%
71.9%
Dimension
DV1 Factual Knowledge
DV2 Contextual Importance – Year 1*
DV2 Knowledge of Diverse Perspectives and Their
Roots – Year 2*
DV3 Examining Diversity, History, and Culture
DV4 Evaluating Claims and Theories about Diversity
N
113
61.8%
82.6%
256
Table 6. Diversity Percent of Sample Meeting Benchmark
Results for Global Citizenship
Having greater than 64% of students meeting the benchmarks for this goal suggests that initial
efforts may be effective in meeting expectations for the Living in a Global Society component of the
general education curriculum and that further analysis of these efforts as they are fully implemented is an
appropriate approach. Scores did not improve with credit hours completed, which might be expected for a
new requirement. It may also indicate that more emphasis of Global Citizenship is needed either in
University Studies or the majors.
27
Dimension
GC1 Factual Knowledge
GC2 Knowledge of Connections
GC3 Use of Diverse Cultural Frames
GC4 Tolerance of Differences
GC5 Ethical Responsibility
N
% of Work Products
Meeting Benchmark
63.9%
65.7%
66.5%
76.2%
64.9%
155
Table 7. Global Citizenship Percent of Sample Meeting Benchmark
Results for Foundational Knowledge
Evidence gathered by academic major showed that students were performing at high levels on
discipline knowledge and indicated weakness in applying accumulated knowledge to new situations.
Math results from within general education courses suggested that most students (92%) can work with
linear equations, a benchmark mathematical tool for all students. Sociology results suggest more work
needs to be done on the rubric and the timing of the assessment. Results from PED 101 (Physical
Education) did not show much gain in knowledge. However, it was determined by faculty through a
review of course objectives that a better alignment of assessment and curricular content was needed.
Results for Second Language
Spanish is the language chosen by most UNCW students to meet the second language
requirement. Results over the last three years in Spanish demonstrate strength in most areas of writing,
with some need for growth in grammar and vocabulary variety. French results also show a relative
weakness in grammar. A German study, which looked predominantly at the second language cultural
competence Learning Outcome, showed strong vocabulary ability. Other languages have not been
assessed through this process.
28
Summary of UNCW Learning Goals Findings
Student abilities in understanding purpose, explaining issues, and presenting information—those used
to begin communication and investigation—are stronger than their skills in identifying assumptions,
stating conclusions or positions, and discussing limitations, implications, and consequences—those
used to critically evaluate information. The findings all point to the need to provide students with more
opportunities to apply knowledge in situations where they can practice higher-order thinking skills,
falling within the Learning Goals of Critical Thinking, Inquiry, and Thoughtful Expression.
Narrowing the Topic: Focusing on Applied Learning
UNCW has an extensive history with applied learning. The professional programs rely heavily on
applied learning to give students experiences in applying knowledge and skills gained in courses. In 2006,
the College of Arts and Sciences formally adopted applied learning as a central priority under the
University’s Strategic Goal “Creating a Powerful Learning Experience,” and by fall 2008 applied learning
experiences were available and required for all entering undergraduate students as part of UNCW’s
Applied Learning Initiative. In addition, the general education curriculum includes a common
requirement for students to engage in Explorations Beyond the Classroom (EBC), which are defined as:
discovery, application, regional engagement, and exploration away. The QEP is designed to improve
applied learning using successful practices in pedagogy, provide professional development opportunities
for instructors, and systematically capture evidence of student learning for QEP assessment purposes.
Specifically, the aim is to improve student learning outcomes in Critical Thinking, Thoughtful
Expression, and Inquiry as measured in eTEAL-supported applied learning experiences.
Information about the nature and extent of these applied learning experiences was found in a
number of places. The NSSE findings showed that 85% of seniors reported that they had participated in at
least one applied learning experience at UNCW, with most participating in two or more. However, seniors
reported fewer collaborations with faculty than was expected. The QEP Topic Development Task Force
solicited information on the quantity of applied learning experiences for 2010-11.
To gain a more comprehensive understanding of applied learning experiences, studies were
undertaken in the College of Arts and Sciences in 2008-09 and 2009-10. Evidence from two annual
reports titled Applied Learning in the College of Arts and Sciences provided quantitative and qualitative
information on the experiences and showed that all departments in the College delivered at least one
course-embedded applied experience and most delivered multiple experiences.
29
The types of course-embedded experiences included developing materials for a client, working in
a scientific setting, and working with data or organizing information. For internships and DIS, students
and their faculty mentors reported learning objectives and rationales. The typical rationale involved
applying what they already knew as well as learn new information and skills (see Figures 8 and 9). The
most typical requirements of DIS projects were oral reports and meetings, and the duties for internships
were most often negotiated after the experience started (see Figure 10 and 11). Requirements for Honors
projects were much more standardized than other individualized studies. An important limitation of this
survey of applied learning DIS experiences is that it was based primarily on pre-approval forms. The
survey did not indicate any standardized approach to collecting evidence of the outcomes of the
experiences.
500
450
400
350
300
250
200
150
100
50
0
Figure 8. Students’ reasons for enrolling in Directed Individual Studies, 2008-2009.
30
300
250
200
150
100
50
0
Gain Experience
Learn
Use/apply skills
Figure 9. General internship learning objectives, 2008-2009.
31
450
400
350
300
250
200
150
100
50
0
Oral or written
reports
Meetings
Readings
Lab Hours
Notebook
Literature
review
Other
Figure 10. Types of requirements for students enrolled in Directed Individual Studies, 20082009.
160
140
120
100
80
60
40
20
0
Figure 11. Types of internship duties, 2008-2009.
32
In summary, the QEP Task Force found that there was a need to provide students with
opportunities to apply knowledge in situations where they can practice critical thinking, thoughtful
expression, and inquiry skills. While UNCW students participate in over 7,000 applied experiences each
year, the documentation of the results of these experiences has been very limited, and to date, no
systematic efforts to capture evidence of enhanced student learning directly from applied learning
experiences have been undertaken. Faculty from across the institution perceived that applied learning
experiences were both an area of strength and an area for improvement. They also perceived courseembedded experiences and critical thinking skills as areas of need. This existing evidence provided the
foundation for the QEP topic and framework.
Direct Assessment of Student
Learning
•The QEP should have a direct impact on
critical thinking skills.
•The QEP should provide experience for
developing inquiry skills.
•The QEP should promote and measure
students' skills in thoughtful expression.
•With acceptable scores for information
literacy, diversity, and global citizenship, the
QEP need not focus on these directly.
However, the QEP should aid in maintaining
these scores.
Faculty/Staff Perceptions
•Existing strengths include experiential
educational opportunities:
internships/practica, involvment in facultyled research, individualized instruction.
•Most important academic skills include
critical thinking and communication.
•Enthusiasm for rewarding teaching
experiences that are commonly identified as
"applied learning" at UNCW.
Figure 12. Summary of Institutional Evidence
Forums and Focus Groups Lead to Final Topic Selection
Campus-wide discussions cultivated interest and elicited feedback about basic aspects of UNCW
students and learning. Topic development forums and focus groups findings led the Task Force to refine
ideas from volunteer QEP Ambassadors and other faculty, staff, and students. Participants were asked to
share their experiences and views on possible QEP topics in light of assessment findings documenting
student learning achievements. These events helped the Task Force to narrow the scope of the discussion
into five potential and inter-related topics and realize that the final QEP topic would likely affect all these
areas:
33

Active Learning/Applied Learning

International experience/global awareness

Critical Thinking

Writing

Improve citizenship and civic/regional engagement
Data were collected at these forums and focus groups, compiled and analyzed using qualitative data
analysis techniques. Findings revealed that the majority of campus participants supported the following:

The pursuit of the most powerful learning experience at UNCW through Applied
Learning, which was initially defined by the QEP Task Force as: “Applied learning is
a pedagogical model that places students in authentic experiences. At the core of
applied learning is the idea that students are applying theories, ideas, and skills that
they learn to real-world problems.”

The initial concentration was on applied learning topics related to extensions of the
classroom activities, which are usually outside of the classroom, although real-world
needs could be brought into the classroom, to apply knowledge in solving problems
and synthesizing learning.
This broad topic area could include existing applied learning experiences, by enhancing them to
improve student learning in one or more areas and create an applied learning experience that would be
new to UNCW. Thus, the term applied learning emerged as advantageous to UNCW for the QEP for
several reasons. First, applied learning can be broadly defined and applicable to every major at UNCW,
thereby extending the QEP opportunities across campus. Second, applied learning can have beneficial
impacts on specific UNCW Learning Goals, including Critical Thinking, Thoughtful Expression, and
Inquiry. Third, applied learning can be accomplished in multiple learning venues including but not limited
to undergraduate involvement in research, directed independent study, internships/practica, service
learning, community engagement, and study abroad, as well non-curricular learning experiences. Also,
applied learning can be accomplished in innovative settings including travel to learning sites, group
projects, and opportunities to address specific needs such as “sustainability” or “community health.”
Fourth, scholarly literature demonstrates benefits to critical thinking skills and other transferrable skills of
applied learning pedagogies, all of which aligns with UNCW’s aim of improving student achievement in
Critical Thinking, Thoughtful Expression, and Inquiry (see Literature Review section). Scholarly research
also establishes that within applied learning venues, student learning is enriched through critical reflection
practice, which is a vehicle for improving and capturing direct evidence of student learning in applied
34
learning experiences as well as a mechanism for enhanced pedagogy. Critical reflection is a process
whereby student learning is deepened by guiding students to reflect on what and how they have learned.
Fifth, eTEAL is designed specifically to leverage the high level of campus-wide interest in and
enthusiasm for applied learning (by prioritizing instructors’ professional development) and to ensure that
campus participation is sustainable throughout the five years of implementation. Therefore, UNCW is
exploring the links between applied learning, critical reflection, and the specific Learning Goals of
Critical Thinking, Thoughtful Expression, and Inquiry in order to more effectively move students beyond
basic comprehension to application, inquiry, and synthesis.
Figure 13. eTEAL Conceptualization of Student Learning
Student Learning First and Foremost
With the first goal of improving student learning in applied learning experiences, eTEAL is
designed specifically to leverage the general education curriculum requirement for Explorations Beyond
the Classroom (EBC) that directly relates to specific UNCW Learning Goals. The student learning
outcomes for EBC are as follows:
I.
Student Learning Outcome 1. Students will articulate their expectations, the purpose, and/or the goals
of the experience in terms of their personal educational development. [Thoughtful Expression (TE)]
35
II. Student Learning Outcome 2. Students will synthesize knowledge drawn from their coursework to
address the issues/challenges/questions involved in the experience. [Critical Thinking (CT), Inquiry
(IN)]
III. Student Learning Outcome 3. Students will communicate the impact or significance on their personal
educational development and on others in the profession or in the field at the conclusion of the
experience. [Critical Thinking]
Students in classes designated as eTEAL-Supported Initiatives will pursue enhanced applied learning
where critical reflection products will be collected for assessment purposes. Student learning will be
assessed directly using common rubrics to score student critical reflections and other work products from
their experiences. Student learning, as well as impressions of the experience, will be assessed indirectly
through student surveys administered at the end of the experience.
To improve student learning in applied learning experiences
•Students articulate their expectations, purpose, and goals (TE)
•Students synthesize knowledge drawn (CT, IN)
•Students communicate the significance of their educational development at the conclusion of the experience (CT)
Figure 14. Merged Student Learning Outcomes and UNCW Learning Goals to Serve QEP.
36
The second and third goals of eTEAL will improve the environment supporting learning by
investing in professional development opportunities for faculty and staff instructors who want to enhance
their applied learning experiences for students. Two types of resources will be provided to faculty and
staff - - information about successful pedagogy and practices, and resources to support them as they
incorporate new practices into their courses and other experiences.
UNCW has developed a three-part approach to providing these resources. eTEAL is designed to
resource applied learning through instructor (faculty and staff) professional development opportunities.
The majority of eTEAL funds (see section on Resources) are devoted to resourcing faculty and staff to
deliver enhanced applied learning opportunities to UNCW students, thus enriching the academic core.
Each element of the initiative addresses specific features of the goals identified. Addressing goals 2 and 3,
the Applied Learning Summer Institute is designed to provide faculty and staff with resources (including
concentrated time for interaction) to promote discussion and sharing of strategies for applied learning
pedagogy and to hire national experts to explore successful practices that may apply to UNCW.
Ultimately, those who wish to do so can develop and design their applied learning experiences using
successful practices such as experiential education and critical reflection. This delivers important supports
for the first goal of improving student learning.
Also addressing goals 2 and 3 above, Applied Learning and Teaching Communities (ALTC) are
designed to provide development, discussion opportunities, and dissemination of successful applied
learning practices. Open to faculty, staff, and students, ALTCs provide a vital vehicle for identifying
(discovery), implementing (development), communicating (discussion), and sharing (disseminating)
ideas, skills, support, mentoring, and enthusiasm for eTEAL. Through the assistance of the Center for
Teaching Excellence, there is one ALTC operating now as part of the preliminary implementation.
Multiple Applied Learning and Teaching Communities may emerge over time.
Finally, the eTEAL-Supported Pedagogy Initiatives are designed to provide material support for
faculty and staff instructors to implement and deliver successful practices in applied learning and critical
reflection, and to document student learning through critical reflection in order to capture evidence for
goal #1.
37
Applied Learning Summer Institute
Open to Faculty and Staff Instructors
•Research‐based successful practices
•Develop plans for implementing applied learning experiences
•Encourage participants to propose funded‐initiatives
•Disseminate findings from other successful applied learning experiences
•eTEAL Coordinator organizes ALSI annually
Applied Learning and Teaching Community Open to faculty, staff, and students
•Discuss pedagogy
•Support pedagogy development
•Disseminate research‐based best practices for a applied learning
•Promote critical reflection in applied learning experiences
•Applied Learning Fellows (2/semester) operate the ALTC
•Disseminate lessons learned from pedagogy initiatives
eTEAL‐Supported Initiatives
Competitive proposals to request material support for applied learning
•Instructor and student critical reflection products for assessment purposes
•Participate in ALTC •Disseminate results with ALTC participants and Summer Institute participants
•Serve as mentors for emerging applied learning instructors
•Serve as reviewers for eTEAL Supported Initiatives in subsequent years
•Serve as scorers for assessment of student learning
•Designated as Applied Learning Scholars
Figure 15. Summary of eTEAL Components
Literature Review
A review of the scholarly literature relative to the various dimensions of eTEAL facilitates
understanding of the nuanced relationships between applied learning, critical reflection, and measuring
student learning. The scholarly literature on experiential learning, critical reflection, and Bloom’s
38
Taxonomy (commonly used for critical reflection practices) proved particularly insightful and provided
avenues to articulate these issues and aggregate relevant information.
Experiential Education and Experiential Learning Theory
Applied learning is a pedagogical model which utilizes experiential learning practices to enhance
students’ educational experience (Michaelsen and McCord, 2011). To explore fully the concepts and
practices of applied learning, a review of the literature on experiential education was undertaken within
which applied learning opportunities would sit. Additional information was drawn from the National
Society for Experiential Education (www.nsee.org). When engaging in experiential learning within the
classroom, various activities and methods can be implemented to increase active learning. Active learning
has been shown to improve students’ critical thinking, oral and written skills, and to generate more
positive attitudes about the learning process (Deely, 2010; Kolb and Kolb, 2005; Michaelsen and
McCord, 2011; Prince, 2004). Taking breaks during traditional lectures to introduce activities encourages
discussion, review, and reflection, a practice which is shown to enhance the students’ learning experience
by strengthening their ability to recall the information at a later date. To improve educational experiences
for students, instructors should provide feedback on the effectiveness of their learning efforts (Prince,
2004; Kolb and Kolb, 2005).
Experiential education refers to the transformation of a direct experience, which is processed
through an intentional learning format, into useable knowledge (Dewey, 1938; Kolb and Kolb, 2005;
Kolb, 1984, Michelson and McCord, 2011; NSSE, 1998). This theory provides a unique perspective on
learning and development with a strong emphasis on engaging students in the process of learning through
authentic experiences (Kolb and Kolb, 2005; Kolb, 1984; Michaelsen and McCord, 2011). Any
instructional method that intentionally engages students in the learning process is referred to as active
learning. Active learning is an important element of experiential learning which prevents students from
being passive recipients of what they are taught (Deely, 2010; Prince, 2004; Kolb and Kolb, 2005). To
enhance learning in higher education, the primary focus should be on engaging students through active
learning (Deely, 2010; Kolb and Kolb, 2005; Kolb, 1984; NSSE, 1998; Prince, 2004).
The idea of experiential learning was developed in the late nineteenth century, and gained
popularity through the work of David Kolb. It is a philosophy that has been applied to a variety of
experiences and fields offered at the university level (Deely, 2010: NSSE, 1998). When expanding on the
concept of experiential learning, Kolb draws heavily on the works of John Dewey and Kurt Lewis, who
39
focus on the role of experience in their theories of learning (Kolb, 1984; Kolb and Kolb, 2005; NSSE,
1998). Dewey’s (1938) “theory of experience” suggests that the process and goal of education are the
same. Lewin’s plan for conceptualizing learning through formal, testable theories supports the idea of
taking a scientific approach when developing learning practices (Kolb and Kolb, 2005).
Experiential education is intended to be a holistic, integrative approach to learning that combines
experience, perception, cognition, and behavior (Kolb and Kolb, 2005; Kolb, 1984; NSSE, 1998). Many
traditional and current educational practices are based on the idea that preexisting, fixed ideas are
transmitted to the learner. Experiential education stands in contrast to these philosophies by maintaining
that knowledge is individually and communally constructed through reflection (Kolb and Kolb, 2005;
Kolb, 1984; Michaelsen and McCord, 2011; NSSE, 1998). The emphasis on the process of learning as
opposed to the traditional outcome driven approaches distinguishes experiential learning from other
learning theories (Kolb, 1984).
In order to enhance a student’s educational experience, it is necessary to have a clear
understanding of the processes whereby knowledge is created (Kolb, 1984). Drawing on the experienceconstructivist theory, Kolb (1984) explains that learning is the process of creating knowledge through the
transaction and evolution of social and personal knowledge. Individuals approach each learning
experience with a variety of educational backgrounds, and everyone has pre-existing personal and social
knowledge. An educator’s job is to expose students to new ideas, and doing so may involve challenging
previously held assumptions and beliefs. Experiential education is believed to be one of the most
powerful methods for connecting new ideas and concepts to existing neurological networks (NSSE,
1998). Essentially, learning must be viewed as a continuous process; simply put: all learning is relearning
(Kolb, 1984; Kolb and Kolb, 2005; NSSE, 1998; Prince, 2004). Experiential learning theory focuses on a
unifying process of communication; this is in contrast to the dualisms of experience and knowledge often
found in education theories (NSSE, 1998). By better understanding the preconceived notions and beliefs
students have when entering an experience, improved learning can be facilitated by drawing out students’
beliefs and ideas so that they can be evaluated, tested, and integrated with more refined ideas (Kolb and
Kolb, 2005).
Experiential learning theory is widely accepted as a useful framework for “enhancing curriculum
development, student development, faculty and staff development, instructional design, and encouraging
life-long learning” (Kolb and Kolb, 2005, p. 209). Today, experiential education principles are being
applied to a variety of fields and experiences in higher education. This philosophy is often used to
facilitate critical thinking, problem solving, conflict resolution civic engagement, and team building
(Deely, 2010; Kolb and Kolb, 2005; NSSE, 1998). These experiences can occur within and outside of the
40
traditional classroom setting. The principles of experiential education are often used in the following
experiences common in higher education: service learning, internships, apprenticeships, laboratory
experiments, study abroad, and outdoor education programs (Kolb and Kolb, 2005; Kolb, 1984;
Michaelsen and McCord, 2011; NSSE 1998).
When learning is viewed as a holistic process, knowledge is formulated and tested through the
experiences of the learner (Kolb and Kolb, 2005; Kolb, 1984). Experiential learning theory provides a
framework for enhancing the process of the transformation of the “raw” experience into useable
knowledge (NSSE, 1998; Kolb and Kolb, 2005; Kolb, 1984). When intentionality is present in the design
of a course or experience and this intentionality is communicated to the students, a deeper understanding
of the important concepts and ideas develops (Prince, 2004; Kolb and Kolb, 2005; Kolb, 1984).
“Understanding the design of the activity and the intended learning outcomes promotes thoughtful
engagement on the part of the student “(Prince, 2004). Experiential education can be used to strengthen
student/teacher relationships by intentionally involving the student in the learning process. This is related
to the idea that being involved in the learning process enhances the overall quality of the experience for
the student. The basic elements of experiential learning theory, which are designed to enhance students’
experiences, are referred to as “principles of good practice” and can be implemented in all forms of
experiential education at the university level (NSSE, 1998).
The National Society for Experiential Education offers eight principles of good practice:
intention, preparation and planning, authenticity, reflection, orientation and training, monitoring and
continuous improvement, assessment and evaluation, and acknowledgment. When UNCW students are
exposed to applied learning opportunities framed by good practices such as these, their capacity for
learning should improve. Mechanisms for capturing the evidence of improved student learning depend
upon instructors utilizing techniques for critical reflection.
Critical Reflection
Applied learning pedagogies are designed to enhance student learning by increasing engagement
through the implementation of intentionally designed reflective techniques (Smith, 2011; Brooks, Harris
and Clayton, 2010; Ash and Clayton, 2009). Learning is maximized through critical reflection that
requires students to connect theory with practice. This connection aids students in strengthening and
assessing their understanding of a discipline, its practices, and the relationship between theory and
practice (Smith, 2011; Brooks et al., 2010; Clayton, 2010; Ash and Clayton, 2009). Furthermore, critical
reflection in applied learning settings builds capacity for critical thinking, inquiry, and thoughtful
41
expression. To fully appreciate the power of critical reflection, and its potential value in eTEAL, this
literature review focuses on critical reflection.
In order for critical reflection to develop higher-level thinking processes and improve the overall
learning experience of students, an educational experience must be intentionally designed with learning
goals and objectives in mind. It is through the intentional design and incorporation of reflection in
learning processes that reflection can be transformed into critical reflection (Brooks et al., 2010; Ash and
Clayton, 2009; Ward and McCotter, 2004). Students are often unaware of the learning objectives
associated with their educational experiences. If applied learning experiences are well-designed, students’
understanding of the learning process and its intended outcomes is improved through critical reflection
(Brooks et al., 2010; Ash and Clayton, 2009; Ward and McCotter, 2004). Learning does not simply occur
by exposure or experience alone; it requires critical reflection of the learning process and its intended
outcomes (Smith, 2011; Brooks et al., 2010). Students may have opportunities to engage in learning
experiences outside of the traditional classroom setting; however, they are rarely involved or aware of the
intended learning objectives associated with these experiences. By incorporating critical reflection
techniques into the design of the applied learning experiences, students are given the opportunity to
actively and consciously participate in the learning process (Brooks et al., 2010). “Critical reflection
differs from traditional reflection practices by prompting students to make connections between theory
and practice. It is through this process that students develop higher-order reasoning and critical thinking
skills” (Brooks et al., 2010; Ash and Clayton, 2004; Ward and McCotter, 2004).
The effectiveness of applied learning pedagogies depends on the ability of educators to
continuously improve upon their methodology and course design throughout the learning experience (Ash
and Clayton, 2009). Important to this process is the ability of these facilitators of student learning to
recognize the strengths and weaknesses of their students, as well as their own. Ash and Clayton (2009)
define “reflective practitioners as individuals who improve their own thinking and action, in an attempt to
generate, deepen and document student learning through the same type of critical reflection they expect
from their students.” Educators are actively involved in the learning process, both within the classroom
and in experiences that take students outside of the traditional classroom setting, where they act as
mentors or guides (Smith, 2011; Brooks et al., 2010). Teaching critical reflection supports the wellestablished argument that pedagogical approaches include opportunities for students to assimilate
previously held beliefs with current experiences in a way that would allow them to apply their knowledge
or experiences in the future (Smith, 2011; Ash and Clayton, 2004). There is a growing consensus among
educators that critical reflection is not only valuable for student learners, but also as a means of
42
facilitating professional and personal development for faculty in higher education (Smith, 2011; Ash and
Clayton, 2009).
Scholars of critical reflection and curriculum development draw upon the works of Dewey
(1938), Bloom (1956), and Kolb (1984) to explain the hierarchical structuring of learning processes and
objectives in higher education. Using Bloom’s Taxonomy of Educational Objectives to guide the design
and development of critical reflection enables teachers and instructional designers to target
developmentally appropriate learning objectives (Smith, 2011; Brooks et al., 2010; Ash and Clayton,
2009; Ward and McCotter, 2004). Kolb’s (1984) well-known “cycle of learning” explains the learning
process beyond receipt of knowledge, through the notion that reflection is part of learning process, not
simply the outcome of an experience. Critical reflection must be understood as both a summative and
formative process, a way to assess and cultivate learning by transforming learning goals into assessable
learning objectives, which drives the learning process (Smith, 2011; Brooks et al., 2010; Ash and
Clayton, 2009). Building on previous learning theories to develop an intentional instructional design
process, which includes critical reflection, allows educators to communicate effectively to their students
and colleagues the rationales and goals of applied learning pedagogies (Brooks et al., 2010; Ash and
Clayton, 2009).
Designing critical reflection in a way that truly enhances student learning requires beginning with
the end in mind. Student learning facilitators must begin by identifying desired learning outcomes. These
outcomes are then transformed into accessible learning objectives using Bloom’s taxonomy of
educational objectives (Ash and Clayton, 2009). The designs of teaching, learning, and assessment
strategies should be intentionally developed in accordance with learning goals and objectives (Brooks et
al., 2010; Ash and Clayton, 2009; Bloom, 1956). By developing a broad and flexible structure to express
and organize desired learning outcomes, educators communicate more effectively the expectations and
objectives of an applied learning experience (Brooks et al., 2010; Ash and Clayton, 2009). Ash and
Clayton (2004) developed the DEAL model for critical reflection, which is the product of a multi-year
teaching and learning project which identified three sequential steps for incorporating effective critical
reflection in higher education. The first step involves the Description of experiences, followed by the
Examination of those experiences in relation to specific learning objectives, and the final step requires the
Articulation of Learning from student participants (Brooks et al., 2010; Ash and Clayton, 2009).
Critical reflection enhances applied learning by generating, deepening, and documenting learning
in higher education (Brooks et al., 2004; Ash and Clayton, 2009). Students’ overall learning experiences
are enhanced through intentional instructional designs when critical reflection techniques are developed
by using well-established learning theories to guide the design process. Intentional instructional designs
43
are characteristic of applied learning pedagogies and promote the production of artifacts that document
learning for grading or research purposes (Ash and Clayton, 2009).
Bloom’s Taxonomy
Ash and Clayton (2009) specifically encourage using Bloom’s Taxonomy as a conceptual scheme
for instructors to utilize for articulating intentions and purpose for learning. Bloom’s Taxonomy was
developed as a framework for the classification of goals within education, creating a shared language for
academic assessment, research, and curriculum development (Hanna, 2007; Handy and Basile, 2005;
Krathwohl, 2002). By standardizing the language used, assessment measures, and curriculum
development processes, the aim is to improve the exchange of ideas and materials among educators
(Bloom et al., 1956; Handy and Basile, 2005; Hanna, 2007). It provides a framework for categorizing the
objectives, expectations, and intentions of the learning experience and has been utilized to enhance course
design, test construction, the development of rubrics and assessment measures (Betts, 2008; Bloom et al.,
1956; Halawi and McCarthy, 2009; Handy and Basile, 2005; Hanna, 2007; Krathwohl, 2002; Lipscomb,
2001).
Bloom’s Taxonomy groups behaviors into three categories: cognitive, affective, and psychomotor
(Bloom et al., 1956; Halawi and McCarthy, 2009; Lipscomb, 2001) and is formatted into a hierarchy with
the categories of mental processes arranged from simple to complex, and from concrete to abstract
(Halawi and McCarthy, 2009; Handy and Basile, 2005; Krathwohl, 2002; Lipscomb, 2001). The
hierarchy is composed of six levels: knowledge, comprehension, application, analysis, synthesis, and
evaluation (Bloom et al., 1956; Handy and Basile, 2005; Lipscomb, 2001). It is essential for learners to
master the lower levels before continuing to higher levels of learning (Handy and Basile, 2005). The first
level, knowledge, includes behaviors that focus on remembering by recognition or recall. Comprehension
is the next level, and the emphasis here is on the learner being able to understand and use the concepts to
which she or he is exposed. The third level, application, refers to the learner’s ability to extract relevant
concepts from the material presented. Analysis, the fourth level of Bloom’s hierarchy, implies that the
learner will be able to break down the material and recognize relationships in the organization of ideas or
concepts presented. The next level, synthesis, requires the learner to combine concepts and ideas which
have been presented singularly or in a disconnected format. The final level, evaluation, includes making
judgments and interpretations about the material presented and its value (Bloom et al., 1956; Handy and
Basile, 2005).
Since its publication in 1956, Bloom’s Taxonomy has been revised and modified to accommodate
a variety of academic fields and experiences (Betts, 2008; Krathwohl, 2002). The levels of the new
44
taxonomy are: remember, understand, apply, analyze, evaluate, and create (Anderson and Krathwohl,
2001; Handy and Basile, 2005; Krathwohl, 2002). Each level of the revised taxonomy contains a verb and
a noun to provide a more comprehensive description of the objective. For example, in the first level of the
hierarchy, remembering describes the process or behavior, while the sub-objective “recalling facts”
describes the knowledge or expected outcome (Anderson and Krathwohl, 2001; Handy and Basil, 2005).
Bloom’s Taxonomy of Educational Objectives has been applied to a variety of disciplines and academic
fields. The framework for the taxonomy is considered an important tool when designing, developing, and
assessing learning objectives (Hanna, 2007; Handy and Basile, 2005; Krathwohl, 2002). The Taxonomy
has helped to establish a common language among educators, which enhances higher education by
facilitating interdisciplinary research and standardizing learning objectives, regardless of the discipline.
By providing guidelines for assessment and curriculum design, educators are able to better communicate
learning goals and expectations. The ability to articulate the intended outcomes of a course or learning
experience helps to increase student engagement which then can lead to higher quality experiences
(Anderson and Krathwohl, 2001; Betts, 2008; Halawi and McCarthy, 2009; Handy and Basile, 2005;
Krathwohl, 2002).
Figure 16. Bloom’s Taxonomy.
45
Timeline and Actions to Be Implemented
With one year of experience conducting the preliminary implementation, UNCW is wellpositioned for a successful and smooth implementation and competition of eTEAL within the required
five year timeline, once approved. The timeline intentionally extends past the due date for the five year
impact report; this extension reflects a commitment to program sustainability and long term integration of
the QEP into the UNCW culture.
The University appointed an Interim Associate Vice Chancellor and Dean of Undergraduate
Studies whose position began January 1, 2013. That individual will establish the process for identifying
the new Director of University Studies and QEP. With those two positions filled, the current QEP Topic
Development Coordinator and Task Force will work closely to prepare the campus for a seamless
transition.
The annual Applied Learning Summer Institutes are scheduled to begin in 2013 and planning will
begin in the spring semester. The Applied Learning and Teaching Community has been established and
enjoys a great deal of support across campus. Identifying the Fellows who will facilitate that community
will be in the hands of the eTEAL Advisory Board and the Director of University Studies and Quality
Enhancement Plan. Fellows will be identified prior to the start of each semester, and terms of service will
be negotiated with their direct supervisors on a case by case basis.
The eTEAL-Supported Pedagogy Initiatives will be available to faculty during each academic
year. A Request For Proposals (RFP) will be distributed the semester prior to the initiation of support to
allow ample time for review by a committee appointed by the QEP Director, and (when necessary)
revisions to worthy proposals. As eTEAL continues to develop, the RFP will be revised to accommodate
emerging successful practices and innovative pedagogies that continuously enhance applied learning and
positively impact student learning.
Below is the timeline for implementation steps over the course of the five-year project.
46
eTEAL Activities
year 0
Summer Fall
2012 2012
year 1
Spring Summer Fall
2013 2013 2013
year2
Spring Summer Fall
2014 2014 2014
year 3 year 4
Spring Summer Fall
Spring Summer Fall
2015 2015 2015 2016 2016 2016
year 5
Spring Summer Fall
2017 2017 2017
year 6
Spring Summer Fall
2018 2018 2018
conclusion
Spring Summer Fall
2019 2019 2019
5 Year Impact Report for SACS
2013 Summer Institute
Planning & secure speakers
Advertise to campus
Workshop
2014 Summer Institute
Planning & secure speakers
Advertise to campus
Workshop
2015 Summer Institute
Planning & secure speakers
Advertise to campus
Workshop
2016 Summer Institute
Planning & secure speakers
Advertise to campus
Workshop
2017 Summer Institute
Planning & secure speakers
Advertise to campus
Workshop
2018 Summer Institute
Planning & secure speakers
Advertise to campus
Workshop
2012‐13 ALTC & Fellows
2013‐14 ALTC & Fellows
2014‐15 ALTC & Fellows
2015‐16 ALTC & Fellows
2016‐17 ALTC & Fellows
2017‐18 ALTC & Fellows
2018‐19 ALTC & Fellows
2013‐14 eTeal Supported Initiatives
2014‐15 eTeal Supported Initiatives
2015‐16 eTeal Supported Initiatives
2016‐17 eTeal Supported Initiatives
2017‐18 eTeal Supported Initiatives
2018‐19 eTeal Supported Initiatives
eTEAL Showcases
Assessment
Figure 17. QEP Timeline
47
Organizational Structure
Smooth and successful QEP implementation as well as program sustainability hinge on a
carefully conceived organizational structure. Highlights of the organizational structure include the QEP’s
close coordination with University Studies; allocation of dedicated personnel to support the project; and
the appointment of an advisory board to ensure shared governance. Coordinating the QEP with the
University Studies curriculum demonstrates and affirms its pivotal role across campus and ensures broad
integration of enhanced pedagogy for applied learning experiences developed through the eTEAL
program.
48
49
Figure 18. QEP Organizational Chart
The program will be administered by the Director of University Studies and QEP and s/he will
report to the Associate Vice Chancellor and Dean of Undergraduate Studies. Overseeing the University
Studies curriculum and the eTEAL program as a combined effort ensures the conceptual clarity of the
QEP and its relationship to the general education program.
The Director will serve in a 12-month administrative position with an associated faculty
appointment at the level of tenured Professor or Associate Professor, responsible for teaching assignments
within University Studies and/or eTEAL related courses. Responsibilities specific to administering
eTEAL include the following:









Maintain accurate records for all eTEAL events, including the annual showcase
Facilitate maintenance of a resource library for successful practices in applied learning
Chair the eTEAL Advisory Board to ensure effective program development and operation
Announce and administer peer-reviews for eTEAL-Supported Initiatives
Coordinate the annual Applied Learning Summer Institute
Compile records for program assessment
Maintain open communication across campus about eTEAL related activities and
opportunities
Prepare annual report on eTEAL and its impact at UNCW
Supervise the Social Research Assistant who will assist in various aspects of eTEAL related
duties
In addition, UNCW will hire a Social Research Assistant who will work closely with the Director
of University Studies and QEP. The Social Research Assistant will hold at least a master’s degree and
have specific experience with assessment of student learning outcomes and program assessment, website
design and maintenance, and excellent organizational skills.
To ensure shared governance, faculty expertise and input, as well as open communication and
implementation support, UNCW will appoint an eTEAL Advisory Board to advise the Director and assist
with specific responsibilities. The Advisory Board will include 11 members:

the Director (non-voting),

General Education Assessment Director
50

One faculty member from each: College of Arts and Sciences, College of Health and
Human Services, Watson College of Education, and Cameron School of Business.

Student Affairs

Business Affairs

Staff Senate

Two students: one undergraduate and one graduate student.
The Advisory Board will be responsible for assisting in the daily activities of eTEAL, providing support
and advising the Director, and meeting at the beginning of each academic semester. The Advisory Board
will maintain three standing committees that will meet as needed: the Resource Committee (budget
management, announcements, and review of eTEAL-Supported Initiatives proposals, etc.); the
Assessment Committee (assessment of student learning outcomes and program outcomes); and Pedagogy
Committee (assist with organizing the Applied Learning Summer Institute, represent eTEAL in the
Applied Learning and Teaching Community events, and select the ALTC Fellows annually). These
committees will meet regularly to achieve the goals of the program.
The Applied Learning Summer Institute is intended to facilitate participants’ understanding of
high-impact practices in applied learning and critical reflection. Featuring national experts (including
UNCW-based experts), this annual event will educate participants on cutting-edge practices and ideas,
and create for them a network of peers dedicated to the delivery of enhanced applied learning to UNCW
students.
The ALTC is intended to continue the enthusiasm for these high-impact practices throughout the
academic year by providing regular opportunities for interested faculty and staff to discuss their
experiences. The ALTC will allow for broader participation across the UNCW community.
The eTEAL-Supported Pedagogy Initiatives will materially invest in the enhanced applied
learning activities that instructors deliver. Critical reflection practices will be used, and a final critical
reflection product will capture evidence of student learning for assessment purposes. The initial
organizational structure and implementation plan may be revised as the project develops over time.
Resources
UNCW support for eTEAL is evident in the allocation of human and financial resources to
implement and sustain the program. With three years of planning and testing, the University has identified
clear goals, key personnel, and corresponding financial resources to support transformative education
51
through applied learning. UNCW views its instructional faculty (instructors) as a most valued resource for
delivering this plan. Instructors who participate earn specific recognition as Applied Learning Scholars
and Applied Learning Fellows. UNCW’s promotion and tenure procedures include consideration of
course development, student mentoring, curriculum development, and faculty development, all of which
provide mechanisms whereby participation in eTEAL can be recognized. The plan has been endorsed by
the University Studies Advisory Committee, Faculty Senate, Student Government Association, Graduate
Student Association, Staff Senate, and the Division of Student Affairs.
Detailed Budget to Support eTEAL
The proposed administration and programming budget to fund eTEAL is designed to be
institutionally efficient. Investment in the professional development of instructors at UNCW for the
purpose of enhancing student learning in applied learning venues is a core component of the QEP and
budget. Salary support for the Director of University Studies and QEP reflects an “in-kind” commitment
for UNCW. The remainder of the budget reflects new resources allocated to deliver and sustain eTEAL.
Through the 2012-13 campus initiated tuition increase process, $320,000/year of permanent tuition
revenues have been allocated to fund this program. The budget is presented below, followed by details of
personnel and non-personnel items therein. The section concludes with consideration of costs associated
with assessment and with an overview of in-kind contributions.
52
eTEAL Administration and Programming Budget
Y1 (2013‐2014) Y2 (2014‐2015) Y3 (2015‐2016) Y4 (2016‐2017) Y5 (2017‐2018) Y6 (2018‐2019)
Personnel
Director University Studies and QEP ($90,000 +30% benefits): 50% time devoted to QEP (in‐kind annual UNCW contribution )
QEP Social Research Asst ($45,000 + benefits) (starts January 2014): full time
Applied Learning Fellows: ($3500/pp/semester)x4
Graduate Assistants ($12,000/GA)
Applied Learning Summer Institute (outside contractors and UNCW‐based experts)
eTEAL Supported Initiatives, stipends of $3500/course (30/yr)
eTEAL mentors ($500/mentor)
Non‐Personnel (particpation supports)
Travel (QEP staff, fac and/or student)
Miscellaneous Support for A.L.
Supplies, Promotions & Marketing
Equipment (computers)
Office Furniture
Telecom
Library & Learning Resources
$ 117,000 $ 117,000 $ 117,000 $ 117,000 $ 117,000 $ 117,000
$ 29,250 $ 58,500 $ 58,500 $ 58,500 $ 58,500 $ 58,500
$ 14,000 $ 14,000 $ 14,000 $ 14,000 $ 14,000 $ 14,000
$ 24,000 $ 36,000 $ 48,000 $ 48,000 $ 48,000 $ 48,000
$ 7,000 $ 7,000 $ 7,000 $ 7,000 $ 7,000 $ 7,000
$ 105,000 $ 105,000 $ 105,000 $ 105,000 $ 105,000 $ 105,000
$ 1,500 $ 2,000 $ 3,000 $ 3,500 $ 4,000 $ 4,500
$ 10,000
$ 5,000
$ 20,000
$ 5,000
$ 6,000
$ 900
$ 2,000
$ 15,000
$ 5,000
$ 20,000
$ 15,000
$ ‐
$ 900
$ 2,500
$ 20,000
$ 5,000
$ 20,000
$ 15,000
$ ‐
$ 900
$ 3,000
$ 20,000
$ 5,000
$ 20,000
$ 15,000
$ ‐
$ 900
$ 3,500
$ 20,000
$ 5,000
$ 20,000
$ 15,000
$ ‐
$ 900
$ 3,500
$ 20,000
$ 5,000
$ 10,000
$ ‐
$ ‐
$ 900
$ 1,500
Assessment
Faculty/Staff Scorers
$ 5,000 $ 7,500 $ 7,500 $ 7,500 $ 7,500 $ 7,500
Technology (instruments and software, $ 12,500 $ 12,500 $ 12,500 $ 12,500 $ 12,500 $ 12,500
Trust
Catering for events (Summer Institute, workshops, and annual showcase)
Yearly Total
$ 4,000 $ 4,000 $ 4,000 $ 4,000 $ 4,000 $ 4,000
$ 368,150 $ 421,900 $ 440,400 $ 441,400 $ 441,900 $ 415,400
Grand total
$ 2,529,150
Table 8. eTEAL Budget
Details about personnel items:
1. Director of University Studies and QEP – described earlier.
2. Social Research Assistant–described earlier.
3. Applied Learning Fellows – The project requires that two Applied Learning Fellows be appointed
each semester to facilitate the Applied Learning and Teaching Communities. They will be
selected by the eTEAL Advisory Board Pedagogy Committee based on their expertise with
53
applied learning. They will earn either $3500 in salary stipend, or be provided a one course
buyout to be paid for by the eTEAL budget during the semester in which they serve as Fellows.
4. Graduate Assistants – To implement eTEAL and work with faculty and staff across the campus,
as well as to improve opportunities for developing applied learning teaching skills for graduate
students, eTEAL will require two graduate assistants in the first year, three the following year,
and four in the remaining years of the project. The graduate assistants will be available to the
Director, her/his Social Research Assistant, as well as to the faculty/staff as needed to assist with
collecting information and organizing learning materials for the eTEAL experiences. They will
assist with record keeping for the Applied Learning Fellows, work with undergraduate students,
and coordinate with the eTEAL Office staff.
5. Applied Learning Summer Institute -- The main vehicle for improving applied learning at UNCW
is the Summer Institute which will provide professional development for faculty and staff
instructors. This annual Institute will provide participants with information and skills to
implement applied learning in their classes, consistent with the design outlined previously. The
Summer Institute will be led by nationally known applied learning experts and/or UNCW-based
experts.
6. eTEAL Supported Pedagogy Initiatives – A vital element in our plan is to support instructors in
their efforts to enhance applied learning pedagogy, based on successful practices discussed during
the Summer Institute and in the ALTC, UNCW will solicit and fund qualifying proposals (see
RFP in Appendix for more detail).
7. eTEAL Mentors -- Participating faculty/staff will be invited to serve as mentors for subsequent
cohorts of faculty/staff participating in the training. Should they choose to serve as mentors, they
will be compensated with a $500 salary stipend during one mentoring year and may serve in this
capacity multiple times. Mentors must have attended the Applied Learning Summer Institute, be
regular participants in the ALTC, and have been awarded an eTEAL-Supported Pedagogy
Initiative. Having mentors is crucial to the long-term sustainability of the eTEAL project.
Maintaining regular contact with colleagues across campus interested in applied learning
pedagogies will enrich the teaching and learning experiences for faculty, and thus have a positive
impact on the instructional experiences for students. Furthermore, applied learning mentoring will
make important contributions to solidifying the powerful learning experience UNCW promotes as
part of its institutional identity and legacy.
Details about non-personnel items:
54
1. Travel -- The Director will require professional development opportunities of her/his own, so
travel to the annual SACSCOC conferences and summer institutes will be essential. Other
potential professional association meetings that specialize in applied learning and assessment will
be important for the Director to attend. Also, eTEAL related student learning may require travel
(to internship sites, practicum or clinical sites, field research, and conference trips to deliver
professional presentations) for students and instructors. eTEAL funds student travel so long as
it’s associated with their applied learning experiences, and includes faculty/staff mentors when
appropriate.
2. Supplies, Promotions, and Materials -- Applied learning experiences often involve necessary
equipment and supplies; the eTEAL budget should fund the necessary ingredients for successful
learning experiences. Furthermore, an eTEAL Showcase will highlight the work of faculty/staff
and students who pursue applied learning opportunities. The campus will be provided with
promotional materials and information to advertise eTEAL and promote participation and
participating students and faculty will be recognized.
3. Computer Technology (administrative and instructional) -- In the first year, the eTEAL office
will need to be equipped with computer and other technology support related to student learning
to ensure that faculty/staff and students involved in eTEAL will have access to appropriate
technology.
4. Library and Learning Resources: Cost projections above assume that 1) the costs will include
print and nonprint sources; 2) eContent will increase throughout the project period; and 3)
inflation will continue at rates comparable to those of recent years.
UNCW In-Kind Support for Applied Learning
There is in-kind support for a wide variety of applied learning activities from existing resources
on campus, including partnerships with several established units. Leveraging the expertise and
opportunities in these units will support eTEAL in numerous ways. These units support applied learning
in myriad ways including specific aspects of the QEP.
55
UNCW In-Kind Support for eTEAL
UNCW Unit
eTEAL-related Purpose
Budget
Center for the Support of Undergraduate
Research and Fellowships (CSURF)
Includes: Undergraduate Research
Involvement and Explorations: The
Journal of Undergraduate Research and
Creative Activities for the State of North
Carolina
To assist in providing expertise and opportunities for
undergraduate research involvement as applied learning
experiences.
$187,000
Honors College
To assist in providing expertise in undergraduate honors
instruction and theses projects as applied learning
experiences.
$375,000
Center for Teaching Excellence and
Faculty Leadership (CTE)
To assist in supporting professional development activities
for instructors related to applied learning pedagogy.
$251,000
Office of International Programs (OIP)
To assist in providing applied learning experiences in the
form of study abroad opportunities for instructors and
students.
$1,961,000
Center for Leadership Education and
Service (CLES)
To assist in providing community-based leadership and
applied learning experiences, and to maintain records of
student involvement.
$260,858
Total UNCW In-Kind Support for
eTEAL
All units providing support for applied learning and
professional development.
$2,796,858
Table 9. In-Kind Support for eTEAL
Assessment
The UNCW QEP, eTEAL, emerges from UNCW’s continued commitment to applied learning
and the goals of improving student learning and providing resources and support to faculty and staff. The
three eTEAL goals and their underlying outcomes provide the framework for the assessment process. The
eTEAL assessment methodology captures direct evidence of student learning and program effectiveness.
Both student learning and program outcomes are the foci of assessment efforts. Foundational
principles and activities in support of each of these efforts are described below. The section concludes
with discussion of an assessment timeline and use of assessment findings. An assessment matrix is
included in the appendices.
56
Assessment of Student Learning Outcomes
The primary goal of the QEP is to positively impact student learning within applied learning
experiences.
Goal 1: To improve student learning in applied learning experiences.
The Student Learning Outcomes (SLOs) for eTEAL experiences are the same as the UNCW student
learning outcomes for all applied learning experiences (all experiences in the Exploration Beyond the
Classroom [EBC] component of University Studies).
SLO 1. Students will articulate their expectations for, and the purpose and/or the goals of the experience
in terms of their personal educational development. [Thoughtful Expression]
SLO2. Students will synthesize knowledge drawn from their coursework to address the
issues/challenges/questions involved in the experience. [Critical Thinking, Inquiry, Thoughtful
Expression]
SLO 3. Students will communicate the impact or significance on their personal educational development
and on others in the profession or in the field at the conclusion of the experience. [Critical Thinking,
Thoughtful Expression]
It is expected that most eTEAL experiences will be already approved for Explorations Beyond
the Classroom (EBC) or will be submitted for approval, although there may be a small number of eTEAL
experiences that fall outside of EBC. The major difference between eTEAL experiences and other EBC
experiences is that eTEAL instructors will be implementing specific good practices introduced in the
various eTEAL venues, whereas non-eTEAL EBC instructors are not expected to do so. Furthermore,
eTEAL instructors will be capturing systematic evidence of student learning through critical reflection
practices in their applied learning experiences. The methods of assessing student learning within eTEAL
experiences will be the same as those used for general education assessment. Doing so, whether or not
these are QEP sponsored and supported courses, will assure equal expectations and treatment during the
scoring process. In addition, comparisons can be made between eTEAL and non-eTEAL experiences to
determine the effectiveness of specific practices introduced through professional development. To do this,
information collected from assessment of Goals 2 and 3 will be cross-referenced with student learning
outcomes results.
57
Student work projects that will be sampled from eTEAL experiences include student critical
reflections, reports, projects and presentations in both written and oral form. These products will be
assessed on the UNCW Learning Goals linked to the three SLOs above, thoughtful expression, critical
thinking, and inquiry, as well as on a critical reflection rubric that assesses the three SLOs specifically.
The rubrics used for critical thinking, inquiry, written and oral communication are the Association of
American Colleges and Universities (AAC&U) Valid Assessment of Learning in Undergraduate
Education (VALUE) rubrics. The dimensions of the critical reflection rubric are taken from VALUE
rubrics, which are part of the AAC&U Liberal Education and America’s Promise (LEAP) initiative and
were developed by over 100 faculty and other university professionals. Each rubric contains the common
dimensions and most broadly shared characteristics of quality for each dimension. This common
consensus assures that they are valid measures of performance. Reliability is supported through scorer
norming workshops and through scoring in groups during the scoring events. Reliability measures for
both agreement and consistency will be calculated on multiply-scored work products, and the results will
inform future scorer norming activities. All rubrics are presented in Appendix E. Benchmark levels for
student performance on the rubric will be based on the course level. For example, the benchmark for
student work from 200-level courses would be a 2, for student work from 300-level courses, a 3, etc.
While sample selection for EBC courses will be based on the General Education Assessment
schedule and sampling matrix, student work will be sampled from all eTEAL-Supported Initiative
courses (the size of the sample from any eTEAL course will depend on the number of courses offered in
a semester). In addition, student work from courses that are influenced by the Summer Institute, ALTC
activities, and other eTEAL workshops will also be sampled in greater proportion than non-eTEAL EBC
courses. Scoring of student work from all sources, eTEAL and non-eTEAL applied learning experiences,
will take place together, and the work products will have no information on them identifying eTEAL
courses.
A mapping of student learning assessment measures to outcomes, performance benchmarks and
assessment schedules are listed in Table 10 eTEAL Assessment Matrix.
Assessment of Program Outcomes
An evaluation of program outcomes is vital to any effort to improve student learning. The
eTEAL process of support for faculty and staff in the area of applied learning will require careful
tracking of activities and their effects on applied learning experiences. Program goals and outcomes
include focusing on providing information and resources on good practices in applied learning and
58
providing support for implementation of these practices, which ultimately support the goal of student
learning.
Goal 2: To provide faculty and staff with information and resources about high-impact practices in
applied learning pedagogy as appropriate to their discipline and responsibilities.
Outcome 1. UNCW will sponsor multiple opportunities for formal and informal faculty and staff
development to enhance applied learning pedagogy. Activities will include the eTEAL Summer Institute,
Applied Learning Workshops, and Applied Learning and Teaching Community gatherings.
Outcome 2. UNCW will develop a library of information about high-impact applied learning practices.
Activities will include developing and maintaining a web-based repository of high-impact applied
learning practices, including critical reflection prompts.
Goal 3: To promote the incorporation and implementation of high-impact practices of applied
learning throughout UNCW.
Outcome 1. UNCW will sponsor faculty and staff efforts to enhance academic achievement in applied
learning through eTEAL Supported Initiatives. Activities include periodic calls for proposals, scoring of
proposals, selection and funding of proposals, support activities throughout the duration of the proposed
experience, and follow up activities, including participants sharing results.
Outcome 2. UNCW will host teaching and learning community events where faculty and staff can share
implementation activities and results. Activities will include Applied Learning and Teaching Community
(ALTC) meetings and other events.
Outcome 3. UNCW will support and follow up with Applied Learning Institutes and Workshop
participants. Activities include contacts by the QEP coordinator with all Summer Institute and other
sponsored workshop participants.
Program assessment activities will consist of the following:

Quantifying activities and participation

Quantifying online resources and website activity

Quantifying proposals submitted for eTEAL-Supported Initiatives, their scores, number
supported, and dollar amount of support

Participant satisfaction surveys
59

Participant follow-up surveys and interviews on incorporation of practices

Website user surveys on usefulness of and incorporation of practices

Participant critical reflections

Analysis of the effects of specific instructional practices on student learning outcomes
The first level of program assessment will be maintaining records of professional development
activities sponsored. The Director of University Studies & QEP will ensure the following duties are
completed:

maintain records of the number and type of eTEAL sponsored institutes, workshops, and ALTC
activities, including lists of participants at these events.

monitor the growth in web library of applied learning resources through periodic counts.

maintain records on all eTEAL -Supported Initiative proposal submissions, proposal scores,
awards made, and proposals not selected.

assess proposals using a rubric based on the requirements for selection.

maintain records of the budget for all activities, including the student learning outcomes
assessment activities described in section 5.1.
The next level of assessment will involve determining participant satisfaction with professional
development activities (including the ALTC website) and collecting information on the implementation
trends of the methods and practices introduced, as well as barriers to implementation. The Director of
University Studies and QEP will work with the Director of General Education Assessment and others to
ensure the following duties are completed:

collect post-event participant surveys, and follow up with participants at the end of the year to
ascertain the efficacy of the methods and practices that were incorporated into instructors’ applied
learning experiences.

conduct follow up surveys with participants in eTEAL workshops and institutes at 12 month
intervals to determine the extent of implementation and the effectiveness of workshop and
institute material and ideas.

administer periodic follow up surveys after the end of the support period to determine the extent
of continued implementation of practices.

analyze information from director contacts, participant reflections, and participant presentations
to the ALTC to capture information on the process of implementing new ideas and pedagogy and
their perceived effects on student learning during the support period.
60

capture faculty and staff perceptions of usefulness of eTEAL activities, identify additional
faculty and staff development needs related to applied learning, and provide ideas for additional
mechanisms to support faculty in designing effective applied learning experiences.
The final level of assessment will be an analysis comparing the specific practices implemented, as
well as the instructors’ assessment of the level and success of the implementation to the SLO assessment
results. Qualitative methods will be used to code survey, interview, and instructor critical reflection
responses, and the categories determined by these measures will be used to disaggregate student learning
outcomes data from corresponding courses to determine patterns of performance on the SLOs and UNCW
Learning Goals. The expectations for this analysis are that it will highlight specific high-impact applied
learning practices. Results of this analysis will be used to inform subsequent professional development
activities and will be shared during eTEAL professional development activities.
Program assessment measures to outcomes, performance benchmarks and assessment schedules
are listed in Table 10, eTEAL Assessment Matrix.
Assessment Timeline
Assessment data for all student learning and program outcomes will be collected on an ongoing
basis throughout the academic year. Satisfaction surveys will be administered directly after institutes and
workshops and periodically with members of the ALTC. Student artifacts for student learning assessment
will be collected throughout the academic year. Once a year, in early spring, scorers will be solicited for a
scoring event. and workshops will be held about one week before the event, sometime in late spring.
Assessment results will be compiled into an assessment report each summer. The first of these assessment
processes began in spring 2011 with the call for proposals for the pilot QEP Supported Initiatives and the
assessment report containing the first pilot findings submitted in spring 2012.
Use of Assessment Findings
The findings gathered from the measures of assessment for all goals and outcomes will be used
by the QEP Director and Advisory Board to evaluate progress on QEP outcomes and make programmatic
decisions during the period of QEP implementation (formative purposes) and to ultimately determine the
success of the QEP (summative purposes). Annual reports prepared by the Director of University Studies
& QEP (with the help of the Director of General Education Assessment for student learning outcomes)
will be presented to the QEP Advisory Board for analysis of progress on each goal and outcome and
61
recommendations for actions for improvement. The reports will be shared with the Provost and other
UNCW constituents for further feedback on the process. These annual reports along with actions taken for
improvement will be used to complete the QEP impact report in 2018.
Assessment Matrix
The eTEAL assessment process is summarized in the following table.
62
Assessment Matrix
Outcome
Goal 1. Outcome 1. Students will
articulate their expectations, the
purpose, and/or the goals of the
experience in terms of their personal
educational development.
[Thoughtful Expression]
Goal 1. Outcome 2. Students will
synthesize knowledge drawn from
their coursework to address the
issues/challenges/questions involved
in the experience. [Critical Thinking,
Inquiry, Thoughtful Expression]
Assessment Methods
Responsible
Entities
Performance Benchmark
Schedule
Student critical reflections during their
applied learning experience (direct)
[Intention dimensions of the Critical
Reflection and Written Communication
Rubrics, Appendix X]
Benchmark scores will be based on
the level of the course (i.e., the
benchmark score for work from
200-level courses will be a 2 on all
rubrics and dimensions)
Student work products will be
collected during each
semester. Student work will
be scored annually in late
spring.
Other work products from the experience
such as initial application or discussion with
mentor, if applicable (direct) [Intention
dimensions of the Critical Reflection Rubric,
Appendix X]
Benchmark scores will be based on
the level of the course (i.e., the
benchmark score for work from
200-level courses will be a 2 on all
rubrics and dimensions)
QEP Director
and staff,
General
Education
Assessment
Director and
staff
Student critical reflections during their
applied learning experience (direct) [Critical
Reflection and Written Communication
Rubrics, Appendix X]
Benchmark scores will be based on
the level of the course (i.e., the
benchmark score for work from
200-level courses will be a 2 on all
rubrics and dimensions)
Student work products will be
collected during each
semester. Student work will
be scored annually in late
spring.
Other work products from the experience
such as project reports and presentations
(direct) [Critical Thinking, Inquiry, Written
Communication and Oral Communication
Rubrics, Appendix X]
Benchmark scores will be based on
the level of the course (i.e., the
benchmark score for work from
200-level courses will be a 2 on all
rubrics and dimensions)
QEP Director
and staff,
General
Education
Assessment
Director and
staff
63
Student work products will be
collected during each
semester. Student work will
be scored annually in late
spring.
QEP Director
and staff,
General
Education
Assessment
Director and
staff
One Summer Institute each year.
One workshop each semester.
Baseline data will be collected
during initial year and used to set
benchmark.
Descriptions and number of
events will be cataloged on a
continuous basis.
Participation will be counted
at each event.
QEP Director
and staff
Participant satisfaction surveys and follow up
surveys and interviews on implementation
Baseline data will be collected
during initial year and used to set
benchmark.
Satisfaction surveys will be
administered within two days
of each event.
Follow up surveys will be
administered once a year.
Number of articles/items in web-based
repository
Baseline data will be collected
during initial year and used to set
benchmark.
Number of articles will be
counted once a year.
Number of hits on repository pages
Baseline data will be collected
during initial year and used to set
benchmark.
Number of hits will be
recorded monthly.
Faculty and staff follow up surveys on the
use of the repository
Baseline data will be collected
during initial year and used to set
benchmark.
Follow up surveys will be
administered once a year.
Goal 1. Outcome 3. Students will
communicate the impact or
significance on their personal
educational development and on
others in the profession or in the field
at the conclusion of the experience.
[Critical Thinking, Thoughtful
Expression]
Student critical reflections during their
applied learning experience (direct) [Critical
Reflection and Written Communication
Rubrics, Appendix X]
Benchmark scores will be based on
the level of the course (i.e., the
benchmark score for work from
200-level courses will be a 2 on all
rubrics and dimensions)
Other work products from the experience
such as exit interviews, if applicable (direct)
[Critical Reflection, Critical Thinking,
Written Communication and Oral
Communication Rubrics, Appendix X]
Benchmark scores will be based on
the level of the course (i.e., the
benchmark score for work from
200-level courses will be a 2 on all
rubrics and dimensions)
Goal 2. Outcome 1. UNCW will
sponsor multiple opportunities for
formal and informal faculty and staff
development to enhance applied
learning pedagogy.
Number of events and number of participants
at events
Goal 2. Outcome 2. UNCW will
develop a library of information about
high-impact applied learning
practices. Activities will include a
web-based repository of high-impact
applied learning practices, including
critical reflection prompts.
QEP Director
and staff
64
Goal 3. Outcome 1. UNCW will
sponsor faculty and staff applied
learning implementation through
eTEAL Supported Initiatives.
Activities include annual call for
proposals, selection and funding of
proposals.
Goal 3. Outcome 2. UNCW will host
teaching and learning community
events where faculty and staff will
share implementation activities and
results. Activities will include
Applied Learning and Teaching
Community (ALTC) discussion
groups, reading groups.
Goal 3. Outcome 3. UNCW will
support and follow up with Applied
Learning Workshop participants.
Number of proposals submitted and scores
on these proposals and number of proposals
supported and dollar amount of support
Baseline data will be collected
during initial year and used to set
benchmark.
Collected after each call for
proposal ends
Instructor critical reflections at the end of the
supported applied learning experience
Baseline data will be collected
during initial year and used to set
benchmark.
Collected at the end of each
semester, analyzed using
qualitative techniques
Annual post-implementation follow up
surveys and interviews on continued
implementation
Baseline data will be collected
during 2012-2013 and used to set
benchmark.
Annually
Instructor data will be coded by categories of
practices and pedagogies implemented, and
student learning outcomes data will be
analyzed with respect to those categories
There are no a priori expectations
about which practices will be
determined to be most beneficial.
Annually
Number of events and number of
participants at events
Baseline data will be collected
during initial year and used to set
benchmark.
Events and participants will
be listed at the end of each
event.
Participant satisfaction surveys
Baseline data will be collected
during initial year and used to set
benchmark.
Surveys will be administered
at the end of workshops and at
the end of each semester for
ongoing activities.
Participant follow up surveys and interviews
on implementation
Baseline data will be collected
during initial year and used to set
benchmark.
Follow up surveys and
interviews will be
administered annually.
Notes on periodic contacts during
implementation of eTEAL sponsored
experiences
Baseline data will be collected
during initial year and used to set
benchmark.
On going
Follow up survey with Summer Institute and
other workshop participants
Baseline data will be collected
during initial year and used to set
benchmark.
Annually
QEP Director
and staff,
General
Education
Assessment
Director and
staff
QEP Director
and staff
QEP Director
and staff
Table 10. eTEAL Assessment Matrix
65
References
Anderson L. W. and Krathwohl D.R. (2001). A taxonomy for learning, teaching and assisting: A revision
of Bloom’s taxonomy of education objectives. New York: Longman.
Betts, S.C. (2008). Teaching and assessing basic concepts to advanced applications: Using Bloom’s
taxonomy to inform graduate course design. Academy of Educational Leadership Journal, 12(3),
99-106.
Ash, S.L. and Clayton, P.H. (2009). Generating, deepening, and documenting learning: The power of
critical reflection in applied learning. Journal of Applied Learning in Higher Education, 1, 25-48.
Ash, S. L. and Clayton, P.H. (2004). The articulated learning: An approach to guided reflection and
assessment. Innovative Higher Education, 29(2), 137-152.
Bloom, B. S., Englehart M.D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956). Taxonomy of
educational objectives, Handbook 1: Cognitive Domain. New York: Longman.
Brooks, E., Harris, C.R., and Clayton, P.H. (2010). Deepening applied learning: An enhanced case study
approach using critical reflection. Journal of Applied Learning in Higher Education, 2, 55-76.
Deely, S. J. (2010). Service-learning: Thinking outside the box. Active Learning in Higher Education,
11(1), 43-53.
Desimone, L.M. (2009). Improving impact studies of teacher professional development: Toward better
conceptualizations and measures. Educational Researcher, 38(3), 181-199.
Dewey, J. 1938. Experience and Education. New York: MacMillan Publishing Co.
Halawi, L.A. and McCarthy R.V. (2009). An evaluation of e-learning on the basis of Bloom’s taxonomy:
An exploratory study. Journal of Education for Business, 84 (6), 374-380.
Handy, S.A. and Basile, A. (2005). Improving accounting education using Bloom’s taxonomy of
educational objectives. Journal of Applied Research for Business Instruction, 3(3), 1-6.
Hanna, W. (2007). The new Bloom’s taxonomy: Implications for music education. Arts Education
Policy Review, 108(4), 7-16.
Henderson, C., Beach, A., and Finkelstein, N. (2011). Facilitating change in undergraduate STEM
instructional practices: An analytic review of the literature. Journal of Research in Science
Teaching, 48(8), 952-984.
Knefelkamp, L. (1989). Assessment as transformation. Speech given to the American Association for
Higher Education Fourth National Conference on Assessment in Higher Education. Atlanta, June
21-24, 1989.
Kolb, A.Y. and Kolb, D.A. (2005). Learning styles and learning spaces: Enhancing experiential learning
in higher education. Academy of Management Learning and Education, 4(2), 193-212.
Kolb D.A. (1984). Experiential learning: Experience as a source of learning and development.
Englewood Cliffs NJ: Prentice Hall.
66 Krathwohl D.R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4),
212-218.
Lipscomb, J. W. (2001). Is Bloom’s taxonomy better than intuitive judgment for classifying test
questions? Journal of Education, 106(1), 102-107.
Michaelsen, L.K. and McCord M. (2011). The integrative business experience: A practical approach for
learning by doing. Journal of Applied Learning in Higher Education, 3, 25-43.
NSSE Foundations Document Committee. (1998). Foundations of Experiential Education, December
1998. NSSE Quarterly, 23(3), 16-22.
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering
Education, 93 (3), 223-231.
Rutz, C., Condon, W., Iverson, E., Manduca, C., and Willet, G. (2012). Faculty professional development
and student learning: What is the relationship? Change the Magazine of Higher Learning, 44 (3),
40-47.
Smith E. (2011). Teaching critical reflection. Teaching in Higher Education, 16 (2), 211-223.
Ward, J.R. and McCotter, S.S. (2004). Reflection as a visible outcome for pre-service teachers. Teaching
and Teacher Education, 20, 243-257.
67 Appendices
Appendix A. QEP Topic Development Task Force Members
2010-2011 Members of the QEP Task Force

Ravija Badarinathi Professor, Department of Information Systems and Operations Management

Lisa Castellino Assistant Vice Chancellor, Director of Institutional Research and Assessment

Carrie Clements Professor, Department of Psychology and Director, Center for Teaching Excellence

Kimberly J. Cook Professor and Chair, Department of Sociology and Criminology, and QEP Coordinator

John Fischetti, Professor, Department of Educational Leadership

Jose Hernandez Associate Provost, Institutional Diversity and Inclusion

Leslie Hossfeld Associate Professor, Department of Sociology and Criminology

Jeanne Kemppainen Professor, College of Health and Applied Human Services (Nursing)

Leah Kraus Associate Vice Chancellor, Information Technology Systems Division

Patrick LeCompte Undergraduate Student, Finance and Criminology

Martin Posey University Accreditation Coordinator, and Professor of Biology and Marine Biology

Thom Rakes Career Services, Student Affairs

Jimmy Reeves Professor and Chair, Department of Chemistry

Colleen Reilly Associate Professor, Department of English

Kim Sawrey Associate Professor, Department of Psychology

Linda Siefert Director of Assessment, College of Arts and Sciences, and General Education Assessment Director

Brian Victor Associate Vice Chancellor, Division of Student Affairs

Sarah Watstein University Librarian, William Madison Randall Library

David Webster Associate Dean, College of Arts and Sciences and Professor, Department of Biology and Marine
Biology

Rick Whitfield Associate Vice Chancellor, Division of Business Affairs

Becky Warfield Graduate Student, Department of English
2011-2012 Members of the QEP Task Force





















Ravija Badarinathi Professor, Department of Information Systems and Operations Management
Lisa Castellino Assistant Vice Chancellor, Director of Institutional Research and Assessment
Kimberly J. Cook Professor and Chair, Department of Sociology and Criminology, and QEP Coordinator
Darwin Dennison Professor, School of Health and Applied Human Sciences (Health)
Jose Hernandez Associate Provost, Institutional Diversity and Inclusion
Jennifer Horan Assistant Professor, Department of Public and International Affairs
Jeanne Kemppainen Professor, College of Health and Applied Human Services (Nursing)
Leah Kraus Associate Vice Chancellor, Information Technology Systems Division
Patrick LeCompte Undergraduate Student, Finance and Criminology
Anne Pemberton Librarian, Randall Library
Martin Posey University Accreditation Coordinator, and Professor of Biology and Marine Biology
Melissa Rogan Graduate Student, Department of Sociology and Criminology
Kim Sawrey Associate Professor, Department of Psychology
Rebecca Stultz Academic Advisor, University College
Linda Siefert Director of Assessment, College of Arts and Sciences, and General Education Assessment Director
Stephanie Smith External Program Director, Division of Public Service
Brian Victor Associate Vice Chancellor, Division of Student Affairs
Sarah Watstein University Librarian, William Madison Randall Library
Adam Webb Undergraduate Student, Economics, Finance and English
David Webster Associate Dean, College of Arts and Sciences and Professor, Department of Biology and Marine
Biology
Rick Whitfield Associate Vice Chancellor, Division of Business Affairs
2012-2013 Members of the QEP Task Force

Ravija Badarinathi Professor, Department of Information Systems and Operations Management
68 



















Lisa Castellino Assistant Vice Chancellor, Director of Institutional Research and Assessment
Kimberly J. Cook Professor and Chair, Department of Sociology and Criminology, and QEP Coordinator
Darwin Dennison Professor, School of Health and Applied Human Sciences (Health)
Christina Foster Graduate Student, Department of Sociology and Criminology
Jose Hernandez Associate Provost, Institutional Diversity and Inclusion
Jennifer Horan Assistant Professor, Department of Public and International Affairs
Carol Highsmith Professor, College of Health and Applied Human Services (Nursing)
Leah Kraus Associate Vice Chancellor, Information Technology Systems Division
Patrick LeCompte Undergraduate Student, Finance and Criminology
Michael Mills Lecturer, Department of English and Honors College
Martin Posey University Accreditation Coordinator, and Professor of Biology and Marine Biology
Kim Sawrey Associate Professor, Department of Psychology
Rebecca Stultz Academic Advisor, University College
Linda Siefert Director of Assessment, College of Arts and Sciences, and General Education Assessment Director
Brian Victor Associate Vice Chancellor, Division of Student Affairs
Brad Walker Professor, Watson College of Education
Sarah Watstein University Librarian, William Madison Randall Library
Adam Webb Undergraduate Student, Business Administration and English
David Webster Associate Dean, College of Arts and Sciences and Professor, Department of Biology and Marine
Biology
Rick Whitfield Associate Vice Chancellor, Division of Business Affairs
69 Appendix B. QEP Topic Development Events
Event
Event
QEP Open Forums
QEP Kick-Off
Date
Date
September 27, 2011
November 1, 2010
# participants
#23participants
66
QEP Open Forums
November 10, 2010
31
QEP Open Forums
QEP Open Forums
September 28, 2011
November 15, 2010
21
19
QEP Student Forum
November 17, 2010
23
QEP Open Forums
QEP Open Forums
November 18, 2010
October 3, 2011
27
26
QEP Open Forums
November 30, 2010
36
Focus Group
January 19, 2011
13 QEP Ambassadors
QEP Explorations of the
Natural World Pilot Student
Focus
Focus Group
Group
December 8, 2011
5
January 20, 2011
15 QEP Ambassadors
December 8, 2011
4
January 24, 2011
15 QEP Ambassadors
December 9, 2011
5
January 25, 2011
17 QEP Ambassadors
QEP Improving the Southeast
Focus
Group
Student
Focus Group
December 9, 2011
January 28, 2011
3
14 QEP Ambassadors
QEP Expanding the
Focus
Group:Graduate
Deans Student
Professoriate
Focus Group
December 9, 2011
February 3, 2011
4
7
Faculty Focus Group
Focus Group: Department
Chairs
January 18, 2012
February 4, 2011
3
6
Faculty Focus Group
Town Hall Meeting
January 19, 2012
February 10, 2011
6
72
Applied Learning Summer
QEP
Pre-Proposal
Workshop
Institute:
Good Practices
in
Experiential Education
July 30, 2012
February 24, 2011
34
Not recorded
QEP
Pre-Proposal
Workshop
Applied
Learning Summer
Institute: Critical Reflection
February
25, 2011
July 31, 2012
Not
31 recorded
QEP
Finalist
Celebration;
Applied
Learning
Summer
Wine
andWrap-Up
Cheese Reception
Institute
and RFP
May
10,1,2011
August
2012
Not
28 recorded
QEP Standardized Evaluation
of Applied Learning Student
Focus
Focus Group
Group
QEP Experience Research
Student Focus Group
Focus Group
Distribution
Outcomes
Outcomes
Disseminated, discussed, and
captured feedback
eTeal idea,
Celebration
of QEPon:
planning
best practices
and and
assessment,
process;
feedback
survey data
important academic skills, and
collected.
desired professional
development
Facilitated
interest, collected
ideas
to enhance
applied
learning.
and
feedback
from forum
Disseminated, discussed, and
participants.
captured feedback
eTeal idea,
Facilitated
interest,on:
collected
best practices
assessment,
feedback
fromand
forum
participants.
important academic
skills, and
Facilitated
interest, collected
desired professional
feedback
from forumdevelopment
participants.
to enhance applied learning.
Facilitated interest, collected
Disseminated,
discussed,
and
feedback
from forum
participants.
captured feedback on: eTeal idea,
Facilitated interest, collected
best practices and assessment,
feedback from forum participants.
important academic skills, and
In-depth
discussion on
best
desired professional
development
learning
experiences,
and merits
to enhance
applied learning.
of multiple topic areas; captured
In-depth discussion of highest
feedback to analyze.
quality applied learning
In-depth
discussion
bestand
experience,
student on
gains,
learning
and merits of
ideas forexperiences
improvement.
multiple topic areas; captured
In-depth discussion of highest
feedback to analyze.
quality applied learning
In-depth
discussion
bestand
experience,
student on
gains,
learning
experiences
and
ideas for improvement. merits of
multiple topic areas; captured
In-depth discussion of highest
feedback to analyze.
quality applied learning
In-depth
discussion
merits
experience,
student on
gains,
andof
multiple
areas; captured
ideas fortopic
improvement.
feedback to analyze.
In-depth discussion of highest
In-depth
discussion
on best
quality applied
learning
learning
experiences
and merits
experience,
student gains,
and of
multiple
topic
areas;
captured
ideas for improvement.
feedback to analyze.
In-depth discussion of highest
In-depth
discussion
on best
quality applied
learning
learning
experiences
and merits
experience, student gains,
and of
multiple
areas; captured
ideas fortopic
improvement.
feedback to analyze.
In-depth discussion of highest
In-depth
discussion
on best
quality applied
learning
learning
experiences
and merits
experience,
student gains,
and of
multiple
areas; captured
ideas fortopic
improvement.
feedback to analyze.
In-depth discussion of highest
Announced
topic
area of “applied
quality applied
learning
learning”;
collected
more and
experience, student gains,
feedback
on the topic; released
ideas for improvement.
call for pre-proposals.
Discussed and explored good
Discussed
practices inrequirements
Experientialand
criteria
for facilitated
pre-proposals
to be
Education
by Dr.
successful.
Roseanna Ross.
Discussed
Discussed requirements
and explored and
good
criteria
forinpre-proposals
to be
practices
Critical Reflection
successful.
facilitated by Dr. Patti Clayton.
Announcement
and affirmation
Discussed best lessons
from of
proposal
began discussed
to
previous finalists;
days’ material,
develop
plans
for
pilot
projects
in
and disseminated RFP for
2011-2012.
eTEAL-Supported Initiatives for
Fall 2012.
70 Appendix C. QEP Topic Development Campus Outreach/Inclusion Efforts
Campus Constituency
Dates
Outcome
Faculty Senate
October 2010
Announced, outlined QEP
requirements, and invited
participation.
Update, invite feedback.
Announced topic area of Applied
Learning, distributed call for preproposals, invited participation and
feedback.
Updated, announced the Pilot
Program, invited feedback.
Report to senate on proposed
eTEAL program, focused on
improving applied learning,
discussion, and invited feedback.
Motion passed: Faculty Senate
endorsed a QEP centered on Applied
Learning with an emphasis on
instructor-centered development
opportunities; invited ideas and
feedback.
Updated, outlined revised eTEAL
plan, and invited feedback.
Faculty Senate unanimously
endorsed eTEAL final plan.
Announced, outlined QEP
requirements, and invited
participation.
Updated, outlined pilot program,
discussed, invited ideas and
feedback.
Updated, discussed, SGA endorsed
eTEAL unanimously.
Announced, outlined QEP
requirements, invited participation
Updated, outlined eTEAL, Staff
Senate unanimously endorsed
eTEAL.
Announced, outlined QEP
requirements, invited feedback and
ideas.
Updated, outlined Pilot program,
discussed, invited feedback.
Updated, described findings from
pilot program, outlined eTEAL,
invite feedback.
Updated, described eTEAL, invited
feedback; eTEAL was endorsed by
GSA unanimously.
January 2011
February 2011
October 2011
March 2012
April 2012
November 2012
December 2012
Student Government Association
October 2010
September 2011
September 2012
Staff Senate
October 2010
November 2012
Graduate Student Association
October 2010
September 2011
April 2012
September 2012
71 College of Arts and Sciences
August 2011
Updated, discussed, invited feedback
and ideas.
CAS Forum to discuss first draft of
plan, invited feedback and ideas for
revisions.
April 2012
Cameron School of Business
April 2012
College of Health and Human
Sciences
April 2012
University Chairs meeting
February 2011
CSB faculty meeting presented first
draft of plan, invited feedback and
ideas for revisions.
CHHS Dean and Directors meeting,
presented first draft of plan, invited
feedback and collected ideas for
revisions.
Updated, discussed, invited feedback
and ideas.
Updated, discussed, invited feedback
and ideas.
July 2012
Division of Student Affairs
November 2010
August 2011
September 2012
University Studies Advisory
Committee
Represented on QEP TF
April 2011
September 2012
October 2012
Academic Coordinating Council
Chancellors Expanded Cabinet
October 2012
April 2011
UNCW Board of Trustees
February 2011
72 Announced, outlined QEP
requirements, invited feedback and
participation.
Annual retreat; featured speaker on
Applied Learning and its benefits,
discussion, invited feedback and
participation.
Updated, outlined eTEAL proposal,
invited participation and feedback.
Constant flow of information and
ideas, invited feedback and ideas.
Consulted on points of connection
and potential synergy.
Discussed merging SLO for EBC.
Discussed QEP and USAC mutual
responsibilities and opportunities;
support from USAC to move
forward; USAC endorsed eTEAL.
Updated, invited feedback and ideas.
Updated, discussed, invited feedback
and ideas.
Updated, discussed, invited feedback
and ideas.
Appendix D. eTEAL: Expanding the Professoriate Pilot Activities
Date
Event Title
Friday, September 9, 2011
QEP Lunch: Expanding
the Professoriate
14
Friday, September 9, 2011
Thursday, September 22,
2011
QEP pilot project meeting
“Applied Learning:
Increasing Engagement
through Honors and DIS”
QEP Expanding the
Professoriate meeting
10
9
Thursday, September 29,
2011
Wednesday, October 19,
2011
# Participants
25
“Applied Learning within
the Classroom: Increasing
Engaged Learning in 1-3
Classes”
“Applied Learning Outside
of the Classroom:
Increasing Engaged
Learning in the
Community”
10
Wednesday, October 26,
2011
QEP pilot group meeting
20
Tuesday, November 29,
2011
Applied Learning faculty
meeting
20
Wednesday, March 21,
2012
“Adding Intention and
Reflection to Your
Applied Learning
Components”
Applied Learning Poster
Exhibit
18
Thursday, October 20,
2011
Summer Session I
Friday, April 27, 2012
n/a
“Applied Learning:
Reflections of the QEP
pilots”
50
73 8
Special notations
12:00 PM
Long Leaf Pine Room
1041
3:00 PM
12:30 PM
Facilitated by Kate Bruce
4:00 PM
Located in the Honors
Seminar Room
2:00 PM
Facilitated by Candace
Bredbenner and Colleen
Reilly
1:00 PM
Facilitated by Morse,
Hossfeld, and Smith
Located in Randall 2038
4:00 PM
Masonboro Island Room
2011
4:00 PM
Discussion led by Candace
Bredbenner and Colleen
Reilly
Cape Fear Room 2019
1:00 PM
Hosted by Paul Townend,
Lynn Mollenauer and Jess
Boersma
Randall Library
5:00 PM
Hosted by Paul Townend,
Lynn Mollenauer and Jess
Boersma
Hosted at the Clock Tower
Lounge
Appendix E. Assessment Rubrics
74 1.1.1. Applied Learning Critical Reflection Scoring Rubric
Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.
Benchmark 1
Milestone 2
Milestone 3
Capstone 4
Intention1
EBC 1
Demonstrates limited ability to
identify the problem/topic to
be explored and/or the purpose
for engaging in the experience
in terms of personal
educational development.
Loosely identifies the
problem/topic to be explored
and the reasons for engaging in
the experience in terms of
personal educational
development.
Constructs a statement that
describes the problem/topic to
be explored and the expected
educational outcomes of the
experience in terms of personal
educational development.
Transfer of
Knowledge2
EBC 2
Makes vague references to
previous learning but does not
demonstrate how it affected
performance in the applied
learning experience.
Describes own performances
with general descriptors of
success and failure, without
indicating a broader
perspective about personal
educational development.
Refers to previous learning and
provides some insight into how
it affected performance in the
applied learning experience.
Reviews results superficially,
and with no consideration of
the impact on others or on the
field.
Reviews results in terms of the
problem/topic defined and
gives little, if any,
consideration of the impact on
others or on the field.
Connects previous learning
and provides concrete evidence
of how it benefited
performance the applied
learning experience.
Evaluates strengths and
challenges encountered in the
experience, recognizing
complex contextual factors,
and revealing broader
perspectives about personal
educational development.
Reviews results relative to the
problem/topic defined with
some consideration of the
impact on others or on the
field.
Constructs a clear and
insightful statement that
discusses the problem/topic to
be explored and provides
detailed expected educational
outcomes of the experience in
terms of personal educational
development.
Connects and extends previous
learning and synthesizes it in
an innovative way with
performance in the applied
learning experience.
Synthesizes learning to
envision a future direction for
growth and application of
strengths, acknowledging
significantly broadened
perspectives about personal
educational development.
Reviews results relative to the
problem/topic defined with
thorough, specific
considerations of the impact on
others or on the field.
Reflection3
EBC 3
Evaluation
of Impact4
EBC 3
Identifies own strengths and
challenges highlighted by the
experience, indicating
somewhat broader perspectives
about personal educational
development.
Score
Footnotes:
1
Loosely based on Define Problem dimension of Problem Solving VALUE Rubric and Curiosity dimension of Foundations and Skills for Lifelong Learning VALUE Rubric.
2
Modified from Transfer dimension of Foundations for Lifelong Learning VALUE Rubric
3
Modified from Reflection dimension of Foundations for Lifelong Learning VALUE Rubric and Reflection and Self-Assessment dimension of Civic Engagement VALUE Rubric
4
Modified from Evaluate Outcomes dimension of Problem Solving VALUE Rubric
75 1.1.2.
CRITICAL THINKING VALUE RUBRIC
AAC&U Rubric Modified January 2011 UNCW
Definition: Critical Thinking is a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or
formulating an opinion or conclusion.
Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.
Benchmark
Milestones
Capstone
1
2
3
4
Issue/problem to be considered
Issue/problem to be considered
Issue/problem to be considered
Issue/problem to be considered
Explanation of Issues
critically is stated without
clarification or description.
Critically examining viewpoints
of experts
Viewpoints of experts are taken as
fact, without question.
Influence of context and
assumptions
Shows an emerging awareness of
present assumptions (sometimes
labels assertions as assumptions).
Begins to identify some contexts
when presenting a position.
Student’s
position(position, perspective,
Specific position is stated, but is
simplistic and obvious.
critically is stated but description
leaves some terms undefined,
ambiguities unexplored,
boundaries undetermined, and/or
backgrounds unknown.
Information is taken from
source(s) with some
interpretation/evaluation, but not
enough to develop a coherent
analysis or synthesis.
Viewpoints of experts are taken as
mostly fact, with little
questioning.
Questions some assumptions.
Identifies several relevant contexts
when presenting a position. May
be more aware of others’
assumptions than one’s own (or
vice versa).
Specific position acknowledges
different sides of an issue.
Conclusion is inconsistently tied
to some of the information
Conclusion is logically tied to
information (because information
Evidence
Selecting and using information
to investigate a point of view or
conclusion
Information is taken from
source(s) without any
interpretation/evaluation.
thesis, or hypothesis)
Conclusions and related
outcomes
critically is stated, described, and
clarified so that understanding is
not seriously impeded by
omissions.
critically is stated clearly and
described comprehensively,
delivering all relevant information
necessary for full understanding.
Information is taken from
source(s) with enough
interpretation/evaluation to
develop a coherent analysis or
synthesis.
Viewpoints of experts are subject
to questioning.
Information is taken from
source(s) with enough
interpretation/evaluation to
develop a comprehensive analysis
or synthesis.
Viewpoints of experts are
questioned thoroughly.
Identifies own and others’
assumptions and several relevant
contexts when presenting a
position.
Thoroughly (systematically and
methodically) analyzes own and
others’ assumptions and carefully
evaluates the relevance of context
when presenting a position.
Specific position takes into
account the complexities of an
issue.
Others’ points of view are
acknowledged within position.
Conclusion is logically tied to a
range of information, including
Specific position is imaginative,
taking into account the
complexities of an issue. Limits of
position are acknowledged.
Others’ points of view are
synthesized within position.
Conclusions and related outcomes
are logical and reflect student’s
76 Score
(implications and consequences)
discussed; related outcomes are
oversimplified.
is chosen to fit the desired
conclusion); some related
outcomes are identified clearly.
opposing viewpoints; related
outcomes are identified clearly.
informed evaluation and ability to
place evidence and perspectives
discussed in priority order.
1.1.3. INQUIRY AND ANALYSIS VALUE RUBRIC
for more information, please contact value@aacu.org
Definition
Inquiry is the ability to know when there is a need for information, to be able to identify, locate, evaluate, and effectively and responsibly use and share
that information for the problem at hand. – The National Forum on Information Literacy
Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.
Capstone
Milestones
Benchmark
4
3
2
1
Topic selection
Identifies a creative, focused, and
manageable topic that addresses
potentially significant yet previously
less-explored aspects of the topic.
Synthesizes in-depth information
Existing
from relevant sources representing
Knowledge,
Research, and/or various points of view/approaches.
Views
Identifies a focused and
manageable/doable topic that
appropriately addresses relevant
aspects of the topic.
Identifies a topic that while
manageable/doable, is too narrowly
focused and leaves out relevant
aspects of the topic.
Identifies a topic that is far too
general and wide-ranging as to be
manageable and doable.
Presents in-depth information from
Presents information from relevant
Presents information from irrelevant
relevant sources representing various sources representing limited points of sources representing limited points of
points of view/approaches.
view/approaches.
view/approaches.
Design Process
All elements of the methodology or
theoretical framework are skillfully
developed. Appropriate methodology
or theoretical frameworks may be
synthesized from across disciplines or
from relevant subdisciplines.
Critical elements of the methodology Critical elements of the methodology Inquiry design demonstrates a
or theoretical framework are
or theoretical framework are missing, misunderstanding of the methodology
appropriately developed, however,
incorrectly developed, or unfocused. or theoretical framework.
more subtle elements are ignored or
unaccounted for.
Analysis
Organizes and synthesizes evidence
to reveal insightful patterns,
differences, or similarities related to
focus.
Organizes evidence to reveal
important patterns, differences, or
similarities related to focus.
Organizes evidence, but the
organization is not effective in
revealing important patterns,
differences, or similarities.
Lists evidence, but it is not organized
and/or is unrelated to focus.
Conclusions
States a conclusion that is a logical
extrapolation from the inquiry
findings.
States a conclusion focused solely on
the inquiry findings. The conclusion
arises specifically from and responds
specifically to the inquiry findings.
States a general conclusion that,
because it is so general, also applies
beyond the scope of the inquiry
findings.
States an ambiguous, illogical, or
unsupportable conclusion from
inquiry findings.
77 Limitations and
Implications
1.1.4.
Insightfully discusses in detail
relevant and supported limitations
and implications.
Discusses relevant and supported
limitations and implications.
Presents relevant and supported
limitations and implications.
Presents limitations and implications,
but they are possibly irrelevant and
unsupported.
WRITTEN COMMUNICATION VALUE RUBRIC
for more information, please contact value@aacu.org
Definition: Written communication is the development and expression of ideas in writing. Written communication involves learning to work in many genres and
styles. It can involve working with many different writing technologies, and mixing texts, data, and images. Written communication abilities develop through
iterative experiences across the curriculum.
Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.
Capstone
Milestones
Benchmark
4
3
2
1
Context of and Purpose
for Writing
Includes considerations
of audience, purpose,
and the circumstances
surrounding the writing
task(s).
Demonstrates a thorough understanding
of context, audience, and purpose that is
responsive to the assigned task(s) and
focuses all elements of the work.
Demonstrates adequate consideration of
context, audience, and purpose and a
clear focus on the assigned task(s) (e.g.,
the task aligns with audience, purpose,
and context).
Demonstrates awareness of context,
audience, purpose, and to the assigned
tasks(s) (e.g., begins to show awareness
of audience's perceptions and
assumptions).
Demonstrates minimal attention to
context, audience, purpose, and to the
assigned tasks(s) (e.g., expectation of
instructor or self as audience).
Content Development
Uses appropriate, relevant, and
compelling content to illustrate mastery
of the subject, conveying the writer's
understanding, and shaping the whole
work.
Uses appropriate, relevant, and
compelling content to explore ideas
within the context of the discipline and
shape the whole work.
Uses appropriate and relevant content to Uses appropriate and relevant content to
develop and explore ideas through most develop simple ideas in some parts of
of the work.
the work.
Genre and Disciplinary
Conventions
Formal and informal
rules inherent in the
expectations for writing
in particular forms
and/or academic fields
(please see glossary).
Demonstrates detailed attention to and
successful execution of a wide range of
conventions particular to a specific
discipline and/or writing task (s)
including organization, content,
presentation, formatting, and stylistic
choices
Demonstrates consistent use of
important conventions particular to a
specific discipline and/or writing
task(s), including organization, content,
presentation, and stylistic choices
Follows expectations appropriate to a
Attempts to use a consistent system for
specific discipline and/or writing task(s) basic organization and presentation.
for basic organization, content, and
presentation
Sources and Evidence
Demonstrates skillful use of highquality, credible, relevant sources to
develop ideas that are appropriate for
the discipline and genre of the writing
Demonstrates consistent use of credible,
relevant sources to support ideas that
are situated within the discipline and
genre of the writing.
Demonstrates an attempt to use credible Demonstrates an attempt to use sources
and/or relevant sources to support ideas to support ideas in the writing.
that are appropriate for the discipline
and genre of the writing.
78 Control of Syntax and
Mechanics
8.2.5
Uses graceful language that skillfully
communicates meaning to readers with
clarity and fluency, and is virtually
error-free.
Uses straightforward language that
generally conveys meaning to readers.
The language in the portfolio has few
errors.
Uses language that generally conveys
meaning to readers with clarity,
although writing may include some
errors.
Uses language that sometimes impedes
meaning because of errors in usage.
ORAL COMMUNICATION VALUE RUBRIC
for more information, please contact value@aacu.org
Definition: Oral communication is a prepared, purposeful presentation designed to increase knowledge, to foster understanding, or to promote change in the
listeners' attitudes, values, beliefs, or behaviors.
Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance.
Capstone
4
Milestones
3
2
Benchmark
1
Organization
Organizational pattern (specific
introduction and conclusion, sequenced
material within the body, and
transitions) is clearly and consistently
observable and is skillful and makes
the content of the presentation
cohesive.
Organizational pattern (specific
introduction and conclusion,
sequenced material within the
body, and transitions) is clearly and
consistently observable within the
presentation.
Organizational pattern (specific
introduction and conclusion, sequenced
material within the body, and
transitions) is intermittently observable
within the presentation.
Organizational pattern (specific introduction and
conclusion, sequenced material within the body,
and transitions) is not observable within the
presentation.
Language
Language choices are imaginative,
memorable, and compelling, and
enhance the effectiveness of the
presentation. Language in presentation
is appropriate to audience.
Language choices are thoughtful
and generally support the
effectiveness of the presentation.
Language in presentation is
appropriate to audience.
Language choices are mundane and
commonplace and partially support the
effectiveness of the presentation.
Language in presentation is appropriate
to audience.
Language choices are unclear and minimally
support the effectiveness of the presentation.
Language in presentation is not appropriate to
audience.
Delivery
Delivery techniques (posture, gesture,
eye contact, and vocal expressiveness)
make the presentation compelling, and
speaker appears polished and
confident.
Delivery techniques (posture,
gesture, eye contact, and vocal
expressiveness) make the
presentation interesting, and
speaker appears comfortable.
Delivery techniques (posture, gesture,
eye contact, and vocal expressiveness)
make the presentation understandable,
and speaker appears tentative.
Delivery techniques (posture, gesture, eye
contact, and vocal expressiveness) detract from
the understandability of the presentation, and
speaker appears uncomfortable.
Supporting
Material
A variety of types of supporting
materials (explanations, examples,
illustrations, statistics, analogies,
quotations from relevant authorities)
make appropriate reference to
information or analysis that
significantly supports the presentation
or establishes the presenter's
credibility/authority on the topic.
Supporting materials (explanations,
examples, illustrations, statistics,
analogies, quotations from relevant
authorities) make appropriate
reference to information or analysis
that generally supports the
presentation or establishes the
presenter's credibility/authority on
the topic.
Supporting materials (explanations,
examples, illustrations, statistics,
analogies, quotations from relevant
authorities) make appropriate reference
to information or analysis that partially
supports the presentation or establishes
the presenter's credibility/authority on
the topic.
Insufficient supporting materials (explanations,
examples, illustrations, statistics, analogies,
quotations from relevant authorities) make
reference to information or analysis that
minimally supports the presentation or establishes
the presenter's credibility/authority on the topic.
79 Central Message
Central message is compelling
(precisely stated, appropriately
repeated, memorable, and strongly
supported.)
Central message is clear and
consistent with the supporting
material.
Central message is basically
understandable but is not often
repeated and is not memorable.
Central message can be deduced, but is not
explicitly stated in the presentation.
Appendix F. 2011-2012 eTEAL Pilot Budget
Proposal
Resource
Applied Learning Improving
the Southeast
“
“
“
Faculty Pilot Coordinator
“
“
“
Explorations of the Natural
World
“
“
“
“
“
“
Standardized Evaluation of
Applied Learning
Experience Research
“
“
“
“
Graduate Assistant
Poverty Simulations materials/resources
Common course reading for 7 courses x 30
students
Faculty mini-grants
Pilot Evaluation
Travel reimbursement
Graduate Assistant
Garmin eTrex handheld GPS units
5 digital cameras
General Equipment (nets, batteries, waders,
etc.)
Software
10 storage devices/flash drives
Office supplies
Chemicals, DNA sequencing, barcoding,
fixatives, slides
Flourometer
Centrifuge
Field Guides
Personnel Temp EPA faculty
Personnel stipends EPA faculty
Graduate Assistant
Supplies
CSURF Travel Awards (extension of new
program)
Research Grants for students
Graduate Assistantships for CSURF (new
additions)
eTeal Showcase
QEP/UNI/URI/XXX 191 classes as pilot
classes (2 Faculty Development Stipends)
Personnel Supplies/Materials Curriculum Events Assessment
Totals
7,000
11,000
5,000
1,000
7,000
2,500
300
11,000
1,490
850
600
250
200
300
1,500
5,500
800
400
5,310
7,000
11,000
2,000
10,000
5,000
11,000
4,000
“
80 Expanding the Professoriate
“
Graduate Assistant
Faculty Fellows (1 course release each)
4 graduate student stipends @ $1,500
eTeal Showcase
Assessment
GRAND TOTAL
11,000
14,000
6,000
2,000
96,810
24,890
11,000
5,000
5,000
12,300
150,000
Appendix G. eTEAL Preliminary Implementation Budget (2012-2013)
Personnel
Applied Learning Summer Institute
NSEE contract
PHC Ventures contract
eTEAL Supported Pedagogy Initiatives
Michele Parker (WCE)
Kristen DeVall (CAS-SOC)
Nicholaos Syrpis (student)
Jennifer Horan (CAS-PIA)
Shannon Santana (CAS-SOC)
Patricia White (NUR/CHHS)
Michelle Scatton-Tessier (CAS-FLL)
Kathleen Ennen (NUR/CHHS)
Colleen Reilly (CAS-ENG)
Rajni Shankar-Brown (WCE)
Applied Learning &Teaching
Community
Applied Learning Fellow (fall) Atkins
Applied Learning Fellow (fall) Boersma
Applied Learning Fellow (spring) Frampton
Applied Learning Fellow (spring)
Pemberton
Melanie Forehand (half-time graduate
assistant)
Blaine Prescott (half time graduate
assistant)
Applied Learning Newsletter
Pilot: Frampton (BIO) (Collaborative
DIS)
Student travel
Plastics (supplies)
Reagents/chemicals (supplies)
Poster paper/ink
Pilot: Bruce (Undergrad Research)
Student project supplies
Supplies
Faculty/Student Travel
totals
2000
3500
1,000
1,000
2,500
1,000
1,000
1,000
1,000
1,000
1,000
250
500
600
400
500
12,750
350
25,150
3,450
3,450
3,450
3,450
5,500
5,500
1,750
2,074
2,312
864
3,000
81 Instructor stipend (spring)
Instructor stipend (spring)
Instructor stipend (spring)
Instructor stipend (spring)
Instructor stipend (spring)
Totals
1,200
1,200
1,200
1,200
1,200
40,300
16,950
2,150
9,000
59,400
82 Appendix H. eTEAL: Preliminary Implementation Events (ALTC)
Date
Event Title
Monday, August 8, 2012
The Engaged Teaching
Institute at New Faculty
Orientation
“Engaging Students
through
Intention and Reflective
Practice.”
“Enhancing
Student Engagement:
Applied Learning
for Language Learners at
UNCW”
70+
Welcome event for
Applied Teaching and
Learning Community
Workshop: “Simulations,
Games, and Other Virtual
Teaching Tools”
20
ALTC Community Event:
“Your Courses, Your
Materials: Finding
Opportunities
for Applied Learning”
ALTC Community Event:
“Challenges in Working
with Students in Applied
Learning Experiences”
Workshop: “The Group
DIS: Collaboration and
Sustainability”
21
CTE Event: “Celebrating
Teaching with Provost
Denise Battles”
52
Wednesday, September
12, 2012
Thursday, September 15,
2012
Friday, September 21,
2012
Wednesday, October 17,
2012
Wednesday, October 24,
2012
Friday, November 9, 2012
Wednesday, November 14,
2012
Thursday, December 6,
2012
# Participants
25
27
17
Special notations
Facilitated
by Diana Ashe and Jess
Boersma
Conducted by Jess
Boersma, Colleen Reilly,
and Kate Bruce
Organized by Jess
Boersma
The panel consisted of
MarianellaJara, Amrita
Das, Scott Juall, Valerie
Rider, and Amanda
Boomershine
4:00 PM
Located in Randall
Auditorium
Presented by Jeremy
Tirrell, Colleen Reilly, and
Jess Boersma
Located in Randall 2038
3:30 PM
Presented by Jess Boersma
Located in CIS 1007
15
3:30 PM
FSC Wrightsville Beach
Room 2017
15
Presenters: Art Frampton,
Lynn Mollenauer, Jess
Boersma, and Colleen
Reilly
12:00 PM
Located in Clock Tower
Lounge
83 Appendix I. 2012-2013 Pilot Implementation RFP for eTEAL-Supported
Initiatives
eTEAL: Experiencing Transformative Education through Applied
Learning
The Quality Enhancement Plan (QEP) invests in faculty, staff, and students efforts to enhance academic
achievement through applied learning. eTEAL seeks to support transformative educational experiences
through the Applied Learning and Teaching Community, the Applied Learning Summer Institute, and
eTEAL -Supported Initiatives.
Submissions will be accepted until November 16, 2012. Awards will be announced by December 7, 2012. Send to Kim Cook at cookk@uncw.edu with cc to direct supervisor (Department Chair or unit Director) to indicate his/her support. In order to expand funding opportunities, preference will be given to first‐time proposals. UNCW faculty and staff are invited to submit proposals to improve applied learning instruction with students. eTEAL has three overarching goals: 1.
2.
To improve student learning outcomes from applied learning experiences. To provide faculty and staff with information and resources about successful practices in applied learning pedagogy as appropriate to their disciplines and responsibilities. 3.
To promote the incorporation and implementation of successful practices of applied learning throughout UNCW. Addressing goals two and three, the eTEAL‐Supported Initiatives are designed to provide material support for faculty and staff instructors to implement successful practices in applied learning and critical reflection. Thus, we aim to improve student learning and to document student learning through critical reflection in order to capture evidence for goal #3. Specific Aims: as we continue to invest in improving applied learning experiences and capture evidence of those
improvements, proposals for funding should include the following:



Concise description of the “applied” component of the learning opportunity for students, and how the
instructional strategy addresses that application.
Clear statement regarding the specific student learning outcomes for the applied learning experience
(aligned with eTEAL student learning outcomes; see next page).
Concise description of “critical reflection” practices employed during the applied learning experience.
eTEAL‐supported applied learning opportunities may be able to provide salary stipend, and/or materials needed for the learning opportunity, as well as travel associated with learning. Course buy‐outs are not possible. Small group proposals may be submitted to promote innovative AL pedagogy. The average individual awards are anticipated to be approximately $1000 ‐ $1500, based on merit and announced by December 7, 2012. Submissions with potential may be provided an opportunity to revise and resubmit within an efficient turn‐around time. Every effort will be made to provide funding as requested so long as the projects contribute to the overall goal of refining eTEAL. 84 Submissions will be evaluated based on: how you plan to enhance and improve applied learning pedagogy with evidence of a) experiential education practices, b) critical reflection practices, and c) commitment to participate in the Applied Learning and Teaching Community, and d) critical reflection artifacts and products suitable for assessment purposes. Resource material can be found here: www.uncw.edu/qep Format for proposals:
1. Write 2-3 page summary of proposed applied learning experience that includes the following
a. Brief background of this learning opportunity and how it will enhance your current
pedagogical practice
b. Description of how the experience addresses “good practices” of experiential education.
Description of how the experience includes “critical reflection” practices.
c. Course specific learning outcomes and how they align with eTEAL student learning
outcomes (SLOs)
i. SLO1. The student will articulate their expectations, the purpose, and/or the goals
of the experience in terms of their personal educational development.
[Thoughtful Expression]
ii. SLO2. The student will synthesize knowledge drawn from their coursework to
address the issues/challenges/questions involved in the experience. [Critical
Thinking, Foundational Knowledge, Inquiry]
iii. SLO3. The student will communicate the impact or significance on their personal
educational development and on others in the profession or in the field at the
conclusion of the experience. [Critical Thinking]
d. Articulated strategy for assessment, including critical reflection.
e. Itemized budgetary request with brief justifications.
2. Include a statement of commitment addressing your willingness to participate in the Applied
Learning and Teaching Community throughout the spring 2013 semester. For more information
about the ALTC please contact the Center for Teaching Excellence at 962-3034. Participation
will likely include discussing your proposal and your experiences with applied learning within the
community.
3. Affirm your commitment to assisting in eTEAL evaluation process by:
a. Providing your own critical reflection essays examining how the principles and practices
were realized in your experience, and the impact the experience had on you and your
students. Also, please explore whatever lessons from this experience for future applied
learning experiences you might provide.
b. Integrating into the course a student critical reflection, and other student products as
necessary, suitable for QEP assessment purposes that address your student learning
outcomes and all 3 QEP SLOs and providing them to the QEP Assessment team.
85 UNCW Quality Enhancement Plan eTEAL: experiencing Transformative Education through Applied Learning Response to On‐site Committee Report August 20, 2013 3.3.2 Quality Enhancement Plan The institution has developed a Quality Enhancement Plan that (1) demonstrates institutional capability for the initiation, implementation, and completion of the QEP; (2) includes broad‐based involvement of institutional constituencies in the development and proposed implementation of the QEP; and (3) identifies goals and a plan to assess their achievement. (Quality Enhancement Plan) Summary of On‐Site Committee Report Recommendation 2: The Reaffirmation Committee recommends that the university develop and implement appropriate assessment processes that identify outcomes in measurable terms for the QEP and identify assessment measures associated with those outcomes. Summarizing on‐site committee written comments: 

“More information is needed to separate the eTEAL experiences from the non‐eTEAL experiences in order to assess the program’s impact.” “The committee was concerned that the assessment tools are still evolving and that controls for confounding factors such as the number of eTEAL experiences taken by a student are under development. Information in the assessment matrix is vague.” UNC Wilmington Response The purpose of UNCW’s Quality Enhancement Plan, eTEAL—experiencing Transformative Education through Applied Learning—is to positively impact student learning within applied learning experiences. Student learning will be improved (Goal 1) through providing faculty and staff with information and resources about high‐impact practices in applied learning pedagogy as appropriate to their discipline and responsibilities (Goal 2), and by promoting the incorporation and implementation of high‐impact practices of applied learning throughout UNCW (Goal 3). eTEAL is a plan to enhance existing curriculum structures rather than create new ones. Since applied learning experiences are already an established component of the culture of UNCW and each student is required as part of University Studies to participate in at least one applied learning experience, this presents challenges to the assessment of the impact of eTEAL that the on‐site committee recognized. For example, not all instructors will participate in eTEAL professional development nor offer eTEAL‐
associated learning opportunities during the five‐year cycle. On the other hand, some instructors will participate in multiple professional development opportunities and will continually improve their applied learning experiences. It is also likely that instructors who do not directly participate in eTEAL professional development will receive information indirectly. On the student side, some students will not take an eTEAL‐associated applied learning experience, while others may take more than one. To address these confounding factors, the assessment plan has been clarified and expanded as follows. 1. An additional outcome has been added; 1 2. The methodology for analyzing the differences between eTEAL and non‐eTEAL courses has been clarified; 3. A new survey has been created to gather information from students and the instructor survey has been enhanced to obtain information for separating eTEAL and non‐eTEAL experiences; and 4. The assessment matrix has been revised. 1. Additional Program Outcome It has always been the expectation that the information, resources, and support provided through eTEAL would result in improved student learning through instructor implementation of the high‐impact practices presented. The list of eTEAL outcomes has been expanded to make this expectation clearer. Under Goal 3 To promote the incorporation and implementation of high‐impact practices of applied learning throughout UNCW, Outcome 4 has been added: Goal 3. Outcome 4. Instructors participating in eTEAL will incorporate high‐impact practices in applied learning. Assessment of this outcome will provide evidence of what instructors learned in workshops that they plan to implement and have implemented in their courses. Information about the impact that specific forms of professional development have on instructional practice will be documented and used to inform the design and delivery of future workshops. 


Participant end‐of‐workshop surveys will contain the question “Tell us one thing that you learned about applied learning in [this event] that you plan to implement in a course. Provide as much information about the idea as you currently have.” End‐of‐course surveys of instructors whose courses are sampled to measure student learning will include questions on the specific high‐impact practices employed in the course and their importance within the course structure (see Appendix A). Follow‐up surveys and interviews done approximately one year after participation in professional development will include questions on how specific material from these activities affected teaching practice. 2. The Revised Model for Evaluating Overall Impact of the QEP The original QEP proposal for eTEAL contained an Assessment Matrix listing the individual items that will be used in the overall assessment process. It also contained a plan to assess differences between student learning in eTEAL and non‐eTEAL courses (p. 57‐58). All features of the original assessment plan will remain in place, and new features have been added. In combination with the features of the original eTEAL assessment plan, the impact of the QEP will be systematically evaluated through the collection and analysis of information on the types of applied learning instructional practices utilized (intention, planning, reflection, monitoring, acknowledgement, etc.) and through measuring student learning. As previously stated, there will be two distinct groups of applied learning experiences offered at UNCW during the QEP implementation—those experiences that are supported or otherwise directly influenced 2 by eTEAL workshops and groups, and those that are not associated with eTEAL. In the following subsections we describe the model for both measuring these differences and evaluating the significance of these differences on student learning. Method of Analysis Descriptive discriminant analysis (DDA) will be used to analyze the data collected from applied learning experiences to determine the impact eTEAL has on instructional practice and student learning. Discriminant analysis will allow us to look at the factors that best describe the differences, or discriminate, between the two sets of experiences (Klecka). DDA is designed to reveal major differences among groups in terms of the variables in the study. Similar to regression analysis, and like logistic regression, DDA is used when the observations come from a finite number of unique sets, in this case two—eTEAL and non‐eTEAL applied learning experiences. In this model, DDA will be used to answer the following questions: 1. Are the applied learning high‐impact practices used more often or to a greater degree in eTEAL than non‐eTEAL courses? (Hypothesis: high‐impact practices are used more often or to a greater degree in eTEAL courses). DDA will be used to determine if there are statistically significant differences in the number, types, and perceived importance of specific practices between eTEAL and non‐eTEAL courses (Goal 3 Outcome 4). 2. Is student achievement higher in eTEAL than non‐eTEAL courses? (Hypothesis: student demonstration of achievement is higher in eTEAL courses). DDA analysis will be used to determine if there are statistically significant differences in student scores on relevant rubrics between eTEAL and non‐eTEAL courses (Goal 1, Outcomes 1, 2, and 3). DDA will also highlight any difference in student learning related to types of practices implemented. 3. Do any differences diminish over time as the effects of eTEAL permeate the UNCW culture? (Hypothesis: differences will diminish over time as the effects of eTEAL permeate the university culture). DDA analysis will be used to determine if there are statistically significant differences in the results under questions 1 and 2 between each year of implementation. It is expected that there will be differences over time for three reasons. First, assessment findings will be analyzed each year in order to make improvements to eTEAL in the subsequent year. Second, information about high‐impact practices will permeate UNCW. Third, students may participate in more than one eTEAL experience. It is therefore expected that instructor use of high impact practices will increase over the timeframe of the QEP in both eTEAL and non‐
eTEAL experiences, and student performance as measured by rubric scores will increase overtime, with more students reaching the performance benchmarks in each assessment cycle. Analysis will be performed at the end of each academic year. DDA will highlight the variables that have the most weight in differentiating between the two types of experiences. Utilizing DDA will help us determine the type and level of impact eTEAL has on the instructional methods employed in applied 3 learning experiences and on student learning. This information will be disseminated according to the assessment plan and used to inform refinements to eTEAL. In this way, DDA will weave the individual parts of the assessment plan into a comprehensive assessment of impact. eTEAL Informaton, Resources, Support
Impacts
Instructional Practices
Impacts
Student Learning
It is also noted that educational research is an iterative process. Early analysis might point to additional research questions. A final advantage of discriminant analysis is that it will provide additional information that encourages rich analysis. Variables and Sources of Data The unit of analysis is a student applied learning experience. Major characteristics of each experience will be described through five sets of variables: instructional variables, instructor variables, student performance variables, student demographic and preparedness variables, and a control variable. Group 1: Experience type and instructional methodology variables These variables include: 


eTEAL or non‐eTEAL experience type of applied experience, (e.g. internship, honors project, course‐embedded, etc.) instructional practices implemented (intention, critical reflection, acknowledgement, etc.) Data for this group of variables will come from the Applied Learning End‐of Course Instructor Surveys and follow up interviews from the courses sampled, both eTEAL and non‐eTEAL (see next section and Appendix A). Group 2: Instructor Variables: These variables include: 




instructor participation in eTEAL and other relevant professional development and support (quantity and type) instructor experience with using applied learning pedagogies instructor self‐rating of knowledge of applied learning pedagogy instructor self‐rating of knowledge of critical reflection pedagogy instructor identification of most effective applied learning practices 4 Data for this group of variables will come from the Applied Learning End‐of Course Instructor Surveys and follow up interviews from the courses sampled, both eTEAL and non‐eTEAL (see next section and Appendix A). Group 3a: Student performance variables These variables include: 
rubric scores on each dimension of the rubric(s) appropriate for the experience. Data for this group of variables will come from independent scoring of student work by trained faculty and staff. The rubrics that will be used were provided in Appendix e (starting on page 74) of the eTEAL Proposal. Group 3b: Student demographic and preparedness variables These variables include: 








gender race or ethnicity total number of credit hours completed grade point average transfer or freshman start SAT scores number of previous applied learning experiences number of previous eTEAL experiences experience with critical reflection Data for this group of variables will come from information in the Banner system, except the last three, which will come from the Applied Learning End‐of‐Course Student Survey from the courses sampled, both eTEAL and non‐eTEAL (see next section and Appendix B). Group 4: Control variable There is only one variable in this group: 
academic year (i.e., 2013‐2014, 2014‐2015, etc.) Data from each academic year will be duly coded for each student experience. 3. New and Revised Assessment Surveys A student end‐of‐course survey has been added and the instructor end‐of‐course survey has been enhanced in order to collect data from both students and instructors to control for confounding factors. Information will be collected, from both eTEAL and non‐eTEAL instructors whose courses are part of the learning assessment sample, about instructor professional development experiences and the 5 pedagogies utilized within the experience. This information will be collected through the Applied Learning End‐of‐Course Instructor Survey provided in Appendix A at the end of each applied learning course sampled. Additional follow up information will be collected through interviews as needed. Information will be collected from all students, in both eTEAL and non‐eTEAL sampled courses, about other applied learning experiences (quantity, types) and about the students’ familiarity with critical reflection. This information will be collected through the Applied Learning End‐of‐Course Student Survey provided in Appendix B at the end of each applied learning course sampled. The information from these surveys will be used as the source of data for previous experience and current experience variables in the impact model. 4. Revised Assessment Matrix The Revised Assessment Matrix is presented below in Appendix C, and contains all of the changes listed above. The matrix contains assessment information for all nine outcomes. The means of assessing each outcome are provided in the matrix. For Goal 1, Outcomes 1, 2, and 3 (student learning outcomes), and Goal 3, Outcome 4 (Instructor use of high‐impact practices), the assessment measures will be part of the discriminant analysis of overall impact. Note: The Revised Assessment Matrix contains the discrete elements of the assessment plan. The overarching impact model utilizes assessment components from across the three goals, and therefore is best described above. 6 References Klecka, W. R. (1980) Discriminant analysis. Beverly Hills: Sage Publications. Rhodes, T.L., ed. (2010) Assessing outcomes and improving achievement: Tips and tools for using rubrics. Washington, D.C.: Association of American Colleges and Universities. 7 Appendix A Applied Learning End‐of‐Course Instructor Survey Applied Learning End-of-Course Instructor Survey
Welcome
The purpose of eTEAL, UNCW’s Quality Enhancement Plan, is to improve student learning through applied experiences. Whether your applied learning course is part of eTEAL or Explorations Beyond the Classroom (or both), information about the design of your experience is important to us, and will help direct future eTEAL efforts. Page 1
Applied Learning End-of-Course Instructor Survey
Tell us about your current applied learning course.
The questions on this page relate to the course you just taught, from which you provided student work. 1. How do you characterize the applied learning experience you offered this semester?
You may choose more that one descriptor.
c Directed Individual Study (DIS)
d
e
f
g
c Group DIS
d
e
f
g
c Internship
d
e
f
g
c Honors project
d
e
f
g
c Service learning
d
e
f
g
c Study abroad
d
e
f
g
c Course­embedded experience (applied project within a course, which might also fall into another category)
d
e
f
g
c Practicum
d
e
f
g
c Student teaching
d
e
f
g
Other (please specify) Page 2
Applied Learning End-of-Course Instructor Survey
2. To what extent (if any) were each of the following Principles of Good Practice (NSEE) an
important part of this applied learning experience?
Intention – communicating Very Important
Important
Slightly Important
Not Important
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
from the outset why the experience is the chosen approach to the learning that is to take place.
Preparedness and Transfer of Knowledge – ensuring that participants have the necessary knowledge and skills (obtained from previous courses or within this course) and then providing the opportunity to apply them.
Authenticity – using a real world context (designed in concert with those that will be affected or use it).
Reflection – using the reflection process to solidify learning.
Orientation and Training – preparing the learner and the facilitators with important background information about each other and about the context and environment surrounding the experience.
Monitoring and Continuous Improvement – utilizing methods to formatively assess the progress of the experience and using that information to adjust as necessary.
Assessment and Evaluation – utilizing formal summative assessment to document learning and other information about the experience.
Acknowledgement – documenting and/or celebrating learning and impact to help provide closure and sustainability.
Page 3
Applied Learning End-of-Course Instructor Survey
3. What principles of applied learning were most helpful to you as you conducted your
applied learning course this semester (from the list above or other principles)?
5
6 4. Can you pinpoint a specific conference or learning opportunity through which you
learned about the principles you listed above?
5
6 Page 4
Applied Learning End-of-Course Instructor Survey
Tell us about your previous experience with applied learning.
5. How many other applied learning courses have you taught? Please enter an estimate
between 0 and 100. Enter 100 if there are too many to count.
6. Have any of these previous applied learning courses been eTEAL courses?
j Yes
k
l
m
n
j No
k
l
m
n
7. Which of the following eTEAL professional development activities have you participated
in?
c eTEAL Summer Institute (2­3 day summer workshop)
d
e
f
g
c eTEAL Applied Learning and Teaching Community (community of practice that meets regularly to discuss applied learning pedagogy)
d
e
f
g
c eTEAL Supported Initiatives (grants to help implement applied learning practices)
d
e
f
g
c None
d
e
f
g
Other eTEAL workshop (please specify) 8. Have you participated in any non­eTEAL professional development activities related to
applied learning? If so, describe them, including where and when you participated.
5
6 9. How do you rate your knowledge of...
applied learning High
Medium
Low
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
j
k
l
m
n
pedagogy?
critical reflection pedagogy?
10. What elements or principles of applied learning do you think are indispensable to any
applied learning experience?
5
6 Page 5
Applied Learning End-of-Course Instructor Survey
11. What else would you like to share with us?
5
6 Page 6
Applied Learning End-of-Course Instructor Survey
Thank you for your time.
If you would like to find out more about the principles of good practice for applied learning or the professional development offered through eTEAL, please contact the eTEAL director, Jess Boersma (boersmaj@uncw.edu). Page 7
Appendix B Applied Learning End‐of‐Course Student Survey Applied Learning End-of-Course Student Survey
Applied learning is a valuable instructional technique used at UNCW in which students apply and practice skills in authentic situations. All UNCW undergraduates must participate in at least one applied learning course/experience. The information you provide in this survey will help the UNCW faculty maintain and improve learning within these experiences. Thank you for your time and thoughtful responses. Your responses are not anonymous, but they are confidential. Your answers will be matched with the work you submitted in the class. However, the instructor will not be provided with individual responses to this survey. Your responses WILL NOT affect your grade in the course, or any other course work at UNCW. They will, however, help the university to identify potential ways to improve student learning. Page 1
Applied Learning End-of-Course Student Survey
1. How many previous applied learning experiences have you had at UNCW or another
college? See question number 2 for a list of the types of experiences that are called
applied learning.
j 0, this is my first
k
l
m
n
j 1
k
l
m
n
j 2
k
l
m
n
j 3
k
l
m
n
j 4
k
l
m
n
j 5 or more
k
l
m
n
2. If this is not your first applied learning experience, which of the following types of
applied learning have you participated in? Select all that apply.
c Directed Individual Study (DIS)
d
e
f
g
c Group DIS
d
e
f
g
c Internship
d
e
f
g
c Honors project
d
e
f
g
c Service learning
d
e
f
g
c Study abroad
d
e
f
g
c Course­embedded experience (applied project within a course, which might also fall into another category)
d
e
f
g
c Practicum
d
e
f
g
c Student teaching
d
e
f
g
Other (please specify) 3. Were you asked to write a critical reflection during this current course? (Critical
reflection is the act of examining an experience in order to gain insight into your
educational growth during the experience and the implications of this for your future.)
j Yes
k
l
m
n
j No
k
l
m
n
Page 2
Applied Learning End-of-Course Student Survey
4. Did you have experience with reflection before this course?
j Yes
k
l
m
n
j No
k
l
m
n
5. If so, in how many situations? (Type in a number 0 or above.)
in UNCW courses
in other college courses
in extracurricular activities
in high school
Page 3
Applied Learning End-of-Course Student Survey
You have completed the survey. Click done to submit.
Thank you for your responses. UNCW Quality Enhancement Plan Team Page 4
Appendix C Revised Assessment Matrix 1 Revised Assessment Matrix (Additions and changes in red.)
Outcome Assessment Methods Goal 1: To improve student learning in applied learning experiences
Goal 1. Outcome 1. Students Student critical reflections during their will articulate their applied learning experience (direct) expectations, the purpose, [Intention dimensions of the Critical and/or the goals of the Reflection and Written Communication experience in terms of their Rubrics, Appendix X] personal educational development. [Thoughtful Other work products from the experience such as initial application or Expression] discussion with mentor, if applicable (direct) [Intention dimensions of the Critical Reflection Rubric, Appendix X] DDA will determine if there are differences in student scores between eTEAL and non‐eTEAL experiences. Goal 1. Outcome 2. Students Student critical reflections during their will synthesize knowledge applied learning experience (direct) drawn from their coursework [Critical Reflection and Written to address the Communication Rubrics, Appendix X] issues/challenges/questions involved in the experience. Other work products from the [Critical Thinking, Inquiry, experience such as project reports and presentations (direct) [Critical Thinking, Thoughtful Expression] Inquiry, Written Communication and Oral Communication Rubrics, Appendix X] DDA will determine if there are differences in student scores between eTEAL and non‐eTEAL experiences. Responsible Entities Performance Benchmark Schedule Benchmark scores for graduating seniors is a level 4. For 100‐, 200‐, and 200‐level courses, the corresponding milestones are 1, 2, and 3, respectively. 1
Baseline data of percentages of students at each of these levels from 2013‐2014 will be used to set targets for growth. Student work products will be collected from eTEAL and non‐eTEAL courses during each semester. Student work will be scored annually in late spring. The DDA impact assessment will be performed each summer. QEP Director and staff, General Education Assessment Director and staff Benchmark scores for graduating seniors is a level 4. For 100‐, 200‐, and 200‐level courses, the corresponding milestones are 1, 2, and 3, respectively.1
Baseline data of percentages of students at each of these levels from 2013‐2014 will be used to set targets for growth. Student work products will be collected from eTEAL and non‐eTEAL courses during each semester. Student work will be scored annually in late spring. The DDA impact assessment will be performed each summer. QEP Director and staff, General Education Assessment Director and staff 2 Revised Assessment Matrix (Additions and changes in red.)
Outcome Assessment Methods Performance Benchmark Responsible Entities Schedule Goal 1. Outcome 3. Students will communicate the impact or significance on their personal educational development and on others in the profession or in the field at the conclusion of the experience. [Critical Thinking, Thoughtful Expression] Benchmark scores for Student work products and QEP Director and Student critical reflections during their graduating seniors is a level 4. surveys will be collected applied learning experience (direct) staff, General For 100‐, 200‐, and 200‐level from eTEAL and non‐eTEAL Education [Reflection and Impact dimensions of courses, the corresponding courses during each Critical Thinking Rubric] Assessment Director milestones are 1, 2, and 3, semester. Student work will and staff Other work products from the respectively. 1
be scored annually in late Baseline data of percentages of spring. experience such as exit interviews, if students at each of these levels applicable (direct) from 2013‐2014 will be used to [Reflection and Impact dimensions of set targets for growth. Critical Thinking Rubric] Benchmarks are not Student end‐of‐course surveys on appropriate for this tool. previous applied experiences. The DDA impact assessment DDA will determine if there are will be performed each differences in student scores between summer. eTEAL and non‐eTEAL experiences and based on previous experience. Goal 2: To provide faculty and staff with information and resources about high‐impact practices in applied learning pedagogy as appropriate to their discipline and responsibilities. Goal 2. Outcome 1. UNCW One Summer Institute each Number of events and number of Descriptions and number of QEP Director and will sponsor multiple year; participants at events events will be cataloged on staff opportunities for formal and One workshop each semester; a continuous basis. informal faculty and staff Participation will be development to enhance counted at each event. applied learning pedagogy. Participant satisfaction surveys which will Baseline data will be collected Satisfaction surveys will be administered within two on satisfaction and contain questions regarding expected days of each event. implementation during initial impact of workshop material on future Follow up surveys will be year and used for continual instructional practice. administered once a year. program improvement. Follow up surveys and interviews on implementation. 3 Revised Assessment Matrix (Additions and changes in red.)
Outcome Assessment Methods Goal 2. Outcome 2. UNCW will develop a library of information about high‐
impact applied learning practices. Activities will include a web‐based repository of high‐impact applied learning practices, including critical reflection prompts. Performance Benchmark Responsible Entities Schedule Baseline data will be collected Number of articles will be on the number of items counted once a year. uploaded during initial year and used to set benchmarks for continual program improvement. Baseline data will be collected Number of hits will be on site usage during initial year recorded monthly. and used to set benchmarks for continual program improvement. Baseline data will be collected Follow up surveys will be administered once a year. on ways information is used during initial year and used for continual program improvement. Goal 3: To promote the incorporation and implementation of high‐impact practices of applied learning throughout UNCW.
Goal 3. Outcome 1. UNCW Number of proposals submitted and
Baseline data will be collected Collected after each call for will sponsor faculty and staff scores on these proposals on the number of submissions proposal ends and applied learning and the scores during initial selections made. implementation through year and used to set eTEAL Supported Initiatives. benchmarks for continual Activities include annual call program improvement. for proposals, selection and funding of proposals. Thirty proposals will be Number of proposals supported and supported each year; $105,000 dollar amount of support per year Number of articles/items in web‐based repository Number of hits on repository pages Faculty and staff follow up surveys on the use of the repository 4 QEP Director and staff QEP Director and staff Revised Assessment Matrix (Additions and changes in red.)
Outcome Assessment Methods Goal 3. Outcome 2. UNCW will host teaching and learning community events where faculty and staff will share implementation activities and results. Activities will include Applied Learning and Teaching Community (ALTC) discussion groups, reading groups. Number of events and number of participants at events Participant satisfaction surveys containing questions regarding expected impact of workshop material on future instructional practice. Goal 3. Outcome 3. UNCW will support and follow up with eTEAL participants. Notes on periodic contacts during implementation of eTEAL sponsored experiences. Notes on periodic contacts with Summer Institute participants. Goal 3. Outcome 4. Instructors participating in eTEAL will incorporate high‐
impact practices in applied learning. Performance Benchmark Baseline data will be collected during initial year and used to set benchmarks for continual program improvement. Baseline data will be collected during initial year and for continual program improvement. 100% of Sponsored Initiative
participants will receive on‐
going support of the QEP director and advisory board. Baseline data will be collected during initial year and used for continual program improvement. For instructors selected for Follow up surveys and interviews with eTEAL Supported Initiatives, eTEAL Supported Initiative instructors, 100% will report utilizing at Summer Institute, other workshop, and least one high‐impact practice ALTC participants, and users of the web repository. (Moved from Goal 3 Outcome during the supported period. For non‐eTEAL activities, 1) baseline data will be collected in 2013‐2014 and used for continual program improvement. Instructor critical reflections at the end of eTEAL Supported Initiative Baseline data on 5 Responsible Entities Schedule Events and participants will be listed at the end of each event. Surveys will be administered at the end of workshops and at the end of each semester for ongoing activities. On going
On going QEP Director and staff Annually
Collected at the end of each semester, analyzed using QEP Director and staff, General Education Assessment Director and staff QEP Director and staff Revised Assessment Matrix (Additions and changes in red.)
Outcome Goal 3. Outcome 4 cont’d. Responsible Entities Assessment Methods Performance Benchmark Schedule experiences. (Moved from Goal 3 Outcome 1) Annual post‐implementation follow up surveys and interviews on continued implementation (Moved from Goal 3 Outcome 1) End‐of‐course survey of instructors of applied learning courses (eTEAL and non‐
eTEAL) sampled for assessment of student work. Information from these sources will be analyzed on their own and as part of the descriptive discriminant analysis to determine if there are differences in instructional practice between eTEAL and non‐eTEAL experiences. implementation will be collected during initial year and used to set benchmarks for continual program improvement. Baseline data on implementation will be collected during 2013‐2014 and used to set benchmarks for continual program improvement. There are no a priori expectations about which practices will be used or determined to be most beneficial. qualitative techniques
Follow up surveys and interviews will be administered annually. The DDA impact assessment will be performed each summer. It will be presented to the Advisory Board for discussion of potential program improvements. Annual reports will be made to the Board of Trustees for further feedback. 1
The VALUE rubrics and the locally created rubrics are designed on a 0 to 4 scale. According to AAC&U, “the capstone [4] level reflects the demonstration of achievement for the specific criterion for a student who graduates with a baccalaureate degree. Milestones [2 and 3] suggest key characteristics of progressive learning as students move from early in their college experience to the completion of the baccalaureate degree” (Rhodes, 2010, p.3). Locally‐created rubrics were designed to follow these same levels. 6