“Davenport’s Excellence System Assessment Pilot:
Measurable Competencies for Improved Teaching and
Learning, Student Certification, and Workplace Success”
Wynalda TLI Presentation
August, 2014
Dr. Irene Bembenista: Vice Provost for Assessment & Graduate Studies
Dr. Wayne Sneath: Program Director Experiential Learning
Mike Carey: Department Chair, Marketing
Audience Question Cards:
Please include e-mail address
& questions for follow up
Session Objectives

To explain the purpose and design of the Excellence System
Assessment Pilot

To explain the proposal for certification of students in the
Excellence System

To present data on the assessment of all nine Excellence
System competencies in courses from two years of the pilot

To generate discussion on the implications of this ongoing
assessment on improved teaching and learning and
workplace success
The DU Excellence System History
The Mission of Davenport University is “to prepare
individuals and organizations to excel in the knowledgedriven environment of the 21st Century.”
To that end, the Davenport University Excellence System
was created in partnership with employers who helped to
define and reinforce the necessity of these competencies.
The Excellence System consists of nine professional
competencies necessary for graduates to engage in life
long-learning and succeed in their chosen profession.
The Provost’s Charge
Committee Formed in January, 2012
“The committee shall use the DU Excellence System to
develop a methodology and a process that will validate
students graduating from DU shall have achieved the
skills necessary for each of the DU principles of
excellence”
Committee Proposal for Excellence System Certification
Summary:
1.
Assessment of proficiency in all 9 Excellence System competencies in
key Foundations of Excellence courses & provision of feedback via the
development of an electronic portfolio system
2.
Assessment of proficiency in all 9 Excellence System outcomes in midlevel and capstone-level courses in each program via the portfolio system
3.
Determination of minimum cumulative GPA in all assessed courses
What is Proposed as Certification in the Excellence System?
Minimum score at the Mastery Level in each Excellence System
Competency via analysis of assignments from capstone and/or
other program courses (score TBD via pilot)
+
Minimum cumulative GPA in all assessed courses in the
Foundations + Program Level (GPA TBD)
=
Certification
Year I: Goals of the Pilot
1.
Determine a baseline of performance in all 9 Excellence System
competencies by examining required assessment assignments
from three Foundations of Excellence courses:
COMM 120: Presentation Techniques
RAA:
Final Informative or Persuasive Speech
Competencies Assessed:
• Ethical Reasoning and Action
• Information & Tech Proficiency
• Professional Communication
ENGL 311: Professional Writing
RAA:
Group Project
Competencies Assessed:
• Critical & Creative Thinking
• Analysis & Problem Solving
• Written Communication
• Leadership & Teamwork
SOSC 201: Diversity in Society
RAA:
Experiential Learning Paper
Competencies Assessed:
• Global & Intercultural Competence
• Civic and Social Responsibility
2.
Year 1: Goals of the Pilot
Determine performance in all 9 Excellence System
competencies at mid-level and capstone-level courses
in programs:
Marketing was the pilot program with assessment in:


MKTG 213: Advertising Foundations
MKTG 441: Strategic Marketing (Capstone)
3.
Create common rubrics for all 9 Excellence System
competencies utilizing cross-college faculty teams
4.
Report data to the Provost with recommendations by
May 1, 2013, including a recommendation for
development of an electronic portfolio system through
which chosen assignments can be deposited and
assessed by course-level faculty
Some Important Questions to be Answered via the Pilot
1. What are the appropriate Foundations of Excellence,
Foundations of Business, Technology, or Health, and/or
Program courses in which to assess the competencies?
2. Is a review of course RAAs via a common rubric the best
means of assessment? Should we recommend nationally
normed exams or create our own?
3. How can we effectively train all instructors in these courses
to perform consistent assessment of competencies and
report this data?
4. What are the best means to assess competencies not
easily evaluated by a team via an artifact (technology
proficiency, teamwork, professional communication?)
5. What role should ongoing assessment teams play for the
Excellence System? Are they cross-college or only by
Program? Should they be a “quality check” on course-level
instructor assessment scores?
Process for Collection & Assessment of RAAs
• Faculty in chosen sections require students to post
RAAs or RAAs are downloaded from existing course
shells
• A sample of all RAAs submitted were downloaded &
identifying information was removed
• 4 person, cross-college faculty assessment teams
were formed and trained in inter-rater reliability and
sample scoring; teams read samples to calibrate
before scoring independently
• Teams assessed 6 of 9 competencies
• Course faculty assessed “Observable” competencies
of Information & Technology Proficiency, Leadership
& Teamwork, and Professional Communication
E-Portfolio System Vetting Committee
Purposes:
•
Research options for an e-portfolio system which
would assist with ongoing Excellence System
assessment
•
Recommend a system with proposed costs and
implementation timeline to the Provost as part of the
pilot
•
Portfolio system should serve multiple purposes:




ES Assessment & Student Formative Feedback
Faculty Rank & Promotion Dossiers
Student Career Development
Prior Learning Assessment Portfolios
Irene
Year One Pilot Results
for:
Global &
Intercultural
Competence
Global & Intercultural Competence
Graduates understand that working and succeeding in an inclusive, multicultural,
and international world involves complex issues present in diverse environments
1.
Evaluates and demonstrates culturally-appropriate behavior in diverse domestic and
global contexts
2.
Interprets issues and events from diverse domestic and global perspectives
3.
Demonstrates an understanding of the dynamics of power, privilege and inequality and
appropriate interventions/responses to bias in diverse domestic and
global contexts
4.
Demonstrates awareness of personal culture and biases and how these influence
interactions/relationships in diverse domestic and global contexts
5.
Demonstrates an ability to interact successfully with people from diverse domestic and
global perspectives
6.
Understands and examines differing value systems to determine appropriate courses of
action and decision making in the diverse work environment and the domestic and
global community
RAAs Mapped to GIC
-- Introductory Level—SOSC 201’s Experiential
Learning Paper
-- Reinforced Level—MKTG 213’s Group Advertising
Plan
-- Mastery Level—MKTG 441’s Group Strategic
Marketing Plan
Inter-Rater Reliability for GIC
-- Only 32% of RAAs required a 3rd reader, thus 68%
agreement
-- AACU’s standard—reliable at 70% or greater when two
readers are within one level of scoring agreement on a
rubric
-- GIC was one of the highest reliabilities of the 9 Excellence
System competencies assessed in Year One
2012-13 Results: GIC in SOSC 201
Global & Intercultural Competency Results by Rubric Outcome In SOSC 201
Scale 0-3
1.40
1.34
1.30
1.28
1.20
1.05
0.98
1.00
0.78
0.80
0.60
0.70
0.54
0.58
0.62
0.64
0.62
Mean
Standard Deviation
0.40
0.20
0.00
Outcome 1
Outcome 2
Outcome 3
Outcome 4
Outcome 5
Outcome 6
--
Mean score was 6.7 of 18 rubric points (6 outcomes x 3)
--
94% of students were proficient in this competency in SOSC 201
--
27% of these scored above the Introductory Level
Implication I
-- 94% proficiency at Introductory Level does NOT
demonstrate the global aspect of this competency
-- GIC Rubric at Introductory Level assesses either domestic
OR global competence
-- GIC Rubric at the Mastery Level assesses both domestic
AND global competence
% of Students Proficient in
Global & Intercultural Competence at Introductory Level in SOSC 201
6%
Proficient
Not Proficient
N=110
94%
Implication II
-- Students scored lowest in GIC Competency Outcome 6:
“Understands and examines differing value systems”
Aggregate Results of
All 9 Excellence System
Competencies
Percentage of Pilot Students Proficient vs. Not-Proficient
in all Excellence System Competencies: Year One Pilot Data
Excellence System Competency in Foundations of Excellence Courses:
COMM 120, SOSC 201, & ENGL 311
100%
94%
90%
91%
80%
78%
80%
63%
70%
81%
77%
63%
60%
49% 51%
50%
37%
40%
30%
20%
20%
10%
6%
37%
22%
23%
19%
9%
0%
% Proficient
% Not Proficient
Total Unique
Students Assessed:
N=1076
Mike
Year One Pilot:
Marketing Program Data
Year One Pilot: MKTG Program
Percentage of MKTG441 Students Proficient v. Not
Proficient by Excellence System Competency
87%
90%
78%
80%
71%
63%
70%
56%
60%
44%
50%
40%
54%
46%
37%
29%
30%
22%
13%
20%
% Proficient
10%
% Not Proficient
0%
Critical &
Creative
Thinking N= 24
Analysis &
Problem
Solving N= 24
Leadership &
Teamwork N=
41
Information &
Technology
Proficiency
N=41
Written
Professional
Communication Communication
N= 24
N= 41
Total Unique
Students
Assessed: N=65
Marketing Program Excellence System Certification
Analysis for MKTG 441
Comparison of Mastery Competency Proficiency, 2.3 or Higher Program GPA, and
Combined Mastery Proficiency & Minimum Program GPA
94%
100%
90%
80%
70%
%
60%
50%
N=65
29%
26%
40%
30%
20%
10%
0%
% of MKTG 441 students scoring
at or above Mastery level in at
least 3 ES Competencies
% meeting minimum program
major GPA
% meeting both Mastery level
proficiency & minimum program
GPA for Excellence System
Certification
Correlation Analysis of
MKTG Program GPA to RAA Assessment Performance
60
Competency Score Total
50
40
30
Series1
Linear (Series1)
20
10
0
0
0.5
1
1.5
2
2.5
MKTG Major GPA
3
3.5
4
4.5
Irene
Year Two Pilot Planning
&
Results
Conclusion to Year One Pilot
• Average rubric scores compiled and report data was
provided to the Provost with recommendations in June,
2013
• Provost and Deans requested a second year of
comparative data
• Work began to plan Year Two of the pilot: 2013-14
Planning Year Two of the Excellence System Pilot
• Score a sample of papers from Fall 2013
courses and compare assessment teams’
scores to those already completed by course
faculty to determine reliability of instructor
scores as a means to do ongoing
assessment
• Continue to refine RAA assignments and
Excellence System rubrics
• Use data to determine baseline proficiency
scores for certification at the Program Level
• Inform curriculum mapping and construction
of new RAA assignments
• All PROGRAMS in the colleges charged to
identify courses to assess RAAs for all levels
of Excellence System proficiency—I, R, and
M
2013-14 Pilot: Phase II
Pilot Courses:
12 courses, 80+ sections, 1100+ students in
Fall 2013 & Winter 2014
CONTINUED ASSESSMENT OF THE FOLLOWING COURSES:
COMM 120: Professional Communication
ENGL 311: Leadership & Teamwork, Written Communication
SOSC 201: Global & Intercultural Competence, Civic & Social
Responsibility
ADDITIONAL COURSES ADDED FOR ASSESSMENT IN YEAR TWO:
HLTH 101: Critical & Creative Thinking, Analysis & Problem Solving
FRSM 100: Critical & Creative Thinking, Analysis & Problem Solving
BUSN 210: Ethical Reasoning & Action
CISP 112: Information & Technology Proficiency
NEW Marketing Program Courses: MKTG 310, 412, 421, 430, 441
Year Two Pilot Results:
General
Total Unique
Students Assessed:
2013: N=1076
Total Unique
Students Assessed:
2014: N=654
Excellence System Competency Means 2013 v. 2014
2013 Mean Score
2013 v. 2014 Comparison:
Mean Scores by
Excellence System Competency
2014 Mean Score
2014 I=4-7
2013 I=6-11
19.0
20.0
18.0
16.0
14.0
10.0
I =4-7
8.0
6.7
4.0
13.2
R=10-14
R=8-10
12.0
6.0
I=4-7
2014 I=4-7
2013 R=8-12
2014
I =3-6
2013
I=4-7
5.4
5.0
2014
R=8-12
2013
I=4-7
2014 I=4-7
2013 R=8-10
10.3
7.4
5.9
11.2
9.9
9.9
6.3
4.9
4.6
3.5
2.0
0.0
*The number of ITP Rubric Outcomes was reduced from 8 to 5, CSR
from 5 to 4, and APS from 6 to 5 in 2014 reflecting different range
9.7
10.3
12.4
Excellence System Proficiency Comparison 2013 v. 2014
2013 % Proficient
2013 % Not Proficient
2014 % Proficient
2014 % Not Proficient
2013 v. 2014 Comparison:
Percentage of Students Proficient
vs. Not-Proficient
by Excellence System Competency
100%
97%
94%
90%
86%
80% 82%
84%
78%
83%
63%
58%
72%
63%
60%
49%
50%
37%
37%
28%
30%
20%
20%
0%
51% 52%
48%
42%
40%
10%
96%
81%
77%
80%
70%
91%
6%
18%
23%
22%
14%
16%
17%
19%
9%
3%
4%
Mike
Marketing Program:
Year Two Pilot Results
2014 Marketing Program Courses & Sample Sizes
GIC (MKTG 421: N=26)
CSR (MKTG 310: N=0)
ERA (MKTG 310: N=17)
CCT (MKTG 430 N=40)
APS (MKTG 412: N=46)
LT (MKTG 441: N=33)
ITP (MKTG 412: N=46)
WC (MKTG 421: N=26)
PC (MKTG 441: N=33)
Total Unique
Students Assessed:
2014: N=162
Marketing Program Competency Means 2013 v. 2014
2013 v. 2014 Comparison:
Marketing Student Mean Scores by
Excellence System Competency
2014 M=11-15
2013 M=17-24
12.0
2014 M=13-18
2013 M=11-15
M=13-18
8.7
10.0
M=15-21
12.0
M=13-18
11.0
M=11-15
9.6
10.0
10.4
10.0
8.8
8.0
8.0
7.2
7.0
M =13-18
5.5
6.0
4.0
M=13-18
4.4
4.9
2014 M =9-12
2013 M=11-15
2.0
N/A
N/A N/A
N/A
0.0
*The number of ITP Rubric Outcomes was reduced from 8 to 5,
CSR from 5 to 4, and APS from 6 to 5 in 2014 reflecting different
proficiency range scores.
2013 Mean Score
2014 Mean Score
Marketing Program Proficiency Comparison 2013 v. 2014
2013 % Proficient
2013 v. 2014 Comparison:
Percentage of Marketing Students Proficient
vs. Not-Proficient
by Excellence System Competency
100%
2013 % Not Proficient
2014 % Proficient
2014 % Not Proficient
100%
100%
100%
87%
90%
87%
85%
78%
80%
71%
70%
70%
56%
60%
44%
50%
40%
61%
55%
45%
54%
46%
39%
63%
37%
30%
29%
30%
22%
13%
20%
10%
0%
N/A 0%
N/A
N/A 0%
15%
13%
0%
MKTG Program Lessons Learned
Need to Determine the Right:
•
Course aligned with the Excellence System competency
•
Assignment (RAA) with the Excellence System competency
 Team v. Individual assignments
•
Rubric language
•
Training for faculty on scoring using rubrics
 Variance in Team v. Faculty scoring in Year Two
Remember:
• Not a lot of additional time in reading assignments: most RAAs are read
once for a course grade, but Excellence System rubric scores do NOT
impact students’ course grades
• Not all 9 Excellence System competencies are assessed in each course;
assessment is spread out across Foundations and Program-Level
courses
MKTG Program Next Steps
Davenport University Excellence System Assessment Mapping
Program: Marketing Program 2014-15
REINFORCED MASTERED
RAA
ASSIGNMENT
Excellence System Competencies
INTRODUCED
Global & Intercultural Competence
SOSC201
ECON200
MKTG 421
Country Notebook
Civic & Social Responsibility
SOSC201
BUSN225
MKTG 310
Ethics/CSR Paper
Ethical Reasoning & Action
ENVS125
BUSN210
MKTG 310
Ethics/CSR Paper
Critical & Creative Thinking
ENGL109
LEGL210
MKTG 430
New Product Development Plan
Analysis & Problem Solving
ENGL110
STAT220
MKTG 412
Primary Research Project
Leadership & Teamwork
ENGL110
ENGL311
MKTG 441
Strategic Marketing Plan
Information & Technology Proficiency
COMM120
ACCT202
MKTG 412
Primary Research Project
Written Communication
ENGL109
ENGL311
MKTG 421
Country Notebook
Professional Communication
COMM120
ENGL311
MKTG 441
Strategic Plan Presentation
Wayne
Current Work
&
Overall Project
Next Steps
Curriculum Mapping, CRIRs, and ES Data Collection
Summer 2014 Work
1. Excellence System competencies mapped at
I,R, and M levels for all academic programs
across the colleges
2. CRIR revisions to identity RAAs to be
assessed against ES common rubrics
3. Development of an Excellence System
assessment website:
www.davenport.edu/academics/assessment/excellence-system
4. Development of Google Drive score sheet
system for 9 Excellence System
competencies to be used by course faculty to
report data
Project Next Steps
• Determination on final criteria for Excellence
System Certification
• Begin assessment in all sections of all
identified Excellence Systems assessment
courses across all academic programs in Fall
2014
• E-Portfolio pilot with College of Urban
Education cohort (Fall 2014)
• Broader portfolio pilot sample (Winter 2015)
Audience Questions
and Comments