Course Embedded Assessment of Graduate Learning: An
Innovative Approach Based on The Degree Qualifications
Profile
Presented by
William M. White, Ed. D. and Indre Cuckler, M. A.
School of Graduate Studies
Who We Are
Who We Are
Who We Are
Who We Are
Outline










Levels of Assessment
Definition of Assessment
Theoretical Framework
Program Goals and Outcomes
Competency-Based Assessment
Course Embedded Assessment Model
Assessment Instruments
Curriculum Mapping
Resources
Questions/Discussion
Levels of Assessment
Levels of Assessment:
Level 1. Assessing individual student learning within courses
Level 2. Assessing individual student learning across courses
Level 3. Assessing courses
Level 4. Assessing programs
Level 5. Assessing the institution
(Miller & Leskes, 2005)
Definition of Assessment
Assessment is the systematic collection , review, and use of
information about educational programs undertaken for the purpose
of improving student learning and development.
(Palomba & Banta,1999)
Assessment Cycle
Identify
learning
goals and
outcomes
Select
assessment
instruments
& sources of
evidence
Implement
action (s)
based on
the results
Report
assessment
results
Establish
criteria
for
success
Gather and
analyze
data
Theoretical Framework
Theoretical Framework for MSU’s School of Graduate Studies
Assessment
based on:
•
Course Embedded Assessment model
•
The Lumina Foundation’s Degree Qualifications Profile (DQP)
The Degree Qualifications Profile model allows our graduate programs to frame clear expectations of what students should
be expected to know and be able to do at the associate, bachelor’s, and master’s level regardless of their academic
discipline.
The Degree Qualifications Profile (2011)
Five Areas of Learning:
1.Specialized Knowledge
2.Broad, Integrative Knowledge
3.Intellectual Skills





Analytic Inquiry
Use of Information Resources
Engaging Diverse Perspectives
Quantitative Fluency
Communication Fluency
4. Applied Learning
5. Civic Learning
Graduate Level Learning
•
•
How do I know graduate level learning when I see it?
What are its characteristics?
The Degree Profile provides guidance to the above questions.
Example:
Analytic inquiry
At the associate level, the student
Identifies, categorizes and distinguishes among elements of ideas, concepts, theories and/or practical approaches to
standard problems.
At the bachelor’s level, the student
Differentiates and evaluates theories and approaches to complex standard and non-standard problems within his or
her major field and at least one other academic field.
At the master’s level, the student
Disaggregates, adapts, reformulates and employs principal ideas, techniques or methods at the forefront of his or
her field of study in the context of an essay or project.
Programs at MSU’s School of Graduate Studies
Mission
The School of Graduate Studies believes in the transformative power of lifelong inquiry
to enhance lives. We seek to develop and maintain innovative graduate programs that foster
opportunities for personal and professional success in an increasingly multidisciplinary world.




Master of Health Science (GHSC),
Master of Arts or Science in Interdisciplinary Studies (GIDS),
Master of Criminal Justice Administration (GCJA), and
Master of Arts in Psychology (GPSY)
Master of Health Science (GHSC)
Program
Goals
 G1: Provide an affordable
and high quality graduate
health science program.
 G2: Foster student
achievement of core
competencies appropriate
for health science
professionals.
 G3: Prepare students to
Student Learning Outcomes
(program level)
1. Communication Skills: Students will be able to demonstrate the ability to create logical
oral and written arguments, explanations and reflections based on their individual or
collaborative efforts to both general and specialized audiences.
2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ
core disciplinary concepts, techniques or methods at the forefront of the health science
field in the context of an essay, a thesis or a project.
3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity:
Students will demonstrate knowledge, sensitivity and awareness of individual and cultural
diversity demonstrated through a project, paper or performance.
serve the community and
become leaders of change.
4. Values and Ethics: Students will assess and justify their engagement with respect to civic,
social and global responsibilities.
 G4: Prepare graduates for
5. Application of Knowledge and Skills in Health Science Professional Settings:
Students will be able to create a project, paper, exhibit, performance or other appropriate
artifacts reflecting the integration of knowledge acquired in a practicum, work community
and /or research activities with knowledge and /or skills gained from at least two academic
disciplines in different segments of the curriculum.
careers in health
administration and health
services related areas.
6. Qualitative and Mixed Methods Fluency: Students will demonstrate knowledge and
ability to interpret, evaluate design and implement qualitative and mix-methods research
appropriate to health science studies in a formal research document or by performance a
project.
Note: student learning outcomes (1,2,&5) are adopted from the Lumina Foundation’s
Degree Qualifications Profile
Master of Arts or Science in Interdisciplinary Studies (GIDS)
Program
Goals
 G1: Provide an affordable
and high quality
interdisciplinary studies
program.
 G2: Foster student
achievement of core
competencies appropriate
for graduate level learners
and professionals.
 G3: Prepare students to
serve the community and
become leaders of change.
 G4: Prepare graduates for
careers in
interdisciplinary fields
according to students’
academic concentrations.
Student Learning Outcomes
(program level)
1. Communication Skills: Students will be able to demonstrate the ability to create logical
oral and written arguments, explanations and reflections based on their individual or
collaborative efforts to both general and specialized audiences.
2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ
core disciplinary concepts, techniques or methods at the forefront of their field of study in
the context of an essay, a thesis or a project.
3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity:
Students will demonstrate knowledge, sensitivity and awareness of individual and cultural
diversity demonstrated through a project, paper or performance.
4. Broad, Integrative Knowledge: Students will articulate how their field (academic
concentration) has developed in relation to other major domains of inquiry and/or
practice.
5. Specialized Knowledge in an Academic Concentration Area: Students will be able
to elucidate major theories, research methods, and approaches to inquiry and/or schools
of practice in their field (academic concentration).
6. Research and Evaluation in Academic Concentration Area: Graduates will be able
to utilize the scientific approach to knowledge generation and apply the appropriate
research methods to practice.
Note: student learning outcomes (1,2,&5) are adopted from the Lumina Foundation’s
Degree Qualifications Profile
Master of Criminal Justice Administration (GCJA)
Program
Goals
 G1: Provide an affordable
and high quality graduate
criminal justice
administration program.
 G2: Foster student
achievement of core
competencies appropriate
for criminal justice
administration
professionals.
 G3: Prepare students to
serve the community and
become leaders of change.
 G4: Prepare graduates for
careers in criminal justice
administration field and
related areas.
Student Learning Outcomes
(program level)
1. Communication Skills: Students will be able to demonstrate the ability to create logical
oral and written arguments, explanations and reflections based on their individual or
collaborative efforts to both general and specialized audiences.
2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ
core disciplinary concepts, techniques or methods at the forefront of their field of study in
the context of an essay, a thesis or a project.
3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity:
Students will demonstrate knowledge, sensitivity and awareness of individual and cultural
diversity demonstrated through a project, paper or performance.
4. Broad, Integrative Knowledge: Students will articulate how criminal justice
administration field has developed in relation to other major domains of inquiry and/or
practice.
5. Engaging Diverse Perspectives: Students will address core issues in criminal justice
administration from the perspective of either a different point in time, culture, language,
political order, or technological context.
6. Application of Knowledge and Skills in Criminal Justice Administration
Professional Settings: Students will be able to create a project, paper, exhibit,
performance or other appropriate means reflecting the integration of knowledge acquired
in a practicum, work community and /or research activities with knowledge and /or skills
gained from at least two academic disciplines in different segments of the curriculum.
Note: student learning outcomes (1,2, 4,&6)are adopted from the Lumina Foundation’s
Degree Qualifications Profile
Master of Arts in Psychology (GPSY)
Program
Goals
 G1: Provide an affordable
and high quality graduate
psychology program.
 G2: Foster student
achievement of core
competencies appropriate
for mental health and
human services
professionals.
 G3: Prepare students to
serve the community and
become leaders of change.
 G4: Prepare graduates for
careers in mental health,
human services, and
related areas.
 G5: Prepare students for
doctoral and post
graduate studies in
psychology.
Student Learning Outcomes
(program level)
1. Communication Skills: Students will be able to demonstrate the ability to create logical
oral and written arguments, explanations and reflections based on their individual or
collaborative efforts to both general and specialized audiences.
2. Critical Thinking: Students will be able to deconstruct, adapt, reformulate and employ
core disciplinary concepts, techniques or methods at the forefront of their field of study in
the context of an essay, a thesis or a project.
3. Knowledge, Sensitivity, and Awareness of Individual and Cultural Diversity:
Students will demonstrate knowledge, sensitivity and awareness of individual and cultural
diversity demonstrated through a project, paper or performance.
4. Information Literacy: Students will be able to provide evidence (through papers,
projects, notebooks, computer files or catalogues) contributing to, expanding , assessing
and/or refining either a broadly recognized information resource or an information base
within the field of psychology.
5. Research and Evaluation in Psychology: Graduates will be able to utilize the scientific
approach to knowledge generation in psychology and apply the appropriate research
methods to practice.
6. Quantitative Fluency: Students will apply mathematical, formal logic and/or statistical
tools to problems appropriate to the field of psychology in a project, paper or
performance.
Note: student learning outcomes (1,2,4, & 6) are adopted from the Lumina Foundation’s
Degree Qualifications Profile
Competency-Based Assessment
Competency - based assessment measures an individual’s performance against a
predetermined criterion.
There are three levels of competency achievement throughout a program of study:
competency INTRODUCED (I)
competency PRACTICED (P)
competency REFINED (R)
Note: the competency achievement expectations are established by faculty in each course.
Competency-Based Assessment Chart
Are
competencies
achieved?
(Does the program do
what is intended?)
Evaluate student
work toward the
end of a
program in the
pre-selected
courses
Are competencies
REFINED (R) in
a course?
Yes
No
Evaluate student
work in the
middle of a
program in the
pre-selected
courses
Are competencies
INTRODUCED (I)
in a course?
Write
summary
report
Review
courses and
revise
curriculum
No
Yes
Evaluate student
work at the
beginning of a
program in the
pre-selected
courses
Write
summary
report
Write
summary
report
Review
courses and
revise
curriculum
Write
summary
report
Are
competencies
PRACTICED (P)
in a course?
No
Review
courses and
revise
curriculum
Write
summary
report
Yes
Write
summary
report
Curricular Alignment
Standards/Objectives
(MSU: Core Competencies)
Course
Embedded
Assessment
Assessments/Tests
(MSU: Rubrics)
Instructional Activities/
Materials
(MSU: Assignments)
Figure 1. Relationship among standards/objectives, instructional activities and materials, and assessments/tests. Adopted from
Anderson (2002) Curricular Alignment: A Re-Examination.
A signature assignment to measure achievement of student learning is required in
courses where assessment data are being collected for program level assessment.
Learning Outcome Assessment (2011-2012)
Three student learning outcomes (SLOs) are assessed in 2011-2012 cycle in all four graduate
programs:
1. Communication Skills
2. Critical Thinking
3. Cultural & Social Awareness
(Note: other student learning outcomes will be assessed in 2012-2013)
How are these outcomes assessed? By utilizing
direct measures:
•
•
Written Assignments, and
Theses or Projects
and
indirect measures:
•
•
Student Reflection Journals
Course Evaluations
Assessment Instruments
Rubrics are utilized to assess learning achievement.

Written Communication Skills Rubric

Critical Thinking Rubric

Cultural and Social Awareness Rubric
*Note: rubrics must meet graduate level learning expectations.
Student Learning Outcomes
Program level student learning outcomes must be reflected in individual courses.
Review your program’s outcomes and align them with your course outcomes.
All syllabi must include the following statement—
“Upon successful completion of this course, students will have the following
competencies…” [list competencies]
Student Learning Outcomes (example)
Cultural and
Social Awareness
(program level)
• Upon successful completion of this course, students
will have the following competencies:
• Students will be able to critically evaluate cultural and
socioeconomic differences and the significance of these
differences for personality development as evident in a
formal research paper.
course level
Critical
Thinking
• Upon successful completion of this course, students
will have the following competencies….
• (Give example)
(program level)
IMPORTANT: Student learning outcomes have to be measurable!
Curriculum Mapping
Curriculum maps
 Shows when, what, and on what level learning is delivered throughout the program.
Why curriculum mapping is important?
 Provides clarity and transparency of what is going on in the program and in specific
courses.
 Curriculum mapping is essential to program and course level assessment.
 Program level student learning outcomes should be closely aligned with course level
student learning outcomes and this alignment should be depicted in a curriculum map.
Master of Arts in Psychology Program (GPSY) Curriculum Map
Core Courses
GPSY 523 GPSY 545
Profession History &
al Ethics
Systems
GPSY 563
Research
Methods
GPSY 505
Statistics
GPSY 517
Adv.
Develop.
Psychology
GPSY 527
Social
Psychology
GPSY 520
Physiological
Psychology
GPSY 525
Cognitive
Psychology
GPSY 610
Adv.
Principles of
Learning
GPSY 620
Psychopathol
ogy
GPSY 630
Psychologic
al Testing
GPSY 670
Portfolio
GPSY 680
Thesis or
Project
P
P
P
A
P
P
P
P
R
R
A
P
P
P
P
A
P
P
P
P
R
R
A
P
P
P
P
P
A
P
P
P
P
R
R
A
I
A
P
A
R
R
A
R
R
A
Program
Outcomes
1.Communication
Skills
I
A
2.Critical Thinking
I
A
3.Information
Literacy
I
A
4..Quantitative
Fluency
5.Knowledge
sensitivity,
awareness of
individual and
cultural diversity
6.Research and
evaluation in
psychology
I
A
P
P
A
I
A
P
P
P
P
A
P
P
P
R
A
Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome;
A- data are collected and analyzed for program level assessment.
Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment.
Master of Health Science Program (GHSC) Curriculum Map (Concentration: Health Care Executive)
Core Courses
GHSC
501
Focus &
Design
GHSC 563
Research
Methods
GHSC 616
Health Care
Systems
GHCS 621
Health Care
Financial
Mngm
GHSC 622
The Health
Care
Executive
GHSC 503
GHSC 504
GHSC 580
GHSC 581
Behavior in
Transforming
Concentrat. Concentration
Organizations Organizational area and field area and field
Cultures
experience
experience
P
P
P
P
P
A
P
P
P
P
P
A
GHSC 582
Concentrat.
area and
field
experience
GHSC 583
Concentratio
area and
field
experience
GCJA 670
Portfolio
GCJA680
Thesis or
Project
P
P
R
R
A
R
R
A
P
R
R
A
R
R
R
A
P
R
R
A
R
R
A
Program
Outcomes
1.Communication
Skills
I
A
P
P
2.Critical Thinking
I
A
P
3.Application of
HSC
4.Values
&
Ethics
I
A
5.Knowledge
sensitivity, and
awareness of
individual and
cultural diversity
6.Qualitative and
Mixed Methods
Fluency
I
A
I
A
P
A
P
A
I
A
P
A
P
A
P
P
P
P
P
Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome;
A- data are collected and analyzed for program level assessment.
Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment.
Master of Health Science Program (GHSC) Curriculum Map (Concentration: Aging Studies)
Core Courses
GHSC 501
Focus &
Design
GHSC 563
Research
Methods
GHSC 511
Multidsp.
Aspects of
Aging
GHCS 515
GHSC 516
Psyc. Aspects Soc. Aspects
of Aging
of Aging
GHSC 572
Working
Across
Disabilities
GHSC 575
Public Policy
GHSC 580
Concentrat.
area and
field
experience
GHSC 581
Concentrat.
area and
field
experience
GHSC 582
Concentrat
area and
field
experience
GHSC 583
Concentrat
area and
field
experience
GCJA 670
Portfolio
GCJA680
Thesis or
Project
Program
Outcomes
1.Communica
tion Skills
2.Critical
Thinking
I
A
I
A
P
P
P
P
P
A
P
R
R
A
P
P
P
P
A
P
R
R
A
I
A
I
3.Application
of HSC
4.Values
&
Ethics
5.Knowledge
sensitivity,
and
awareness of
individual
and cultural
diversity
6.Qualitative
and Mixed
Methods
Research
Fluency
I
A
I
A
I
A
P
A
P
A
P
A
P
A
P
P
P
P
P
R
R
A
P
P
R
R
A
P
P
R
R
A
R
R
A
Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome;
A- data are collected and analyzed for program level assessment.
Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment.
MA/MS Interdisciplinary Studies Program (GIDS) Curriculum Map
Core Courses
GIDS 501
Focus &
Design
GIDS 550
Professional
Writing
I
A
P
I
A
P
GIDS 557
Interdisciplinary
Context and
Theory
GIDS 563
Research
Methods
GIDS 567
Statistical
Methods
GIDS 580
Concentration
area
GIDS 581
Concentration
area
GIDS 582
Concentration
area
GIDS 583
Concentration
area
GIDS 670
Portfolio
GIDS 680
Thesis or
Project
P
A
P
P
P
R
R
A
P
A
P
P
P
R
R
A
P
A
P
P
R
R
A
R
R
A
Program
Outcomes
1.Communication
Skills
2.Critical Thinking
Specialized
3.knowledge in an
academic
concentration
4.Broad
Integrative
Knowledge
(interdisciplinary)
5. Knowledge
sensitivity, and
awareness of
individual and
cultural diversity
6.Research and
evaluation in an
academic
concentration
P
I
A
P
R
I
A
P
A
I
A
P
A
P
P
P
R
R
A
P
A
P
P
P
R
R
A
I
A
Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome;
A- data are collected and analyzed for program level assessment.
Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment.
Master of Criminal Justice Administration Program (GCJA) Curriculum Map
Core Courses
GCJA 501
Focus &
Design
GCJA 531
History of
Criminal
Justice
GCJA 563
Research
Methods
GCJA 580
Concentration
area
GCJA 581
Concentration
area
GCJA 582
Concentration
area
GCJA 583
Concentration
area
I
A
P
P
P
A
P
P
P
P
A
P
I
A
P
A
I
A
P
A
I
P
P
A
I
A
P
A
GCJA 584
Concentration
area and field
research
GCJA 585
Concentration
area and field
research
GCJA 670
Portfolio
GCJA680
Thesis or
Project
P
R
R
A
P
P
R
R
A
P
P
P
R
R
A
P
P
P
R
R
A
R
R
A
R
R
A
Program
Outcomes
1.Communication
Skills
2.Critical Thinking
I
A
3. Engaging
Diverse
Perspectives
4.Broad Integrative
Knowledge
5.Application of
CJA
6. Knowledge
sensitivity, and
awareness of
individual and
cultural diversity
P
P
P
P
P
Letters: I – students are introduced to the outcome; P- students further develop the outcome; R- students can demonstrate a refinement of the outcome;
A- data are collected and analyzed for program level assessment.
Colors: orange- indicates outcomes assessed in 11-12 cycle; green- indicates courses where data are collected for program level assessment.
Summary
“ By acting on the calls for reform in assessment, graduate education can truly lead the
way in fulfilling its mission of excellence in teaching, research, and service to its
students and to the broader community.” (Lydell, 2008)
References
Association of American Colleges and Universities (AAC&U). Value Rubrics. Retrieved from
http://www.aacu.org/value/rubrics/index.cfm
Anderson, L.W. (2002). Curricular alignment: A re-examination. Theory into Practice, 41, (4). Retrieved from Academic
Search Premier.
Harden, R. M. (2001). AMEE Guide No 21: Curriculum mapping: a tool for transparent and
authentic teaching and learning. Med Tech, 23, (2).
Lydell, L. (2008). Assessing outcomes in graduate education. On the Horizon, 16, (2). Retrieved from
Proquest data base.
Lumina Foundation. (2011). The Degree Qualifications Profile. Retrieved from
http://www.luminafoundation.org/publications/The_Degree_Qualifications_Profile.pdf
Miller, R., & Leskes, A. (2005). Levels of assessment: From the student to the institution. Washington, DC:
AAC&U. Retrieved from http://www.aacu.org/pdf/LevelsOfAssessment.pdf
Questions/Discussion
For more information please contact:
Dr. William M. White, Dean
Office: (304) 929-1438;
Email: wwhite@mountainstate.edu
School of Graduate Studies
or
Indre Cuckler, M. A.
Psychology Program Specialist
Office: (304)-929-1401;
Email: icuckler@mountainstate.edu
School of Graduate Studies
Thank you !