Implementing Expected Learning Outcomes:
BYU 2006-07 Academic Year Goals
„ Successfully meet by fall 2007, Northwest Commission on Colleges and Universities (NWCCU)
accreditation requirements to:
1. Undertake a systematic process of identifying and publishing expected learning outcomes for
each BYU degree program;
2. Through regular and systematic assessment, demonstrate students who complete their programs
have achieved these stated outcomes, using the full range of appropriate primary and secondary
sources of evidence;
3. Provide evidence consistently across its programs that results of its assessment activities leads
to improvement of learning and teaching.
4. Prepare a focused interim report and host a Commission Representative in fall 2007
(Bold and italics in NWCCU original)
„ Using Learningoutcomes.byu.edu, each BYU student and BYU faculty member will be able to
demonstrate that they can:
1. Locate and explain their degree program’s expected learning outcomes;
2. Articulate evidence (direct and indirect) how they achieve and document their degree program’s
stated outcomes; and
3. Show results of ongoing assessment activities lead to the improvement of their learning and
teaching.
1
1.
Undertake a systematic process of identifying and publishing expected
learning outcomes for each BYU degree program
We hear an increasingly strident and persistent voice in contemporary society calling for greater
accountability in higher education, particularly in regard to what students can expect to learn when they
select a particular academic program offered by a particular university.1 There is a strong movement
within higher education to shift from an “instructional focus” to a “learning focus.” It is, thus, not
surprising that disciplinary and institutional accreditation bodies are placing greater emphasis on
learning outcomes. At BYU, in conjunction with the April 2006 visit of the NWCCU accreditation
team, program-level learning goals for all 415 degree programs were articulated. During August 2006
this information was published via a university website (http://learningoutcomes.byu.edu). This
website will continue to serve as the university’s repository for learning outcomes-related
documentation specified in NWCCU’s three-part charge, on the preceding page. The purpose of this
document is to help programs fully comply with this charge and to realize the benefits of more closely
tracking student learning at the program level.
Program-Level Expected Learning Outcomes2
Expected learning outcomes specify what the departmental faculty expect from their program
graduates—the learning they can demonstrate; the learning that distinguishes a student graduating from
a particular BYU academic program. These expectations will vary depending on the academic
discipline. For example, programs may require their graduates to master mathematical tools, certain
dance routines, the use of proper protocols for diagnosing illnesses, techniques for teaching foreign
languages, or ways of interpreting written, visual, or auditory information. Learning outcomes should
reflect the faculty’s deepest convictions regarding “what’s most important” for students to learn, not
“what we can easily measure.” This is particularly relevant for discussions about program learning
outcomes on this campus because program learning outcomes should contribute to the Aims of a BYU
Education. These Aims describe the ideal attributes of a BYU graduate, not just what they can do in
their chosen field. It is true that learning outcomes need to be assessable in some way, but this doesn’t
necessarily mean reducing outcomes to a score or number on a rating scale. (This subject will be
explored in more detail in the following section.) The following graphic depicts the important “linking
pin” role played by program-level learning outcomes. For more information on linking institutional,
program and course learning outcomes see FAQ Attachments 2-4.
Course
outcomes
contribute to
program
outcomes
Program
A
Program
B
Course
outcomes
reflect
program
outcomes
Program Outcomes
Outcomes 1
Outcomes 2
Outcomes 3
Outcomes …
Outcomes 1
Outcomes 2
Outcomes 3
Outcomes …
Outcomes 1
Outcomes 2
Outcomes 3
Outcomes …
Program
C
Course Outcomes
Course A
Course B
Course C
Course …
Course …
Course …
Course …
Course …
Course …
Course …
Course …
Course …
Program
outcomes
reflect
university
outcomes
Program
outcomes
contribute to
university
outcomes
University Outcomes
Outcomes 1
Outcomes 2
Outcomes 3
Outcomes …
Outcomes …
Outcomes …
Outcomes …
Outcomes …
Outcomes …
Outcomes …
Outcomes …
Outcomes …
2
The Aims of a BYU Education highlights four broad expected outcomes, encompassing what is
commonly referred to as the BYU experience. In the process of designing the BYU alumni survey a
faculty committee broke the four Aims into 24 measurable constructs. These, and their associated
survey items, might help inform discussions of Aims-related program learning outcomes.
Aims of a BYU Education
(EXPECTED)
A BYU education should be
1.
2.
3.
4.
Spiritually strengthening
Intellectually enlarging
Character building, leading to
Lifelong learning and service
BYU Mission and Aims Constructs
Spiritually Strengthening
1. Relationship with God and Religious Identity
Intellectually Enlarging
2. Thinking Habits
3. Thinking Skills
4. Quantitative Reasoning
5. Listening Skill
6. Speaking/Presentation Skills
7. Writing Skills
8. Reading Skills
9. Possesses Historical Perspective
10. Understands, Uses, and Appreciates Science
11. Understands and Enjoys Excellence In The Visual And Performing Arts
12. Understands and Appreciates Literature
13. Possesses Informed Awareness of Peoples, Cultures, Languages And Nations
14. Confidence/Competence In Major
15. Competence in Studying, Interpreting, and Applying Scripture
16. Understands The Doctrines And History of the Church
Character Building
17. Strives to Develop Christ-like Character and Live a Christ-like Life
18. Views the World Through an Eternal Perspective
19. Values Physical, Emotional, and Mental Health
20. Maintains Healthy Relationships with Others
Lifelong Learning and Service
21. Possesses the Desire and Skills Needed for Life-long Learning
22. Uses Technology Effectively
23. Church Service
24. Community Service and Involvement
3
Guidelines for Writing Program-Level Expected Learning Outcomes
Leading Questions
Questions to consider: What is most important for students to learn in this program? What should
graduates of this program be able to do? How do this program’s learning outcomes reflect the Aims of
a BYU Education and requirements of applicable accrediting bodies? Additional questions for
stimulating conversations with faculty and students are included as FAQ Attachment 5.
Tips: Use action verbs** to express what students will do, e.g., analyze, create, compare, evaluate. (The
use of action verbs facilitates alignment of program and course learning outcomes and assessments.)
When writing program learning outcomes, anticipate how student learning will be assessed in relation
to each expectation.
General format: Program graduates will be able to (verb + ________________________________).
Example 1: Program graduates will be able to assess their own strengths and weaknesses and adjust
future performance in light of their self-assessments.
Example 2: Program graduates will be able to understand, interpret, explain, analyze, and assess
representative philosophical texts, teachings, and problems.
Example 3: Program graduates will be able to design, provide, and evaluate healthcare for individuals
and groups, including attention to human responses in actual and potential health problems.
Example 4: Program graduates will be able to effectively discuss important issues in sociology in light
of the gospel of Jesus Christ.
** Verbs to consider when writing learning outcomes include, but are not limited to: analyze, dissect, detect, test,
deconstruct, discriminate, distinguish, examine, focus, find coherence, survey, compare, contrast, classify, investigate,
outline, separate, structure, categorize, solve, diagram, determine evidence and conclusions, judge, calculate, criticize,
debate, experiment, identify, illustrate, infer, inspect, inventory, question, relate, select, coordinate, select, decide, rate,
debate, evaluate, justify, recommend, verify, monitor, test measure, appraise, assess, compare, conclude, contrast, plan,
criticize, discriminate, estimate, explain, interpret, relate, revise, score, summarize, support, value, create, hypothesize,
design, construct, invent, imagine, discover, present, deduce, induce, bring together, compose, pretend, predict, solve,
organize, modify, improve, suppose, produce, set up, propose, formulate, arrange, assemble, categorize, collect, revise,
combine, devise, explain, generate, manage, perform, prepare, rearrange, reconstruct, argue for, relate, reorganize.
4
2.
Through regular and systematic assessment, demonstrate students who
complete their programs have achieved these stated outcomes, using the
full range of appropriate primary and secondary sources of evidence
Assessing Learning Outcomes2
There are many ways to assess learning outcomes—student learning. The characteristics of good
evidence includes a) both direct and indirect methods gathering data, and b) the appropriate use of
quantitative and qualitative data. It is also good to understand the fundamental concepts of formative
assessment, summative assessment, and benchmarking. It is good to have convergence from multiple
sources and types of data when substantiating whether or not stated learning outcomes have been
realized.
The concepts of direct and indirect methods of evaluating student learning are often confused with each
other and with quantitative and qualitative forms of information. Each has its merits and drawbacks.
Direct evidence—demonstrates actual learning, performance based work that is reflected in an
evaluation of something the student has actually done or produced (e.g. homework, tests, papers,
standardized tests, performances, products, etc.).
Indirect evidence—non-performance based data that reveals characteristics associated with learning,
but only implies that learning has occurred (e.g . self report survey data, senior survey data, alumni
questionnaire data, job placement rates, satisfaction rates, etc.).
Examples of Direct and Indirect Measures of Student Learning at the Program Level
Program
(Degree)
Level
Direct (Primary) Measures
• Capstone projects, senior
theses, exhibits, or
performances
• Pass rates or scores on
licensure, certification, or
subject area tests (Educational
Testing Services)
• Student publications or
conference presentations
• Employer and internship
supervisor rating or students’
performance
• GRE scores of students
completing the program
• Performance-based skills
competency exams
• Portfolios mapped to intended
outcomes with imbedded
rubric (criterion-based rating
scale) ratings
Indirect (Secondary) Measures
• BYU Alumni questionnaire
data
• BYU Senior survey data
• National Survey of Student
Engagement (NSSE) data
• Employer surveys (nonsupervisor)
• Graduate school placement
rates
• Departmental exit interviews
• Portfolios with no
performance based linkage to
intended outcomes
• Focus group interviews with
students, faculty, employers,
etc.
5
Questions to consider: How will each learning outcome be assessed; what evidence of student learning
is most relevant for each learning outcome? What criteria will be used to evaluate this evidence? In
other words, what constitutes student performance at an exceptional, acceptable, and unacceptable
level?
Tips: Develop a strategy to systematically and routinely collect, store, and retrieve assessment evidence
pertaining to each learning outcome. Use multiple kinds of assessments for each learning outcome
including: direct outcomes, such as outcome-related course assignments, exams, and capstone course
projects, as well as indirect outcomes, such as employer, alumni and senior surveys. (For additional
information on assessing learning outcomes see FAQ Attachments 6-9).
General format: Student achievement of this learning outcome is assessed by __________________
and _________________.
Example 1: Student achievement of this learning outcome is assessed [directly] by instructor
evaluations and [indirectly] by student self-assessments and acceptance into leading graduate programs.
Example 2: Student achievement of this learning outcome is assessed [directly] by reviewing samples
of student work (i.e., essay examinations, capstone projects) and [indirectly] by employer surveys and
the National Survey of Student Engagement (NSSE).
Example 3: Student achievement of this learning outcome is assessed [directly] by student scores on
the national proficiency exam (CPA) and [indirectly] by program review data and job placement rates.
Example 4: Student achievement of this learning outcome is assessed [directly] by sample student
portfolios that reflect the students’ overall work in the program and [indirectly] by senior surveys.
3.
Provide evidence consistently across its programs that results of its
assessment activities leads to improvement of learning and teaching
A commitment to the assessment of student learning requires a parallel commitment to ensuring its
use.3,4 Perhaps the most difficult part of assessing student learning is the process of effecting change in
teaching and learning as a result of information gained through assessment practices. It is pointless
simply to only “do assessment”; the results of assessment activities should come full circle to have a
direct impact on teaching and learning and the institution’s strategic plan to fulfill its mission2.
Changes in programmatic curricula as a result of assessment data do not happen automatically, as many
can attest. However, if plans at the department level outline specific procedures for examining
assessment results and implementing curricular revision, those changes are more likely to occur.
Assessment results need to be discussed with key stakeholders to formulate the best possible strategies
for narrowing observed gaps between stated expectations and student performance. It is hoped that
these discussions will also be informed by contemporary research on the predictors of student learning.
For more information on this literature, see FAQ Attachments 10, 11.
6
Following is a model that encompasses all three components of the NWCCU charge. This table could
be used to both guide and document each program review cycle.
1. Undertake a
systematic process
of identifying and
publishing
expected learning
outcomes for each
BYU degree
program
2. Through regular and
systematic assessment,
demonstrate students who
complete their programs
have achieved these stated
outcomes, using the full
range of appropriate primary
and secondary sources of
evidence
3. Provide evidence
consistently across its
programs that results of
its assessment activities
leads to improvement of
learning and teaching
Degree Program
Expected Learning
Outcomes
1. Program goal A
2. Program goal B
3. Program goal …
Evidence of Achieved
Stated Outcomes
Direct
Indirect
Assessment & Actions Taken for
Improved Teaching, Learning
Program Review Cycle
References
1. Association of American Colleges and Universities (2004). Taking Responsibility for the Quality of
the Baccalaureate Degree. Washington, D.C.
2. Middle States Commission on Higher Education (2003). Student Learning Assessment: Options and
Resources. Philadelphia.
3. Miller, R. & Leskes, A. (2005). Levels of Assessment from the Student to the Institution. Washington,
D.C.: Association of American Colleges and Universities.
4. Nichols, J. & Nichols, K. (2000) The Departmental Guide and Record Book for Student Outcomes
Assessment and Institutional Effectiveness. Agathon Press, New York
7