Assessment in ITOM and Beyond:
Defining, Assessing, and Documenting
Student Learning Outcomes
GAIL WISAN
UNIVERSITY DIRECTOR OF ASSESSMENT
OFFICE OF INSTITUTIONAL EFFECTIVENESS AND
ANALYSIS
GWISAN@FAU.EDU
X71006
(561) 297-1006
MARC RHORER
ASSISTANT DEAN
COLLEGE OF BUSINESS
MRHORER@FAU.EDU
X70210
SEPTEMBER 2010
Overview :
Describe the Nature and Purposes of Assessment
Relate Assessment in ITOM to Larger Context
Explain the uses of a Curriculum Matrix and how it relates
to the Academic Learning Compact (ALC)
Writing better learning outcomes
Assessing and Documenting Student Learning Outcomes
Close the Assessment Loop: Identify the Do’s and Don’ts of
how you will USE your assessment results
Assessment in FAU ITOM Programs: The Context
1.
2.
3.
4.
5.
National: Department of Education and Public Opinion
Regional Context: SACS Accreditation
Florida Context: Academic Learning Compacts with
required annual reports to the state BOG on
continuous learning improvement
Institutional Context: Strategic Plan
The College of Business Context:
o
Core Curriculum
o
Business programs are accredited by The
Association to Advance Collegiate Schools
of Business (AACSB International)
Assessment in FAU ITOM Programs:
The SACS Accreditor Context
The Commission on Colleges of the Southern Association of
Colleges and Schools (SACS) is the recognized regional
accrediting body in the eleven U.S. Southern states (Alabama,
Florida, Georgia, Kentucky, Louisiana, Mississippi, North
Carolina, South Carolina, Tennessee, Texas and Virginia) and in
Latin America for those institutions of higher education that
award associate, baccalaureate, master's or doctoral degrees. It
is charged with carrying out the accreditation process.
SACS Accreditation (2012-2013): Need 3 years of good data on
learning outcomes assessment and continuous improvement.
o 2009-2010
o 2010-2011
o 2011-2012
Assessment in FAU ITOM Programs:
The SACS Accreditor Context
1.
Accreditation standards requires all
institutions of higher learning to establish
college level learning goals
2. Assess student achievement against those goals
3.
Provide evidence of student achievement
4.
Use results of assessment to improve student
learning outcomes
Assessment in ITOM and College of Business:
The AACSB Context
Accreditation standards requires that:
“The school specifies learning goals and
demonstrates achievement of learning goals
for key management-specific, and/or
appropriate discipline-specific knowledge and
skills that its student achieve in each
undergraduate degree program.”
Assessment in ITOM and College of Business:
The AACSB Context
Accreditation standards requires that:
“…the school specifies learning goals and
demonstrates achievement of learning goals in
each specialized master’s degree
program.”
The AACSB Context
AACSB, at the undergraduate level, is concerned about
“degree programs”
Academic Learning Compact (ALC) required by
Florida for each degree program and could be aligned
with AACSB /COB learning goals. (ALCs will have
some unique content and skills learning goals for
degree programs.)
The COB learning goals (AACSB) for the BBA/BS
program are universal across all majors
The AACSB Context
BBA/BS Learning Goals (approved 2006) – our
graduates will:
Demonstrate functional knowledge specified in the college core
Understand the impact of global economic and multicultural
issues as they related to business
Demonstrate the ability to utilize computing or
telecommunications technologies in business practice and
making business decisions
Demonstrate communication skills appropriate for a business
professional
The AACSB Context
BBA/BS Learning Goals (approved 2006) – our
graduates will:
Demonstrate analytical critical thinking
Understand ethical and legal implications of business decisions
and practices
Understand managerial and legal aspects of conducting
business in a diverse environment
The AACSB Context
Perhaps FAU COB learning goals need slight revision
or combining for conciseness – examples:
2 goals with similar/same skill of computer technology
2 goals with similar/same issue of understanding legal aspects
2 goals with similar/same issue of multiculturalism and
diversity
Several learning goals (AACSB) overlap Academic
Learning Compacts (BOG) – recommend
simplification and reduce duplication of effort
The AACSB Context
Example of concise learning goals from the
undergraduate degree program (Virginia Tech –
Pamplin COB)
Students will be effective communicators
Students will be analytical problem solvers
Students will be proficient with a wide range of relevant
business-related technology
Students will have business skills that include an international
emphasis
The AACSB Context
AACSB emphasizes assurance of learning (AOL) now,
as much as faculty sufficiency
Key areas of emphasis for AACSB in AOL
Actively engaged participants
Faculty involvement and understanding of the process
Assessment of learning, not faculty methods or teaching
Meaningful discussion of improvement among stakeholders,
NOT collection of data and the process of assessment methods
Alignment of learning goals, what faculty teach, and what
students experience/learn
Group work cannot be used for AOL assessment unless specific
contribution/authorship is attributed to each student
The AACSB Context
The essence of AACSB Assurance of Learning in 3
questions
What are students supposed to learn in your program?
How do you know that they are learning these?
What can you do to improve learning?
If these cannot be answered, there will be problems
with the reaffirmation of accreditation
The AACSB Context
Assurance of Learning research and Relationship to
Faculty Sufficiency
AACSB values and recognizes research in AOL
Research in student learning in business programs (AOL) is
applicable and valued by AACSB
Publications from assessment and AOL are applicable to
attainment / maintenance of AQ status for terminally qualified
(PHD) faculty
Journal outlets / examples
What Is Student Learning Assessment?
Assessment is a process in the learning cycle in
which faculty measure and document the degree to
which students are attaining defined (should be
valued and important) learning outcomes
Assessment is a student learning-centered and
faculty-guided approach to improving student
learning in the classroom and in degree programs
Defining Assessment:
The Three Steps of Assessment
1.
2.
3.
Articulate the program/course goals for student learning
Learning Outcomes: Clear Measurable Expected Outcomes
When completing the program, students will be able to ….
Systematically gather evidence about whether goals are
being met.
Where are the learning opportunities in the program?
Measuring outcomes: gathering, analyzing, & interpreting data
Use Information for Improving Student Learning
Outcomes.
Defining Assessment:
The Three Questions to Ask
1.
What are the program student learning outcome goals?
2.
3.
When completing the program, What should students know? What
should students be able to do? What skills should they have?
How will the program/course gather evidence about whether
students have achieved the expected learning outcomes?
In what courses should students have learned the skill? How was the
skill measured? Was it measured at different times in the program?
How will the program/class gathering, analyzing, & interpret results?
How will the program/course Use the Information gathered
to Improving Student Learning Outcomes?
What Assessment Isn’t
Assessment is not just giving grades
Assessment is not an evaluation of faculty
Assessment is not an infringement of
academic freedom
Assessment is not an invasion of student
privacy
Assessment is not punitive
The Value of Assessment: What It Does Best…
“Much of the value of assessment comes from the
systematic way it makes educators question, discuss,
share and observe.”
Palomba and Banta (1999). Assessment essentials: Planning,
implementing, and improving assessment in higher education. San
Francisco: Jossey-Bass.
The Purposes of Assessment:
Why Assess Student Learning?
To provide evidence of institutional effectiveness
To demonstrate the effectiveness of ITOM, College of
Business and FAU courses/programs to all
stakeholders (internal & external)
To know your students’ weakness and strengths and
use those results to improve overall quality of
teaching and learning within your program
To provide critical feedback to students and faculty
To provide students with clear statements about how
they can expect to improve skills and abilities
Types of Assessment: Indirect
Learning is subtle, often supporting direct methods
Grades not based on explicit criteria
Course evaluations
Student hours spent on active learning and/or service
learning
Focus group interviews with students
Employer or alumni surveys
Student perception surveys
Employer and internship supervisor ratings
Types of Assessment: Direct (Preferred)
Learning is tangible and visible
Course assignments
Examinations
Classroom Assessment Techniques (CATs)
Term papers and reports
Research projects
Case study analysis
Portfolios – body of work
Capstone projects
Licensure exams
ANALYTIC Rubrics Used to score performance
So, The Cycle Continues
Use…
Results…
To improve
teaching and
learning
Assess…
Student
achievement
against those
goals or outcomes
Plan…
The
Assessment
Cycle Provides
Feedback for
Continuous
Improvement
Establish
Learning Goals
Learning Outcomes
Implement…
Provide
Opportunities for
students to achieve
those goals
Why Aren’t Grades Sufficient?
Grades (A’s, B’s & C’s) in and of themselves do not
give you enough information about students’
strengths and weaknesses
Increasing emphasis is now being put on students:
thinking critically
acquiring life-long learning and business professional skills
acquiring social values
Assessment can build on to the grading process, but
grades alone do not provide the kind of feedback we
want
What are key Student Learning Outcomes in FAU’s
MIS B.B.A/B.S. Degrees and how are they assessed ?
?
Critical Thinking in MIS
Man. 4720: Students will demonstrate analytical
critical thinking through their ability to apply
relevant business models and/or strategic concepts
and tools in a research project on a firm and/or
industry. (ALC)
Fin. 3403: Students will apply financial math and
valuation principles to value securities, capital
projects and other assets. Students will interpret
financial information and use it in decision making.
(ALC)
Some Key Student Learning Outcomes in ITOM
Programs and their Assessment
Written Communication in ITOM
ENC 3213:Demonstrate Communication Skills appropriate to a
professional (basic level) (ALC)
ISM 4133 Adv. Design & Anal.: ALC (more advanced )
Oral Communication in ITOM
ISM 4133: The presenter effectively communicates the
organizations’ problem(s), solution(s), and methodologies used
to obtain solution(s). (Outcome stated in rubric provided in
ALC.)
Writing Better Learning Outcomes
DO
DON’T
•Better: Students will locate
information, evaluating its
validity and appropriateness
for a project.
•Too Vague: Students will demonstrate information
literacy skills.
•Too Specific: Students will be able to use a specific
institutional databases to demonstrate information
literacy.
•Better: Students will:
1.) accurately describe and
analyze two major
management approaches , and
2.) evaluate each perspectives
strengths and weaknesses.
•Too Vague: Students will demonstrate critical
thinking skills.
•Too Specific: Students will be able to describe the X
management perspective.
•Better: Students will write
clearly organized position
papers, providing financial
and spreadsheet support.
•Too Vague: Students will write proficiently.
•Too Specific: Students will write a five page paper
with no more than 3 grammatical errors.
Writing Better Learning Outcomes
DO
DON’T
•Better: Students will
systematically analyze and
solve problems, advocate and
defend their views, and refute
opposing views.
•Too Vague: Students will solve problems.
•Too Specific: Students will find a solution to
problem x.
•Better: Student teams will
analyze and evaluate case
studies about ethical business
dilemmas.
•Too Vague: Students will demonstrate ethics.
•Too Specific: Students will value the x point of view
of ethical business??? (arguable?)
•Better: Students will be able
to identify the major periods
in 19th and 20th century
European art, describing the
art styles and their cultural
context.
•Too Vague: Students will appreciate art.
•Too Specific: Students will be able to identify 7 of 8
artists from their paintings.
What is a Rubric and Why Use One?
A Systematic Scoring Guideline for
evaluating performance (e.g., case study
analysis, research paper, business plan, book
review, exam) via descriptions of criteria for
different levels of achievement
Rubrics provide structure for consistent and
unbiased grading
What are the 2 Main Types of Rubrics and their Use?
Holistic Rubrics provide a single score
based upon multiple factors for each level of
performance.
Analytic Rubrics provide multiple scores
based upon distinct achievement criteria/factors
from which they can then create a single score.
What are Holistic Rubrics?
Holistic Rubrics provide a single score for a
whole performance (e.g., research paper or case
study analysis is given one score) :
Excellent, Good, Fair, Poor or
Exceeds standards, Meets standards, Below
standards
Each Grading Level using MANY CRITERIA so
you cannot identify specific
strengths/weaknesses
Holistic Rubric’s Value and Purposes:
Easy to Grade
Quick Picture of Performance
Single Dimension is Sufficient for Grading
Helpful tool for calibrating grading
standards across multiple classes
Not useful for improving student learning
outcomes standards across multiple classes
What are Analytic Rubrics? Why Use?
Analytic Rubric provide specific feedback
on multiple criteria.
Value: Performance (e.g., research paper or
case study analysis) is assigned many scores on
key outcome criteria so that student and class
outcomes can be assessed as to strengths and
weaknesses.(See ITOM rubric in ALC.)
Assessment Grading uses MANY Specific
CRITERIA which can then be used to calculate
(automatically if desired) total grade.
Analytic Rubric’s Value and Purposes:
Detailed feedback to learners, professors, and programs
More detailed scoring can be used to guide improvement
Rubric, when provided to students, can help them identify key
factors in their performance and thus guide achievement
Useful feedback to faculty and the ITOM Program on students’
strengths and weaknesses on specific criteria
Assessment results can be used to modify learning opportunities
in the curriculum and improve the program and document
continuous improvement cycle in assessment.
Association of American Colleges and
Universities (AACU): VALUE Project
VALUE: Valid Assessment of Learning in Undergraduate
Education
Valid data needed to guide planning, learning, teaching, and improvement
Good Practice in assessment requires multiple assessments over time
http://www.aacu.org/value/rubrics
Developed Tested Rubrics
Critical Thinking
Problem Solving
Written Communication
Oral Communication
Others
Keep It Simple
Utilize the same student work (artifact) for various
assessments of a learning outcome
Example:
A Case Study Analytic and company or industry problem
solving paper on improving profits
Multiple outcomes assessed using this paper
Critical Thinking
Written Communication
Team Work
Oral Communication
Curriculum Outcomes Matrix: What is it?
Curriculum mapping is a method to align instruction
with desired goals and program outcomes. It can also
be used to explore what is taught and how.
The map or matrix:
Documents what is taught and when
Reveals gaps in the curriculum
Helps design an assessment plan
http://manoa.hawaii.edu/assessment/howto/mapping.
htm
Curriculum Outcomes Matrix: Benefits
Why do it? Benefits
Improves communication among faculty
Improves program coherence
Increases the likelihood that students achieve
program-level outcomes
Encourages reflective practice
http://manoa.hawaii.edu/assessment/howto/mapping.
htm
Assessment In Multiple Courses
Introduce the learning outcome skill (I)
Reinforce the learning outcome skill (R)
Master the learning outcome skill (M)
Department matrix : More Benefits
Process more important than product
Thinking in skills, not only content knowledge
Tool for constructing a degree program
building in redundancy
Plays to strengths of individual faculty
Each person need not cover all skills in a single course
Community and Communication within department
Working with matrix
Requires mutual intellectual respect
Requires willingness to find out what colleagues are doing and
why
Requires opening oneself up to similar scrutiny
e.g. “Why do you think students need to understand spreadsheet
applications?”
Helps Further Develop Sharing, Discussion and Community within the
Department
Additional Skills: Learning Outcomes for Matrix
Information Literacy skills
Statistical analysis skills (broken down in another
document to identify strengths and weaknesses)
Oral Presentation skill
Projection, Simulation and computer modeling skills
Specific language skills
Microsoft office skills
Problem Solving skills: Analysis, evaluation, creative
solutions
The Outcomes Matrix and Assessment
Observations of student success and difficulty in the
capstone course or other core courses
Use the skills/learning outcome categories on the
matrix to frame specific questions
to a class
in exit interviews/surveys
Formative assessment in various courses
How Can Assessment Results be Used to Improve
Student Learning Outcomes for program graduates.
Timetable
When you will start collecting the
assessment data?
How often you will collect it?
When will you plan to report on the
results?
Reporting on the Results
Now, What Are You Going To Do With All
These Good Data?
Collaborate with your departmental colleagues on the
findings
Determine how the results can be used to increase students’
learning and success
Put assessment results into action
Re-assess to measure the effectiveness of your
course/program modifications
Explaining your use of results
DO
DON’T
•DO focus on making specific improvements
based on faculty consensus.
•DON’T focus on simply planning for
improvements or making improvements
without faculty feedback.
•DO address specific program improvements
that will impact student learning.
•DON’T address assessment improvement
plans such as revising the rubric.
•DO use concrete ideas.
•DON’T write vague ideas.
•DO state strategies that are sustainable and
feasible.
•DON’T use strategies that are impossible to
complete in one year considering your
resources.
•DO use strategies that can improve the
curriculum and also help improve student
learning outcomes (e.g. solve problems).
•DON’T focus only on improving the
curriculum.
Suggestions
Review and Revise Academic Learning Compact
Coordinate Assessment Plans with ALC
Set priorities as a department
“Hand off” specific learning outcomes assessment to
course/professor best suited to task but share and use
results
Consider using Analytic Rubrics
Be clear as to how assessment results will be used to
revise pedagogy, curriculum matrix, and
teaching/learning strategies in program/courses
Conclusions?
Align Assessment Activities to make them Useful
Don’t duplicate work: Align the work
Create An Outcomes Curriculum Matrix
Align assessment for AACSB, ALC, and FAU
Assessment Database
Document and Report Results
Identify Specific ways in which assessment results will
be shared
Identify Specif ways to USE data to improve learning
The Flow of the Assurance of Learning - AACSB
Questions?
ASSESSMENT IS A TOOL TO BRING
About
better teaching and learning!
www.fau.edu/iea/assessment
gwisan@fau.edu
Appendix
ITOM ALC on remaining slides
B.B.A. & B.S. Management Information Systems
College of Business
Dep’t. of Information Technology And Operations Management
Rubric
0
