Assessment in ITOM and Beyond:
Defining, Assessing, and Documenting
Student Learning Outcomes
GAIL WISAN
UNIVERSITY DIRECTOR OF ASSESSMENT
OFFICE OF INSTITUTIONAL EFFECTIVENESS AND
ANALYSIS
GWISAN@FAU.EDU
X71006
(561) 297-1006
MARC RHORER
ASSISTANT DEAN
COLLEGE OF BUSINESS
MRHORER@FAU.EDU
X70210
SEPTEMBER 2010
Overview :
 Describe the Nature and Purposes of Assessment
 Relate Assessment in ITOM to Larger Context
 Explain the uses of a Curriculum Matrix and how it relates
to the Academic Learning Compact (ALC)
 Writing better learning outcomes
 Assessing and Documenting Student Learning Outcomes
 Close the Assessment Loop: Identify the Do’s and Don’ts of
how you will USE your assessment results
Assessment in FAU ITOM Programs: The Context
1.
2.
3.
4.
5.
National: Department of Education and Public Opinion
Regional Context: SACS Accreditation
Florida Context: Academic Learning Compacts with
required annual reports to the state BOG on
continuous learning improvement
Institutional Context: Strategic Plan
The College of Business Context:
o
Core Curriculum
o
Business programs are accredited by The
Association to Advance Collegiate Schools
of Business (AACSB International)
Assessment in FAU ITOM Programs:
The SACS Accreditor Context
 The Commission on Colleges of the Southern Association of
Colleges and Schools (SACS) is the recognized regional
accrediting body in the eleven U.S. Southern states (Alabama,
Florida, Georgia, Kentucky, Louisiana, Mississippi, North
Carolina, South Carolina, Tennessee, Texas and Virginia) and in
Latin America for those institutions of higher education that
award associate, baccalaureate, master's or doctoral degrees. It
is charged with carrying out the accreditation process.
 SACS Accreditation (2012-2013): Need 3 years of good data on
learning outcomes assessment and continuous improvement.
o 2009-2010
o 2010-2011
o 2011-2012
Assessment in FAU ITOM Programs:
The SACS Accreditor Context
1.
Accreditation standards requires all
institutions of higher learning to establish
college level learning goals
2. Assess student achievement against those goals
3.
Provide evidence of student achievement
4.
Use results of assessment to improve student
learning outcomes
Assessment in ITOM and College of Business:
The AACSB Context
Accreditation standards requires that:
“The school specifies learning goals and
demonstrates achievement of learning goals
for key management-specific, and/or
appropriate discipline-specific knowledge and
skills that its student achieve in each
undergraduate degree program.”
Assessment in ITOM and College of Business:
The AACSB Context
Accreditation standards requires that:
“…the school specifies learning goals and
demonstrates achievement of learning goals in
each specialized master’s degree
program.”
The AACSB Context
 AACSB, at the undergraduate level, is concerned about
“degree programs”
 Academic Learning Compact (ALC) required by
Florida for each degree program and could be aligned
with AACSB /COB learning goals. (ALCs will have
some unique content and skills learning goals for
degree programs.)
 The COB learning goals (AACSB) for the BBA/BS
program are universal across all majors
The AACSB Context
 BBA/BS Learning Goals (approved 2006) – our
graduates will:




Demonstrate functional knowledge specified in the college core
Understand the impact of global economic and multicultural
issues as they related to business
Demonstrate the ability to utilize computing or
telecommunications technologies in business practice and
making business decisions
Demonstrate communication skills appropriate for a business
professional
The AACSB Context
 BBA/BS Learning Goals (approved 2006) – our
graduates will:



Demonstrate analytical critical thinking
Understand ethical and legal implications of business decisions
and practices
Understand managerial and legal aspects of conducting
business in a diverse environment
The AACSB Context
 Perhaps FAU COB learning goals need slight revision
or combining for conciseness – examples:



2 goals with similar/same skill of computer technology
2 goals with similar/same issue of understanding legal aspects
2 goals with similar/same issue of multiculturalism and
diversity
 Several learning goals (AACSB) overlap Academic
Learning Compacts (BOG) – recommend
simplification and reduce duplication of effort
The AACSB Context
 Example of concise learning goals from the
undergraduate degree program (Virginia Tech –
Pamplin COB)




Students will be effective communicators
Students will be analytical problem solvers
Students will be proficient with a wide range of relevant
business-related technology
Students will have business skills that include an international
emphasis
The AACSB Context
 AACSB emphasizes assurance of learning (AOL) now,
as much as faculty sufficiency
 Key areas of emphasis for AACSB in AOL






Actively engaged participants
Faculty involvement and understanding of the process
Assessment of learning, not faculty methods or teaching
Meaningful discussion of improvement among stakeholders,
NOT collection of data and the process of assessment methods
Alignment of learning goals, what faculty teach, and what
students experience/learn
Group work cannot be used for AOL assessment unless specific
contribution/authorship is attributed to each student
The AACSB Context
 The essence of AACSB Assurance of Learning in 3
questions



What are students supposed to learn in your program?
How do you know that they are learning these?
What can you do to improve learning?
 If these cannot be answered, there will be problems
with the reaffirmation of accreditation
The AACSB Context
 Assurance of Learning research and Relationship to
Faculty Sufficiency




AACSB values and recognizes research in AOL
Research in student learning in business programs (AOL) is
applicable and valued by AACSB
Publications from assessment and AOL are applicable to
attainment / maintenance of AQ status for terminally qualified
(PHD) faculty
Journal outlets / examples
What Is Student Learning Assessment?
 Assessment is a process in the learning cycle in
which faculty measure and document the degree to
which students are attaining defined (should be
valued and important) learning outcomes
 Assessment is a student learning-centered and
faculty-guided approach to improving student
learning in the classroom and in degree programs
Defining Assessment:
The Three Steps of Assessment
1.
2.
3.
Articulate the program/course goals for student learning

Learning Outcomes: Clear Measurable Expected Outcomes

When completing the program, students will be able to ….
Systematically gather evidence about whether goals are
being met.

Where are the learning opportunities in the program?

Measuring outcomes: gathering, analyzing, & interpreting data
Use Information for Improving Student Learning
Outcomes.
Defining Assessment:
The Three Questions to Ask
1.
What are the program student learning outcome goals?

2.
3.
When completing the program, What should students know? What
should students be able to do? What skills should they have?
How will the program/course gather evidence about whether
students have achieved the expected learning outcomes?

In what courses should students have learned the skill? How was the
skill measured? Was it measured at different times in the program?

How will the program/class gathering, analyzing, & interpret results?
How will the program/course Use the Information gathered
to Improving Student Learning Outcomes?
What Assessment Isn’t
 Assessment is not just giving grades
 Assessment is not an evaluation of faculty
 Assessment is not an infringement of
academic freedom
 Assessment is not an invasion of student
privacy
 Assessment is not punitive
The Value of Assessment: What It Does Best…
“Much of the value of assessment comes from the
systematic way it makes educators question, discuss,
share and observe.”
Palomba and Banta (1999). Assessment essentials: Planning,
implementing, and improving assessment in higher education. San
Francisco: Jossey-Bass.
The Purposes of Assessment:
Why Assess Student Learning?
 To provide evidence of institutional effectiveness
 To demonstrate the effectiveness of ITOM, College of
Business and FAU courses/programs to all
stakeholders (internal & external)
 To know your students’ weakness and strengths and
use those results to improve overall quality of
teaching and learning within your program
 To provide critical feedback to students and faculty
 To provide students with clear statements about how
they can expect to improve skills and abilities
Types of Assessment: Indirect
 Learning is subtle, often supporting direct methods
 Grades not based on explicit criteria
 Course evaluations
 Student hours spent on active learning and/or service
learning
 Focus group interviews with students
 Employer or alumni surveys
 Student perception surveys
 Employer and internship supervisor ratings
Types of Assessment: Direct (Preferred)
 Learning is tangible and visible
 Course assignments
 Examinations
 Classroom Assessment Techniques (CATs)
 Term papers and reports
 Research projects
 Case study analysis
 Portfolios – body of work
 Capstone projects
 Licensure exams
ANALYTIC Rubrics Used to score performance
So, The Cycle Continues
Use…
Results…
To improve
teaching and
learning
Assess…
Student
achievement
against those
goals or outcomes
Plan…
The
Assessment
Cycle Provides
Feedback for
Continuous
Improvement
Establish
Learning Goals
Learning Outcomes
Implement…
Provide
Opportunities for
students to achieve
those goals
Why Aren’t Grades Sufficient?
 Grades (A’s, B’s & C’s) in and of themselves do not
give you enough information about students’
strengths and weaknesses
 Increasing emphasis is now being put on students:



thinking critically
acquiring life-long learning and business professional skills
acquiring social values
 Assessment can build on to the grading process, but
grades alone do not provide the kind of feedback we
want
What are key Student Learning Outcomes in FAU’s
MIS B.B.A/B.S. Degrees and how are they assessed ?
?
 Critical Thinking in MIS
Man. 4720: Students will demonstrate analytical
critical thinking through their ability to apply
relevant business models and/or strategic concepts
and tools in a research project on a firm and/or
industry. (ALC)
Fin. 3403: Students will apply financial math and
valuation principles to value securities, capital
projects and other assets. Students will interpret
financial information and use it in decision making.
(ALC)
Some Key Student Learning Outcomes in ITOM
Programs and their Assessment
 Written Communication in ITOM
ENC 3213:Demonstrate Communication Skills appropriate to a
professional (basic level) (ALC)
ISM 4133 Adv. Design & Anal.: ALC (more advanced )
 Oral Communication in ITOM
ISM 4133: The presenter effectively communicates the
organizations’ problem(s), solution(s), and methodologies used
to obtain solution(s). (Outcome stated in rubric provided in
ALC.)
Writing Better Learning Outcomes
DO
DON’T
•Better: Students will locate
information, evaluating its
validity and appropriateness
for a project.
•Too Vague: Students will demonstrate information
literacy skills.
•Too Specific: Students will be able to use a specific
institutional databases to demonstrate information
literacy.
•Better: Students will:
1.) accurately describe and
analyze two major
management approaches , and
2.) evaluate each perspectives
strengths and weaknesses.
•Too Vague: Students will demonstrate critical
thinking skills.
•Too Specific: Students will be able to describe the X
management perspective.
•Better: Students will write
clearly organized position
papers, providing financial
and spreadsheet support.
•Too Vague: Students will write proficiently.
•Too Specific: Students will write a five page paper
with no more than 3 grammatical errors.
Writing Better Learning Outcomes
DO
DON’T
•Better: Students will
systematically analyze and
solve problems, advocate and
defend their views, and refute
opposing views.
•Too Vague: Students will solve problems.
•Too Specific: Students will find a solution to
problem x.
•Better: Student teams will
analyze and evaluate case
studies about ethical business
dilemmas.
•Too Vague: Students will demonstrate ethics.
•Too Specific: Students will value the x point of view
of ethical business??? (arguable?)
•Better: Students will be able
to identify the major periods
in 19th and 20th century
European art, describing the
art styles and their cultural
context.
•Too Vague: Students will appreciate art.
•Too Specific: Students will be able to identify 7 of 8
artists from their paintings.
What is a Rubric and Why Use One?
 A Systematic Scoring Guideline for
evaluating performance (e.g., case study
analysis, research paper, business plan, book
review, exam) via descriptions of criteria for
different levels of achievement
 Rubrics provide structure for consistent and
unbiased grading
What are the 2 Main Types of Rubrics and their Use?
 Holistic Rubrics provide a single score
based upon multiple factors for each level of
performance.
 Analytic Rubrics provide multiple scores
based upon distinct achievement criteria/factors
from which they can then create a single score.
What are Holistic Rubrics?
 Holistic Rubrics provide a single score for a
whole performance (e.g., research paper or case
study analysis is given one score) :
Excellent, Good, Fair, Poor or
Exceeds standards, Meets standards, Below
standards
 Each Grading Level using MANY CRITERIA so
you cannot identify specific
strengths/weaknesses
Holistic Rubric’s Value and Purposes:
 Easy to Grade
 Quick Picture of Performance
 Single Dimension is Sufficient for Grading
 Helpful tool for calibrating grading
standards across multiple classes
 Not useful for improving student learning
outcomes standards across multiple classes
What are Analytic Rubrics? Why Use?
 Analytic Rubric provide specific feedback
on multiple criteria.
 Value: Performance (e.g., research paper or
case study analysis) is assigned many scores on
key outcome criteria so that student and class
outcomes can be assessed as to strengths and
weaknesses.(See ITOM rubric in ALC.)
 Assessment Grading uses MANY Specific
CRITERIA which can then be used to calculate
(automatically if desired) total grade.
Analytic Rubric’s Value and Purposes:
 Detailed feedback to learners, professors, and programs
 More detailed scoring can be used to guide improvement
 Rubric, when provided to students, can help them identify key
factors in their performance and thus guide achievement
 Useful feedback to faculty and the ITOM Program on students’
strengths and weaknesses on specific criteria
 Assessment results can be used to modify learning opportunities
in the curriculum and improve the program and document
continuous improvement cycle in assessment.
Association of American Colleges and
Universities (AACU): VALUE Project
 VALUE: Valid Assessment of Learning in Undergraduate
Education



Valid data needed to guide planning, learning, teaching, and improvement
Good Practice in assessment requires multiple assessments over time
http://www.aacu.org/value/rubrics
 Developed Tested Rubrics





Critical Thinking
Problem Solving
Written Communication
Oral Communication
Others
Keep It Simple
 Utilize the same student work (artifact) for various
assessments of a learning outcome
 Example:

A Case Study Analytic and company or industry problem
solving paper on improving profits
 Multiple outcomes assessed using this paper
 Critical Thinking
 Written Communication
 Team Work
 Oral Communication
Curriculum Outcomes Matrix: What is it?
 Curriculum mapping is a method to align instruction
with desired goals and program outcomes. It can also
be used to explore what is taught and how.
 The map or matrix:

Documents what is taught and when

Reveals gaps in the curriculum

Helps design an assessment plan
http://manoa.hawaii.edu/assessment/howto/mapping.
htm
Curriculum Outcomes Matrix: Benefits
 Why do it? Benefits
 Improves communication among faculty
 Improves program coherence
 Increases the likelihood that students achieve
program-level outcomes
 Encourages reflective practice
http://manoa.hawaii.edu/assessment/howto/mapping.
htm
Assessment In Multiple Courses
 Introduce the learning outcome skill (I)
 Reinforce the learning outcome skill (R)
 Master the learning outcome skill (M)
Department matrix : More Benefits
 Process more important than product
 Thinking in skills, not only content knowledge
 Tool for constructing a degree program

building in redundancy
 Plays to strengths of individual faculty

Each person need not cover all skills in a single course
Community and Communication within department
 Working with matrix
 Requires mutual intellectual respect
 Requires willingness to find out what colleagues are doing and
why
 Requires opening oneself up to similar scrutiny
e.g. “Why do you think students need to understand spreadsheet
applications?”
 Helps Further Develop Sharing, Discussion and Community within the
Department

Additional Skills: Learning Outcomes for Matrix
 Information Literacy skills
 Statistical analysis skills (broken down in another





document to identify strengths and weaknesses)
Oral Presentation skill
Projection, Simulation and computer modeling skills
Specific language skills
Microsoft office skills
Problem Solving skills: Analysis, evaluation, creative
solutions
The Outcomes Matrix and Assessment
 Observations of student success and difficulty in the
capstone course or other core courses
 Use the skills/learning outcome categories on the
matrix to frame specific questions



to a class
in exit interviews/surveys
Formative assessment in various courses
 How Can Assessment Results be Used to Improve
Student Learning Outcomes for program graduates.
Timetable
When you will start collecting the
assessment data?
How often you will collect it?
When will you plan to report on the
results?
Reporting on the Results
 Now, What Are You Going To Do With All
These Good Data?

Collaborate with your departmental colleagues on the
findings

Determine how the results can be used to increase students’
learning and success

Put assessment results into action

Re-assess to measure the effectiveness of your
course/program modifications
Explaining your use of results
DO
DON’T
•DO focus on making specific improvements
based on faculty consensus.
•DON’T focus on simply planning for
improvements or making improvements
without faculty feedback.
•DO address specific program improvements
that will impact student learning.
•DON’T address assessment improvement
plans such as revising the rubric.
•DO use concrete ideas.
•DON’T write vague ideas.
•DO state strategies that are sustainable and
feasible.
•DON’T use strategies that are impossible to
complete in one year considering your
resources.
•DO use strategies that can improve the
curriculum and also help improve student
learning outcomes (e.g. solve problems).
•DON’T focus only on improving the
curriculum.
Suggestions
 Review and Revise Academic Learning Compact
 Coordinate Assessment Plans with ALC
 Set priorities as a department
 “Hand off” specific learning outcomes assessment to
course/professor best suited to task but share and use
results
 Consider using Analytic Rubrics
 Be clear as to how assessment results will be used to
revise pedagogy, curriculum matrix, and
teaching/learning strategies in program/courses
Conclusions?
 Align Assessment Activities to make them Useful
 Don’t duplicate work: Align the work
 Create An Outcomes Curriculum Matrix
 Align assessment for AACSB, ALC, and FAU
Assessment Database
 Document and Report Results
 Identify Specific ways in which assessment results will
be shared
 Identify Specif ways to USE data to improve learning
The Flow of the Assurance of Learning - AACSB
Questions?
ASSESSMENT IS A TOOL TO BRING
About
better teaching and learning!
www.fau.edu/iea/assessment
gwisan@fau.edu
Appendix
 ITOM ALC on remaining slides
B.B.A. & B.S. Management Information Systems
College of Business
Dep’t. of Information Technology And Operations Management
Rubric