Considering how to measure habits of mind in the college context
Marybeth Buechner, Dean of Planning, Research and Institutional Effectiveness, Sacramento City College.
Types of learning evaluation:
Summative versus Formative evaluation
 Different functions: Summative assessment measures “completed” learning and is useful to
evaluate student success on outcomes at the completion of a course or program.
 Formative assessments measure learning “in progress” and allow the teacher and learners to
modify their actions to improve future success in the course/program.
Norm-referenced versus criterion-referenced evaluation
 Norm-referenced evaluations compare the results from a set students to an expected value based
on a larger group. For example, “grading on a curve” is a type of norm-referenced learning
evaluation.
 Criterion-referenced evaluation sets criteria and compares student scores to those criteria. For
example, a rubric might be used to set the standards against which student work is measured.
Subjective versus "countable" scoring
 Some disciplines and/or assessment techniques lend themselves to “countable” scoring (e.g.
scores on a class assignment aligned with habits of mind or multiple choice survey responses).
 Some disciplines and/or assessment techniques may be hard to score by a count, and are more often
scored subjectively (e.g. observational notes from a focus groups/interviews or essays graded via
a rubric)
Characteristics of educational assessment measures:
Several aspects of educational measurement can affect the tools that we could use to measure habits of mind.
Some of those are outlined below.
Validity: Does the assessment tool measure what it is meant to measure?
 Content validity and alignment with objectives
 Expectations of students affect validity
 Validity is established with respect to specific reference groups.
Reliability: How dependably or consistently a test measures a characteristic. If a person takes the test
again will the scores be similar? Are results consistent for separate matched groups?
 Unambiguous directions and consistency of how questions are "read” by the students.
 Consistency in scoring (e.g. of narrative responses)
 Test takers physical or psychological state and environmental factors affect reliability.
Practicality: Is the assessment feasible in terms of time and resources?
 Skill and time needed to produce assessment instrument and analyze outcomes
 Resources (e.g. for a practical exam) needed to use the assessment tool
Positive effect: Do the results lead to improvements?
 Does it encourage deep learning? Will the results be used to improve the teaching/learning
process in the future?
1
Evaluation of learning and the habits of mind that support learning
Tools that might be used in the college setting
A. Measures of Skills, Attitudes, or Behaviors
Direct observation of skills or behaviors that can be conducted in the classroom:
Role plays, presentations, etc.
 Role plays, presentations and similar tools may provide direct observations of behaviors. However,
reliability problems may interfere with the use of these techniques. Detailed rubrics, checklists, and
rating forms, are helpful in solving this problem.
Guided discussion
 Direct observation of student response during guided discussion in groups can make evident both
thinking skills and foundation knowledge.
 Guided discussion is relatively difficult to score. It can be used to provide a reflective qualitative
assessment of some outcomes if rubrics are used to define levels of success for the group.
Performances, structured practical assessments, and demonstrations of specific skills:
 These assess skills which are usually psychomotor in nature (e.g. lab equipment use, dance steps,
construction techniques).
 Validity may be increased by specifying criteria on a rubric, providing clear instructions for the tasks,
and allowing a task to be performed multiple times.
Portfolios
 Portfolios of student work can be used to show achievement of skills. It is helpful to (1) have a clear
evaluation system and rubrics (2) have students do more than just "turn in" items, e.g. have them
explain and evaluate the importance of each item in the portfolio.
Essays (blue book exams, long essays as homework, etc.)
 Essays can measure outcomes linked to critical thinking, or those reflecting analysis, evaluation, and
synthesis skills.
 These are time consuming to grade, and, since, marking may be subjective; rubrics are nearly
essential.
Short-answer questions on homework, exams, etc.
 Short-answer questions are moderately easy to grade and moderately time consuming to write.
Rubrics are helpful, especially when there is a need to be very clear about what alternatives of the
answer are acceptable.
Objective (multiple-choice, true/false, matching) questions.
 These provide a high reliability of scoring and are easy to grade. Considerable skill is required to write
objective tests that measure critical thinking skills, problem solving, or other habits of mind.
2
Direct measures of student behavior outside of the classroom:
Measures of student participation in activities on campus and use of campus resources:
 These are useful to assess some aspects of habits of mind and may provide direct assessment of some
skills (e.g. awareness of and willingness to utilize campus resources).
 Measures of student activity/participation on campus may provide indirect measures habits of mind
related to student interaction with the college. For example, the number of students using the tutoring
center may be an indicator of the willingness of students to seek academic assistance.
Measures of student behaviors related to civility and discipline:
 For example, a decrease in the number of students referred to disciplinary processes for cheating may
provide an assessment of outcomes related to ethical behavior.
Measures of student success at tasks outside of the classroom:
 Completion of some college tasks or processes reflects habits of mind. For example, the percentage of
freshmen who have completed a student education plan may be a reflection of a habit of mind related
to goal seeking and academic planning.
Indirect measures of student attitudes and behaviors:
Focus groups and interviews
 Focus groups and interviews can be designed to assess student attitudes or values (e.g. the importance
of having an educational goal) and to gather information about the behaviors that students report doing.
Focus groups and/or interviews can be used in conjunction with surveys; this combination of techniques
can provide greater information than either alone
 Good rubrics and techniques that increase examiner consistency (check lists, multiple observers, etc.)
are important.
Student self-assessments and surveys
 Student self-assessment tools include student self-evaluation of a skill, attitude, or behavior; individual
discussions between a student and a teacher or researcher; short surveys immediately following an
activity; student journals or reflective essays; major surveys conducted at the completion of a course,
activity, or program, etc.
 Validity and reliability can be major problems with student self-assessments unless the surveys are
well-designed and/or students are trained in how to assess their own success.
B. Indicators: Broad measures that may be affected by many events.
The following measures achievement measures that don’t usually tie directly to a specific single
experience that a student may have. For example, the student may have experienced several interventions
aimed at assisting with useful habits of mind during a single semester. Because of this, it is not usually
possible to tell which of interventions has had a specific impact. They don’t give us answers but do tell is
when to start asking important questions.
 Course completion and course success rates
 Progress through course sequences
 Persistence at college (Fall to Fall, Fall to Spring, etc.)
 Degree/certificate completion and transfer
 Employment information: licensure exam pass rates, job placement, etc.
3
Habits of Mind - Steps in developing indicators:
1. List the things we want to evaluate (that’s the list above)
2. Operationalize the things on the list = map them to measurable behaviors, skills or achievements
3. Choose key indicators that tie directly to the behaviors or skills.
Ask “What would these habits look like in our classes and in our student service programs?” Then to develop
key indicators associated with specific classes or services, or college-wide indicators, which might help us look
at these behaviors in different situations.
Initial List of Key Indicators for Habits of Mind
In specific classes:
It may be possible to map HoM to specific SLOs, behaviors, or assessments in a given class. This can provide
direct measures of HoM in action.
 Achievement of specific SLOs that align with one or more habits of mind.
 Scores on summative assessments (exams, projects, etc.) that align with one or more habits of mind.
 Specific observable student behaviors on graded or ungraded activities that align with one or more
habits of mind.
In specific student services:
It may be possible to map HoM to student behavior when using student services, engagement with those
services, or responses on surveys related to those services.
 Measurements of specific behaviors (e.g. completion of an Student Education Plan, follow up on
Counseling advice, etc.)
 Student engagement with specific services such as tutoring, library services, etc.
 Student responses on surveys asking about specific habits of mind.
College-wide indicators:
These are less likely to link directly to one HoM, but are more likely to be an indicator of the overall collection
of HoM that students may demonstrate.
 Student retention at the college from semester to semester
 Student achievement data (course success, program completion, degree attainment, etc.)
 Student engagement data (e.g. from college survey, CCSSE survey, etc.).
 Student behavior data (participation in activities, data from student disciplinary office, etc.)
4
Examples of how HoM can be mapped to student behaviors:
(With thanks to BSI folks from other colleges!)


Examples for specific classes –Biology and one for Math
A more general example - SCC’s Culture of Scholarship work
Habits of Mind operationalized for a science lab course
Debra Howell, Biology Department, Chabot College
Associated behaviors, skills or achievements that can be
Habit of Mind
measured in the community college environment
1. Persisting
 Assignments completed on time
 Working through entire laboratory period.
 Successful course completion
 Persistence in college from Fall to Spring semester
2. Managing impulsivity
 Students will safely manage laboratory equipment.
3. Listening with understanding and
 Students will collaborate with peers in order to share information,
ideas and responsibilities while preparing for and conducting lab.
empathy
4. Thinking flexibly
 Students will evaluate evidence and consider differing
hypotheses.
5. Thinking about your thinking
 Improvement on skills in courses based on feedback about
(Metacognition)
classwork.
6. Striving for accuracy
 Correctly following laboratory procedures
 Classwork of high accuracy
7. Questioning and problem posing
 Questions asked in classes or during advising sessions
 Design, perform and evaluate experiments.
8. Applying past knowledge to new
 Application of material from previous classes to the work in
situations
current classes
 Students will apply scientific principles to everyday occurrences,
social issues, or novel situations
9. Thinking and communicating with
clarity and precision


Classwork that is well written, clear and precise.
10. Gather data through all senses
11. Creating, imagining, and
innovating
12. Responding with wonderment
and awe
13. Taking responsible risks
14. Finding humor
15. Thinking interdependently
16. Remaining open to continuous
learning


(Don’t want students to smell or taste materials in science.)
Student will create posters, presentations.


Completion of an education plan
Students will learn to read and evaluate newspaper and
journal articles.
Students will express their understanding of scientific concepts by
verbal, written, and illustrative means using correct terminology.
5
Habits of Mind
Habits of Mind operationalized for a Mathematics class
Cynthia Stubblebine, Math Instructor, Chabot College
In the community college
In mathematics
environment
1. Persisting




Assignments completed on
time
Successful course
completion
Persistence in college from
Fall to Spring
Meet educational goal




Turning in all homework
assignments.
Finding out how to do a problem
correctly.
Finishing a course.
Seeking help if doing poorly.
2. Managing impulsivity

Explaining why a mathematical
method is used or why
particular method works.
Improvement on skills in
courses based on
feedback about
classwork.

Correctly following
instructions
Classwork of high accuracy


Looking at errors made on an
exam, determine types of errors
and make adjustments to do better
on next exam.
Determining why the method used
didn’t work or the step in the
method didn’t work.
Students checking their answers.
Neat, organized, logical work.
Questions asked in
classes or during
advising sessions
Application of material
from previous classes to
the work in current
classes.


3. Listening with understanding
and empathy
4. Thinking flexibly
5. Thinking about your thinking
(Metacognition)
6. Striving for accuracy



7. Questioning and problem
posing

8. Applying past knowledge to
new situations





9. Thinking and communicating
with clarity and precision
10. Gathering data through all
senses


Classwork that is well
written, clear and precise.
Presenting a logical
argument.






Asking questions for clarification.
Asking questions that extend the
scope of the material covered.
Applying formulas in application
problems.
Applying life experience to
formulate equations in application
problems.
Applying past techniques to new
material
Writing up solutions to application
problems clearly with complete
English sentences and units.
Including details.
Applying logic in solving
problems.
Neatness.
Determining given information in
an application problem.
Finding the information needed to
solve a problem.
11.Creating, imagining and
6
innovating
12. Responding with
wonderment and awe
13. Taking responsible risks



Starting application problems.
Asking questions in class.
Answering questions in class.
14. Finding humor
15. Thinking interdependently

Completion of an
education plan.

Participating in group work to
solve a problem.




Arriving to class on time.
Not leaving class early.
Coming to class every day.
Turning in assignments on
time.
Doing his/her share of
group work.
Coming to class prepared.





Arriving to class on time.
Not leaving class early.
Coming to class every day.
Turning in assignments on time.
Doing his/her share of group
work.
Coming to class prepared.
16. Remaining open to
continuous learning
17. Honoring commitments


18. Having organizational skills





Scheduling of study time.
Organizing notes, handouts,
exams, study cards, etc.
Turning in assignments on time.
Following directions.
7
Habits of Mind integrated with SCC’s Culture of Scholarship work
Marybeth Buechner, Sacramento City College, PRIE Office
Habit of Mind
With ideas about what it means to be a scholar
1.Persisting
An expectation of excellence and the
understanding that striving toward a difficult
goal is, of itself, admirable.
Associated behaviors, skills or achievements that can be
measured in the community college environment
 Increase of quality of student work related to a given
SLO over the course of a semester.
 Successful course completion.
 Persistence in college from Fall to Spring semester.
 Persistence in a Student Service program (e.g.
EOPS, DSPS, etc.)
 Use of tutoring services.
2.Managing impulsivity

The recognition of the ethical and social
implications of one’s actions.



3.Listening with understanding and empathy

The willingness to consider others points of
view


4.Thinking flexibly

The willingness to consider others points of
view, critically evaluate one’s own ideas, and
the recognition that, while there are wrong
answers, there may also be more than one right
answer.

5.Thinking about your thinking (Metacognition)

Willingness to engage in reflective thinking and
serious analysis even in the face of frustration.

6.Striving for accuracy
A stress on the importance of intellectual
honesty, respect for knowledge, self-reliant
learning, and personal responsibility.
7.Questioning and problem posing
Direct measures via case studies, reflection papers,
analysis of ethical problems associated with the
discipline (e.g. Bioethics), etc.
Reduction in interactions with the student discipline
office.
Indirect measures via classroom feedback, surveys,
etc.
Responses in an academic counseling session.
Direct measures via role plays, reflection papers,
journals, etc.
Indirect measures via classroom feedback, surveys,
etc.
Responses in an academic counseling session.
Direct measures via problem solving assignments,
projects, portfolios, etc.
Indirect measures via classroom feedback, surveys,
etc.

Improvement on skills in courses based on feedback
about classwork.
Increase of quality of student work related to a given
SLO over the course of a semester.
Successful course completion.



Correctly following instructions.
Classwork of high accuracy.
Reduction in cheating.

Questions asked in classes or during advising
sessions.
8
Recognition of the value of discourse, dialogue,
and shared development of ideas. A sense of
deep curiosity about the world.



Student participation in group projects.
Direct measures via problem solving assignments,
projects, etc.
Use of library services.
8.Applying past knowledge to new situations

A deep appreciation of the consilience of
knowledge, the recognition that it is all about
the same world, and an appreciation for
disciplines not one’s own.

9.Thinking and communicating with clarity and
precision


Classwork that is well written, clear and precise.
Direct measures via assignments requiring critical
thinking and clear communication.

Direct measures in lab/activity/studio situations that
require the collection of information through more
than one sense.
11. Creating, imagining, and innovating

Recognition of the emotional, behavioral and
intellectual aspects of learning.

12. Responding with wonderment and awe

The desire to learn, a sense of deep curiosity
about the world, and the ability to find learning
satisfying.

Direct measures via creative projects, class
discussion, etc.
Indirect measures via classroom feedback, surveys,
etc.
Indirect measures via classroom feedback, surveys,
etc.
Involvement with student activities
13. Taking responsible risks

Indirect measures via classroom feedback, surveys,
etc.
14. Finding humor

Joy in learning

Indirect measures via classroom feedback, surveys,
etc.
Involvement in student activities.
15. Thinking interdependently

Direct measures via classwork that indicates
interdependent thinking, group work, etc.

Indirect measures via classroom feedback, surveys,
etc.
Decision-making based on thoughtful
consideration and critical thinking.
10. Gather data through all senses
Observation and objective reflection
The recognition of the implications of one’s
actions.
Recognition of the value of discourse, dialogue,
and shared development of ideas.
16. Remaining open to continuous learning
Pursuit of a deep understanding of the work
done by those who have gone before us and
prior knowledge related to our endeavors.
Application of material from previous classes to the
work in current classes.
Direct measures via assignments that ask the student
to apply previous knowledge.
9