Applying the
QM Rubric
2011-13 Rubric
©2014 MarylandOnline
Workshop Overview
• Today’s Agenda
• Folder Overview
• QM Website
www.qmprogram.org
• QM Contact Information
©2014 MarylandOnline
Login to MyQM
1. Go to MyQM
http://www.qmprogram.org/myQM/
2. Click on Need an Account? link (lower part of page)
or login if you already have an account
3. Review the MyQM site: My Account, My Activity,
My Tools, and Workshops
4. Certificates
©2014 MarylandOnline
After this workshop, you will be able to:
1. Identify the underlying principles of QM. (Recognize key QM
underlying principles and concepts.)
2. Identify the critical elements of the QM quality assurance
program, including the QM Rubric, materials, processes, and
administrative components.
3. Apply the 2011-2013 QM Rubric to review online courses.
4. Decide if the SPCH 1113 course meets selected QM Rubric
standards.
5. Apply the concept of alignment.
6. Write helpful recommendations for course improvement by
citing annotations from the QM Rubric and evidence from the
course.
©2014 MarylandOnline
And in addition, we expect that you will:
1. Collaborate and network with colleagues through substantive
and timely interactions.
2. Reflect on key QM concepts and whether you wish to serve
on a peer review team.
3. Explore the challenges of online teaching by familiarizing
yourself with the QM rubric and the SPCH 1113 course.
4. Relate the QM rubric (standards and annotations) to your
own course and consider changes that might benefit your
online students.
5. Evaluate the QM APPQMR workshop.
©2014 MarylandOnline
Activity 1: Introductions
• Share name, institution, job, and best distance
learning practice.
• Briefly (in one sentence) describe your best
distance learning practice.
©2014 MarylandOnline
Activity 2: Trying it out with Standard 1.2
Standard 1.2: Students are
introduced to the purpose
and structure of the course.
©2014 MarylandOnline
Review Scenario
You are
reviewing a
College
Algebra
course and
find the
following
statement:
This course reflects a shift in the importance that the world outside the schools increasingly places on thinking
and problem solving. Procedural skills alone do not prepare students for that world. Therefore, students
deserve a curriculum that develops their mathematical power and an assessment system that enables them to
show it. Assessments that match the current vision of school mathematics involve activities that are based on
significant and correct mathematics. These activities provide all students with opportunities to formulate
problems, reason mathematically, and make connections among mathematical ideas. Students engage in solving
realistic problems using information and the technological tools available in real life. Moreover, skills, procedural
knowledge, and factual knowledge are assessed as part of the doing of real life mathematics.
1. 5-Tests (Must be taken on campus)
2. 15-Quizzes
3. Project (Optional)
Total number of points possible
400
75
50
475 (no project)
525 (if you do the project)
Grading Scale:
90% to 100% = A 80% to 89% = B 70% to 79% = C 60% to 69% = D 0% to 59% = F
A student who earns a final average of 70% or more on all assessments of objectives and intended learning
outcomes for the College Algebra course has successfully fulfilled the general education and other essential core
skill goals. Any extra credit problems are given to the entire class. There are no extra credit problems for any
one individual during any part of the semester.
©2014 MarylandOnline
About
Quality Matters
©2014 MarylandOnline
Underlying Principles of QM
QM is a faculty-driven, peer review process
that is…
Collaborative
Collegial
Continuous
Centered - in academic foundation
- around student learning
©2014 MarylandOnline
Peer Course Review Process
©2014 MarylandOnline
For Our Purposes, Quality Is…
• More than average; more
than “good enough”
• Attempt to capture what’s
expected in an effective
online course at about an
85% level
• Based on research and widely
accepted standards
©2014 MarylandOnline
Factors Affecting Course Quality
QM Reviews
Course
Design ONLY
©2014
MarylandOnline
Design vs. Delivery
The faculty member is
integral to both design and delivery.
Course Design …
is the forethought and
planning that a faculty
member puts into the course.
Course Delivery …
is the actual teaching of the
course, the implementation of
the design.
QM is about DESIGN - not delivery or faculty performance
©2014 MarylandOnline
Distinguish between design and delivery…
Example: Discussion Board
Design:
Delivery:
Discussion board
planned in course;
students told how they
should participate and
how they can expect the
faculty to participate.
How often the faculty
member actually
participates in the
discussion; what the
faculty member actually
says to students.
©2014 MarylandOnline
Activity 3: Design vs. Delivery
• Find a partner (or two)
• Share stories about your own
experiences with design vs. delivery
(professional or personal)
• Decide on an experience that you’d like
to share with the group.
©2014 MarylandOnline
What QM is NOT About…
• Not about an individual instructor
(it’s about the course)
• Not about faculty evaluation
(it’s about course quality)
• Not about judgment
(it’s about diagnosis and improvement)
• Not about “win/lose” or “pass/fail”
(it’s about continuous improvement in a supportive
environment)
©2014 MarylandOnline
The QM Rubric
©2014 MarylandOnline
About The QM Rubric 2011-2013
Eight General Standards:
1.
2.
3.
4.
5.
6.
7.
8.
Course Overview and Introduction
Learning Objectives (Competencies)
Assessment and Measurement
Resources and Materials
Learner Engagement
Course Technology
Learner Support
Accessibility
Key components must align.
Alignment: Critical course elements work together to ensure
that students achieve the desired learning outcomes.
©2014 MarylandOnline
Key Sections that Must Align
Submitted by James Fowlkes
& Brenda Boyd
©2014 MarylandOnline
QM Rubric – Standard Point Values
8 General Standards
– 21 Standards are “Essential”
– 12 Standards are “Very Important”
– 8 Standards are “Important”
# Standards
Points
Relative Value
21
3
Essential
12
2
Very Important
8
1
Important
©2014 MarylandOnline
Activity 4: QM Rubric Puzzle
• Divide into groups of 3
• You will have 10 minutes to:
– Decide which general standard each specific review
standard supports
– Decide whether your standard is essential, very important
or important.
– Note: your group does not have come to agreement on
where to place the specific standard or its relative value;
majority rules
©2014 MarylandOnline
The Faculty Developer
(Instructor)
Worksheet
©2014 MarylandOnline
Faculty Developer (Instructor) Worksheet
• The Voice of the Instructor
• Key Piece in the review
• Includes information about
– Institutionally mandated objectives, materials, practices,
and policies
– Materials outside the course site
– Types of interaction used and instructor’s statement on
the appropriateness of interaction in the course
• Read it before you begin the review
• Refer to it during the review and in team discussions
©2014 MarylandOnline
Activity 5: Faculty Developer Worksheet
• Locate the Faculty Developer (Instructor) Worksheet
in your packet
• Read the Worksheet, be sure to pay close attention
to the instructor’s answers to:
–
–
–
–
–
Course- and Module-Level Objectives
Student Engagement
Technology
Accessibility
Assessment
©2014 MarylandOnline
Hands-On Practice
©2014 MarylandOnline
Goals for Hands-on Practice
1.
2.
3.
4.
Introduce the General Standards
Use the Faculty Developer Worksheet
Take an in-depth look at Alignment
Make decisions on specific review standards
by looking at an online course
5. Practice writing useful recommendations
6. See how the recommendations affect a
course
©2014 MarylandOnline
Your Point of View …
As a QM Peer Reviewer, you should:
• Take the students’ point of view
• Advocate for the student
• Support your decisions with
– Citations from the standards and annotations
– Evidence from the course
©2014 MarylandOnline
Accessing the SPCH 1113 Course
1. Find a Partner
2. Read Faculty Developer (Instructor)
Worksheet for course
3. Login to Principles of Speech (SPCH 1113)
•
•
•
URL: https://sautech.blackboard.com/
Username: qmstudent
Password: qmstudent
©2014 MarylandOnline
Strategies for Applying the
QM Rubric
General Standard 1:
Course Overview and Introduction
©2014 MarylandOnline
General Standard 1
About the
General Standard
General Standard 1,
Course Overview and
Introduction:
The overall design of the
course is made clear to the
student at the beginning of
the course.
Brief Description
from the Rubric
The course introduction
sets the tone for the
course, tells students what
to expect, and provides
guidance to ensure they get
off to a good start.
©2014 MarylandOnline
To meet the
General Standard
You must share your deep
knowledge of course
1. Organization
2. Navigation
3. Expectations (Policies)
How To Decide… For EACH Standard
1. Read the specific review standard and the annotation;
review the examples.
a. Identify key components: ideas, directions, examples,
etc.
b. Ask relevant questions
2. Look for evidence that the standard is met in this
course.
3. Ask yourself: Does this course meet the standard at an
85% or better level?
4. Decide Met or Not Met.
5. Write Helpful Recommendations.
©2014 MarylandOnline
Look at a Helpful Recommendation
©2014 MarylandOnline
Review Standard 1.1
1.1 Instructions make clear how to get started and
where to find various course components.
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Read the Annotation
Instructions provide a general course overview, present the schedule of activities, guide the new student to
explore the course website, and indicate what to do first, in addition to listing detailed navigational
instructions for the whole course.
Instructors may choose to incorporate some of this information in the course syllabus. In this case, students
should be directed to the syllabus at the beginning of the course. A useful feature is a “Read Me First” or “Start
Here” button or icon on the course home page, linking students to start-up information.
Examples:
1. A course “tour”
2. Clear statements about how to get started in the course
3. A “scavenger hunt” assignment that leads students through an exploration of the different areas of the
course
4. A graphical table or diagram that depicts the relationship between the online and face-to-face portions of
a blended course
Blended Courses: Instructions in the online classroom make it clear to students that the course is a blended
course, with both online and face-to-face components and activities. Instructions specify the requirements for
participation in both the online and face-to-face portions of the course. The introductory information clearly
states when and where students should participate each week, and a structured set of topics and a schedule are
provided for each face-to-face meeting.
©2014 MarylandOnline
Standard 1.1 – How to use the Annotations
Annotations specify:
1. What to look for:
• Instructions: a general course
overview, schedule, website
• Indication of what to do first
• Detailed navigation instructions
2. Where to look:
• Course syllabus
• “Read Me First” or “Start Here” button
or icon
3. Examples
©2014 MarylandOnline
Questions to ask:
1. Does the instructor
provide instructions
about key course
components?
2. Do the students know
what to do first and
how to start the
course?
Apply Standard 1.1
• Apply Standard 1.1 to the SPCH 1113 course
– Imagine that you are a new student (new to the course, the
discipline, and/or the Learning Management System)
• Consider the following questions:
–
–
–
–
Do you know what to do first?
Do you know how to start the course?
Can you easily find clear instructions?
Note your initial reaction to the course:
• Do you feel comfortable with the navigation?
• What are your first impressions about how this course works?
©2014 MarylandOnline
Activity 6: Your Decision
MET
Standard 1.1
©2014 MarylandOnline
NOT MET
Consequences of Your Decision?
In a formal QM review:
—
What does a MET decision mean?
—
What does a NOT MET decision mean?
©2014 MarylandOnline
Importance of Learning
Objectives
General Standard 2:
Learning Objectives (Competencies)
©2014 MarylandOnline
General Standard 2
About the
General Standard
General Standard 2,
Learning Objectives:
Learning objectives are
measurable and are stated
clearly.
Brief Description
from the Rubric
To meet the
General Standard
The learning objectives
establish a foundation upon
which the rest of the course
is based.
You must provide
measurable, precise
learning objectives at the
course- and module-level
objectives so your
students will know what is
expected of them.
©2014 MarylandOnline
Standard 2 as a Sentence
Quality is
measurable objectives (2.1)
and consistent module-level objectives (2.2),
that are written from the student perspective (2.3),
with instructions on how to meet them (2.4),
that are appropriate for the level of the course (2.5).
Submitted
by Sasha Thackaberry, Cuyahoga Community College
©2014 MarylandOnline
Review Standard 2.1
2.1 The course learning objectives describe outcomes
that are measurable.
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Review Standard 2.2
2.2 The module/unit learning objectives describe
outcomes that are measurable and consistent with the
course-level objectives (alignment).
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Standard 2.1 and 2.2
How to apply the Annotations
Questions to ask:
Annotations specify:
1. What to look for:
– Measurable course and module
learning objectives precisely
describe what students are to gain
from instruction and provide the
criteria instructors need to
accurately assess student
accomplishment.
– Objectives describe student
performance in specific, observable
terms.
2. Examples
3. Special situations
4. Alignment
1. Are there course level
objectives?
2. Are there module level
objectives?
3. Are they measurable?
4. Are they precise?
5. Do they provide criteria for
assessment?
6. Do they describe a behavior
that can be observed or
evaluated?
©2014 MarylandOnline
Standards 2.1 & 2.2 – Your Job
Write measurable course and module learning
objectives that
• precisely describe what students will gain from
instruction
• guide instructors to accurately assess student
accomplishment.
• are consistent
©2014 MarylandOnline
Measurable Learning Objective
Component
Example
Completes this sentence:
Upon completion of this
course/module, students will
be able to (DO SOMETHING).
At the end of this course, you
will be able to:
Begins with an action verb:
See: Clemson’s Bloom’s
Taxonomy Action Verbs
Precisely describes behavior
that can be observed or
evaluated
deliver
a carefully constructed
persuasive speech to an
audience
©2014 MarylandOnline
Activity 7: Measurable? Precise?
1. Realize the significance of ethical behavior in the
business environment.
2. Document the critical events leading to the beginning of
World War I.
3. Identify the seven most serious risks associated with
cigarette smoking.
4. Understand the continuing impact of World War II on
the European Union.
5. Demonstrate the effects of improper blood handling in a
critical care environment.
6. Describe how to create a financial statement using MS
Excel.
©2014 MarylandOnline
Bloom’s Taxonomy
Cassinelli, C. (2008). Bloom's Taxonomy: Original and revised. In edTEch Vision, Updated
Bloom’s Taxonomy. Retrieved September 21, 2012, from http://edtechvision.org/?p=123
©2014 MarylandOnline
Action Verbs
generate, plan, produce
judge, check, critique
differentiate, organize,
attribute
execute, implement
interpret, exemplify, classify,
summarize, infer, compare,
and explain.
retrieve, recognize, recall
Cassinelli, C. (2008). Bloom's Taxonomy: Original and revised. In edTEch Vision, Updated
Bloom’s Taxonomy. Retrieved September 21, 2012, from http://edtechvision.org/?p=123
©2014 MarylandOnline
Mnemonic Device: Bloom’s Taxonomy
R U At An
Excellent
College?
©2014 MarylandOnline
Bloom’s For Students
University of Victoria Counseling Services
http://www.coun.uvic.ca/learning/exams/bloomstaxonomy.html
©2014 MarylandOnline
In search of … a “precise” verb
Verbs from Bloom’s Taxonomy
Old and New Version
A list of verbs that focus on performance
(what the students will do)
©2014 MarylandOnline
Activity 8: Learning Objectives Worksheet
For this activity:
• Find a partner
• Locate the Learning Objectives Worksheet in your
packet
• Review the SPCH 1113 course site
• Complete the worksheet by applying Standards 2.1
through 2.5 to the SPCH 1113 course site
©2014 MarylandOnline
Practice Alignment
General Standards 2 – 6
Objectives, Assessments, Materials,
Engagement, and Tools/Media
©2014 MarylandOnline
Key Sections That Must Align
©2014 MarylandOnline
Think About “Alignment”
Objectives From Intro
Psychology:
Assignment: Read Chapter 4 (assume it deals with
relationships between biology and human behavior) in text;
review study guide, objective 2, page 4.
Course : Summarize
the relationships that
exist between biology
and human behavior
Assignment: View Chapter 4 PPT; listen to Chapter 4
podcast
Module
1. Identify and
define 7 major
biological areas.
2. Recognize
examples of how
each area affect
behavior.
Graded Assignment:
Prepare table that lists the 7 major biological areas; ask
students to define/describe each area and list 3 specific
ways this area affects human behavior.
Graded Discussion Forum Question: Psychology is the
study of behavior and mental processes. Why, then, are
psychologists interested in biological matters such as the
nervous system, the endocrine system and heredity?
Read and respond to classmates’ postings.
Exam: 50-item multiple choice exam taken in campus
Testing Center.
©2014 MarylandOnline
General Standard 3
About the
General Standard
General Standard 3,
Assessment and
Measurement:
Assessment strategies are
designed to evaluate
student progress by
reference to stated learning
objectives; to measure the
effectiveness of student
learning; and to be integral
to the learning process.
Brief Description
from the Rubric
Assessment is implemented
in a manner that not only
allows the instructor a broad
perspective on the students’
mastery of the content, but
also allows students to
measure their own learning
throughout the course.
©2014 MarylandOnline
To meet the
General Standard
Your assessments must
align with your stated
learning objectives and
guide students to
measure their own
learning progress.
Review Standard 3.1
3.1 The types of assessments selected measure the stated
learning objectives and are consistent with course activities and
resources.
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Standard 3.1: How to use the Annotations
Annotations specify:
Questions to ask:
1.
1. Do the assessments
flow from the course
and module/unit
objectives?
2. Are they consistent
with these objectives?
3. Do you (and will the
students) recognize
alignment?
Key Ideas (What to look for):
•
•
2.
Consistent with course and module
level objectives.
Alignment: Easy to see relationship
between learning objectives and
assessments. Clear that learning
objectives guide students to mastery
Examples
•
•
Alignment
Lack of Alignment
3. Special situations: Course objectives are
mandated and do not align with
assessments
©2014 MarylandOnline
Activity 9: Apply Standard 3.1
Standard 3.1
Evidence from
the Course
Met or Not
Met
The types of
assessments
selected
measure the
stated learning
objectives and
are consistent
with course
activities and
resources.
©2014 MarylandOnline
Aligned?
General Standard 4
About the
General Standard
General Standard 4,
Instructional Materials:
Instructional materials are
sufficiently comprehensive
to achieve stated course
objectives and learning
outcomes.
Brief Description
from the Rubric
The instructional materials
form the core of the course,
and these standards respect
the instructor’s prerogative
in selecting them. The focus
of this standard is on
supporting the course
objectives and
competencies, rather than
on qualitative judgments
about the materials.
©2014 MarylandOnline
To meet the
General Standard
You must provide rich,
robust and appropriate
resources and materials
that support your stated
learning objectives.
Review Standard 4.1
4.1 The instructional materials contribute to the achievement of
the stated course and module/unit learning objectives.
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Standard 4.1: How to use the Annotations
Annotations specify:
1. Key Ideas (What to look for):
The course materials and resources
enable students to achieve the stated
learning objectives. Consult with the
team SME (subject matter expert) and
use common sense. Focus only on the
alignment of the instructional materials
with the learning objectives rather than
attempt to evaluate the content.
2. Special situations: Use
module level objectives if courselevel objectives are not measurable
3. Alignment: Part of alignment
concept
©2014 MarylandOnline
Questions to ask:
1. What instructional
materials are used in
this course?
2. Do these materials
enable students to
meet learning
objectives?
3. Do I need to consult
with the SME?
Activity 10: Apply Standard 4.1
Standard 4.1
Evidence from the
Course
Met or Not
Met
The
instructional
materials
contribute to
the
achievement of
the stated
course and
module/unit
learning
objectives.
©2014 MarylandOnline
Aligned?
General Standard 5
About the
General Standard
General Standard 5,
Learner Interaction &
Engagement:
Forms of interaction
incorporated in the course
motivate students and
promote learning.
Brief Description
from the Rubric
Engaging students to
become active learners
contributes to the learning
process and to student
persistence.
©2014 MarylandOnline
To meet the
General Standard
You must provide
meaningful and
productive activities that
prompt your students to
actively practice their
learning.
Review Standard 5.1
5.1 The learning activities promote the achievement of
the stated learning objectives.
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Standard 5.1: Analyze the Standard
Annotations specify:
1.
Key Ideas (What to look for): Engage
students in activities that directly
contribute to the achievement of
learning objectives.
2. Examples: readings, student
presentations, science labs, class
discussions, case studies, role
playing, simulations, practice
quizzes, tests, etc.
3. Special situations: blended courses
4. Alignment: Part of Alignment Concept
©2014 MarylandOnline
Questions to ask:
1. What learning activities are
used in this course?
2. Do these activities enable
students to meet learning
objectives?
3. Are the learning activities
engaging and varied?
Activity 11: Standard 5.1
Standard 5.1
Evidence from the
Course
Met or Not
Met
The learning
activities
promote the
achievement of
the stated
learning
objectives.
©2014 MarylandOnline
Aligned?
General Standard 6
About the
General Standard
General Standard 6,
Course Tools and
Technology:
Course navigation and
technology support student
engagement and ensure
access to course
components.
Brief Description
from the Rubric
The technology enabling the
various course components
facilitates the student’s
learning experience and is
easy to use, rather than
impeding the student’s
progress.
©2014 MarylandOnline
To meet the
General Standard
You must select the tools
and media that best
support your learning
objectives and provide
opportunities for your
students to actively
practice their learning.
Review Standard 6.1
6.1 The tools and media support the course learning
objectives.
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Standard 6.1: How to use the Annotations
Annotations specify:
1. Key Ideas (What to look for): The
Standard
tools and media align with objectives
of the course by supporting
assessments, instructional materials,
activities.
2. Examples: tools and media; not
required.
3. Special situations: use module
objectives if course-level not
measurable
4. Alignment: Part of Alignment
Concept
©2014 MarylandOnline
Questions to ask:
1. What tools and media are used
in this course (if any)?
2. Do the tools and media
selected enable students to
meet objectives?
3. Is the purpose of the tools and
media clear?
Activity 12: Standard 6.1
Technology
Used
Evidence From
the Course
Met or Not
Met
Course
Tools
Media
©2014 MarylandOnline
Aligned?
Activity 13: Alignment Worksheet
• Complete the Alignment Worksheet
if you haven’t already done so.
• Share and compare your alignment
worksheets and any “epiphanies”
you had with the group.
©2014 MarylandOnline
Write Helpful
Recommendations
General Standards 7 – 8
Learner Support and Accessibility
©2014 MarylandOnline
No Choice!
You MUST write
a Helpful Recommendation
if you decide that the course
does not meet
the specific standard.
©2014 MarylandOnline
Look at a Helpful Recommendation
©2014 MarylandOnline
Write a Useful Recommendation
Constructive
Specific
Measurable
Try to offer solutions, not just identify
problems.
Include a specific example of what is being
recommended.
How will you or the ID/instructor know when
the recommendation has been implemented?
Sensitive
Avoid negative language. Keep
recommendations and comments on a
positive note.
Balanced
Point out strengths as well as weaknesses.
©2014 MarylandOnline
Mnemonic Device
Can Someone
Make Some
Brownies?
©2014 MarylandOnline
Analyze A Recommendation
Constructive
Specific
Measurable
Sensitive
The Start Here button was a great
idea, but when I read the information
it contained, I still couldn’t tell exactly
how to begin the course. It might be
very helpful to include a prominent
link and directions about what
students should do next to actually
begin the first lesson of the course.
Balanced
©2014 MarylandOnline
Activity 14: Improve Recommendations
For this activity you will work in groups. Each group
will be assigned one of the recommendations listed
below and will rewrite it using the “components” of
an effective recommendation. Each group will
share their recommendation.
1. Assignment instructions weren’t clear.
2. You didn't tell the students how to find the
additional resources.
3. The text on the page was too hard to read.
4. The learning objectives aren’t measurable.
5. Your assessments are weak.
©2014 MarylandOnline
General Standard 7
About the
General Standard
Brief Description
from the Rubric
General Standard 7,
Learner Support:
In the learner support
standard, four different
The course facilitates
kinds of support services are
student access to
addressed: technical
institutional support
support, accessibility
services essential to student support, academic services
success.
support, and student
services support.
©2014 MarylandOnline
To meet the
General Standard
You must direct your
online students to the
support services they
need to be successful and
would be available to
them if they attended
campus-based courses.
Review Standard 7.2
7.2 Course instructions articulate or link to the
institution’s accessibility policies and services.
As a Reviewer:
What do you do first? Where do you start?
Refer to the “How to Decide” handout
©2014 MarylandOnline
Standard 7.2: How to use the Annotations
Annotations specify:
1. Key Ideas (What to look
for): accessibility policies,
accommodations; types of
and how to access services
Questions to ask:
1. Does the institution
have a stated
accessibility or
accommodation
policy?
2. Examples: link to
institution’s policy;
2. Where are they?
Can a student find
access statement
them?
3. Special situations: no
3. What if there is no
institutional policy
stated policy?
©2014 MarylandOnline
Activity 15: Standard 7.2
Standard
Link to the
Statement to
Institution’s
the students
formal policy
included?
included
(evidence from
the course)?
7.2
©2014 MarylandOnline
Your
Decision
Recommendations
Activity 16: Write a Recommendation
• Use the evidence you discovered in the SPCH
1113 course to write a recommendation for
improving Standard 7.2
• Each team will share their recommendation
and point out each component of an effective
recommendation.
©2014 MarylandOnline
General Standard 8
About the
General Standard
Brief Description
from the Rubric
General Standard 8,
Accessibility:
The accessibility standard
incorporates the principles
The course demonstrates a of Universal Design for
commitment to accessibility
Learning (UDL) and is
for all students.
consistent with Web Content
Accessibility Guidelines
(WCAG).
©2014 MarylandOnline
To meet the
General Standard
You must design your
course so students with
special needs can access
course materials and be
successful in your course.
Sites to Explore
Universal Design for Learning (UDL)
Site provides “a framework for:
• designing educational environments that enable all learners
to gain knowledge, skills, and enthusiasm for learning
• reducing barriers to the curriculum
• providing rich supports for learning.”
Web Content Accessibility Guidelines (WCAG)
Site provides documents that
• “explain how to make Web content more accessible to
people with disabilities.”
©2014 MarylandOnline
Review Standard 8.1
8.1 The course employs accessible technologies and
provides guidance on how to obtain accommodation.
As a Reviewer:
What do you do first? Where do you start?
(Refer to the “How to Decide” handout)
©2014 MarylandOnline
Standard 8.1: How to use the Annotations
Annotations specify:
1. Key Ideas (What to look for):
Standard
LMS accessibility statement,
documentation on any
content, tools, and software,
information on how to obtain
accommodation
2. Examples: link to the LMS
accessibility statement, link to
other tools, content, or
software accessibility,
statement from instructor
Questions to ask:
1. Does the course contain
a link to the LMS
statement?
2. Are the other tools,
content, or software
accessible?
3. Do the students know
how to obtain
accommodation if
needed?
©2014 MarylandOnline
Standard 8.1 - What to Look For
• Link to the Learning Management System Accessibility
Statement
• Students will be using Microsoft PowerPoint to create a
presentation:
‒
Microsoft PowerPoint® will be used to create your e-portfolio. The
Microsoft Corporation has developed this software product with
accessibility in mind. Product accessibility information is available on
their website: http://www.microsoft.com/enable/default.aspx
• Students are asked to watch videos created by their
classmates that are not closed captioned.
– If you are unable to view the videos created by your classmates please
contact the Disability Services Center. The Disability Services personnel
will work with you to ensure you have access to this content. The
center can be reached by calling (555-555-1234) or email
(disabilityservices@somecollege.edu). In addition, you can review
their website at: http://www.somecollege.edu/disabilityservices.
©2014 MarylandOnline
Activity 17: Standard 8.1
Standard
Evidence from the
Course
Decision
8.1
©2014 MarylandOnline
Recommendations
Activity 18: Write a Recommendation
• Use the evidence you discovered in the SPCH
1113 course to write a recommendation for
improving Standard 8.1
• Each team will share their recommendation
and point out each component of an effective
recommendation.
©2014 MarylandOnline
Effects of
Helpful Recommendations
SPCH 1113 course
Standards 7.2 and 8.1
©2014 MarylandOnline
Activity 19: Course Improvements
• Read the decision and the recommendations from the review
team for standards 7.2 and 8.1 (located in your packet)
– Did you make the same decision?
– What additional information did you include in your recommendations?
• Log in to the revised version of the SPCH 1113 course (login
information located in your packet)
– Did the recommendations from the review team help the instructor
improve the course?
– What impact will the revisions have on the student’s experience?
– Do you feel the revised course meets these standards?
©2014 MarylandOnline
The Peer Review
Process
©2014 MarylandOnline
QM = Process and Rubric
Process
OFFICIAL
INFORMAL
Rubric
Outcome: Earn QM recognition
Must follow official QM guidelines
and procedures
Outcome: Earn QM recognition
Tool to assess online courses during
formal QM review
Outcome: Improve courses, meet
institutional goals, demonstrate
commitment to quality
Outcome: Improve courses,
meet institutional goals
Outcome: Improve courses, meet
institutional goals, demonstrate
commitment to quality
Institutions determine use and
procedures
Outcome: Improve courses, meet
institutional goals, demonstrate
commitment to quality
Guide to develop new online
courses AND review and update
online courses
©2014 MarylandOnline
Peer Course Review Process
©2014 MarylandOnline
About the Course
•
QM is designed to review “mature” courses (taught
at least two semesters)
•
QM logo indicates year course met expectations
•
Triggers for subsequent reviews:
–
–
–
–
Faculty request
More than 5 years since original review
New textbook or instructor
Professional or accreditation review pending
©2014 MarylandOnline
The Peer Review Team
• 3 faculty peer reviewers:
•
•
•
•
must be experienced online instructors
must attend QM training
one MUST be external to the course’s originating institution
there must be a subject matter expert (SME)
on the team. Note: The SME could also be the external reviewer.
AND
• Faculty developer:
• access to rubric prior to review
• involved in pre-review discussions
• consulted during review
©2014 MarylandOnline
In an Official QM Review
• Team of 3 reviewers initially score individually
– One score per standard based on team majority
– Pre-assigned point value
– Met/Not Met decision; All/None points
– Consensus is NOT required
• To Meet Expectations…Two Thresholds
– “Met” on all 21 of the 3-point “essential” standards.
– A minimum of 81 out©2014
of 95
points (81/95 = 85%)
MarylandOnline
Points are NOT assigned on a sliding scale…
If the standard
is met …
If the standard
is not met …
Essential
3 points
0 points
Very
Important
2 points
0 points
Important
1 point
0 points
Standards
©2014 MarylandOnline
The 85% Rule
Two 85% Uses:
1. A minimum of 81 out of 95
points (81/95 = 85%)
2. Reviewers use their own gauge of
85% as to if a standard is met or
not. Standards do not have to be
100% to be marked “Met.”
©2014 MarylandOnline
About the Review
• On average, a course review takes 7-10
hours
• Factors affecting review time include
• Reviewer familiarity with the discipline
• Reviewer familiarity with the LMS
• Reviewer familiarity with the QM review
process
• Organization of the course
©2014 MarylandOnline
Timeline for a QM Review
• Pre-Review: (1-2 weeks)
–
–
–
–
Instructor Worksheet
Conference Call
Set Team Calendar
Faculty Developer
• Active Review: (3 weeks)
– Reviewers actively review course
• Post-Review: (1 week)
– Have post-review discussion, if applicable
• Revisions: (14 weeks)
– Course improvements made to meet standards
©2014 MarylandOnline
Formal Review Outcome
• If meets expectations:
– Recognized by Quality Matters
– Notifications distributed
– Recognized on QM website
• If does not yet meet expectations:
– Instructor (and/or ID) make changes
– Team Chair/Master Reviewer approve revisions
– Course meets expectations and is recognized
©2014 MarylandOnline
Self-Review Tool
• Used to do a self-review of online or blended courses
• Available through MyQM >> HE tab >> My Tools >>
Course Review System
• Course Review System
– Start a self-review
•
•
•
•
Lists each specific standard with links to the annotations
Allows you to make a decision and add a recommendation
Can save and return
Once all the standards are completed and saved you can “View” or
“Email” the results
• Can do as many self-reviews as you want (one for every course you teach)
• No one else has access to the self-review
©2014 MarylandOnline
Serving as a QM Peer Reviewer
©2014 MarylandOnline
Peer Reviewer Eligibility
How Do I
Become a
QM Peer
Reviewer?
You must be ELIGIBLE:
• Must complete the Applying the QM
Rubric workshop.
• Must complete all assessments in the
online Peer Reviewer Course
– Two weeks online
– Objective and written assessments
– Practice Review
• Must have recent (within last 18
months) online teaching experience in a
for-credit course.
©2014 MarylandOnline
Reflections
©2014 MarylandOnline
Reflections
• What are the three most significant (or surprising)
things you learned by participating in our APP
course?
• What do you intend to do next in regards to
beginning to actively use QM and the QM Rubric
with your course and/or at your institution?
• Do you have plans to become a QM peer reviewer?
Explain, please.
• Do you have any general questions about the course
review process, the rubric, or Quality Matters?
©2014 MarylandOnline
Wrap Up and Evaluation
Link to the online evaluation:
©2014 MarylandOnline
Thanks to you…
Quality Matters!
For more information
visit the QM website at:
www.qmprogram.org
©2014 MarylandOnline