Outcomes Assessment in the
Distance-Learning Environment
We are:
Citrus College, Glendora, California (LA County)
Cedar Valley College, Lancaster, TX (Dallas)
•
•
•
•
Marcy Morris: Learning Center, English (CC)
Theresa Villeneuve: Communications, Fine Arts
Tom Eiland: English (CC)
Lisa Nightingale: Instructional Support & Dist. Ed,
Psychology (CVC)
• Eric Nightingale: Psychology, History (CVC)
Citrus College, Glendora,
California (LA County)
• 11,000 Students
• 100+ DE sections this
semester
• As many as 4000+
students per semester
Cedar Valley College
(Dallas CCCD)
• 4800 students
• 160+ DE sections this
semester
• As many as 4000+
students per semester
The Questions…
• Do differences exist in outcomes assessment
for the distance education / online
environment vs. the on-ground, traditional
environment?
• Should they be different for the DE
environment?
• Citrus’ DE program & the DCCCD have been
evaluating/reevaluating outcomes for 11
years.
DE Curriculum Outlines—
The History…
The first big questions…
• 1996—discussion about whether DE
courses should have a separate course
outline
• contact hours—connection with
students?
More History…
• 1998—incorporated the Distance Education
course outline into the conventional course
outline.
• Independent/Directed Study added as
method of instruction.
• Instructor syllabus included which specifies
how contact occurs.
The History, continued…
• March 10, 1998—California Board of
Governors approved recommendations
to Title 5 Regulations on Distance
Education.
• Each DE course to be separately
reviewed and approved.
• Emphasis on
regular effective contact
And Yet… Even More History…
• 2004: Distance Learning Course
Outline Addendum
• Benefit of providing course in DE format
• Methods of contact
• Course objectives / outcomes /
assignments / assessments
• Methods of evaluation for online mode.
Accreditation Standards...a tilt…?
• An effective institution maintains an
ongoing, self-reflective dialogue about
its quality and improvement (Introduction to the
Accreditation Standards 2002)
But, That Was Not Enough...
• Distance Learning Manual
• published August 2006 by Accrediting
Commission for Community and Junior
Colleges (ACCJC) / Western
Association of Schools and Colleges
(WASC)
ACCJC / WASC
Distance Learning Manual
• Institution: Academic standards for all
courses and programs should be the same as
for all other educational experiences
delivered by the institution.
• Students: Students should expect that
distance learning programs will permit
completion of learning outcomes and
objectives in the same manner as those
delivered in traditional programs.
• which validated our DE curriculum “journey”
Interesting Acknowledgement:
methodology used in DE classes have been
incorporated into classrooms on campus
• Communication:
•
•
•
•
Email
Web
Discussion / message / announcement Board
Chat
• Other:
•
•
•
•
Websites/FAQ’s
Course and Study Materials
Turnitin.com
Sample material
The Charge
• To identify elements of quality online
courses rather than elements of good
instruction.
• To identify class management standards
and responsibilities for teaching online.
The Action
 Review of existing course standards
documents/models
Quality Matters:
Inter-Institutional Quality Assurance
in Online Learning
A Grant Initiative of MarylandOnline
Sponsored by the U.S. Dept. Education Fund
for the Improvement of Postsecondary
Education (FIPSE)
FIPSE Interested Because …
• Quality assurance of courses is important
• Voluntary inter-institutional assurance has
never been done before
• This can serve as a national model
Quality Matters!
For Our Purposes, Quality Is…
• More than average;
more than “good
enough”
• An attempt to capture
what’s expected in an
effective online course
• Based on research and
widely accepted
standards
Higher
Standard
100
50
0
“They” say there is . . . .
“No Significant Difference”
• In student outcomes between alternate
modes of educational delivery . . . but
• The No Significant Difference Phenomenon
(Thomas L. Russell 2001, IDECC)
• includes 355 research reports, summaries,
and papers that document no significant
differences (NSD) http://nosignificantdifference.wcet.info
there is a Significant Difference
• When comparing face-to-face education with
Internet-Based Distance Education, there is a
significant difference in favor of the
Internet-Based Distance Education
. . . . In regards to student outcomes
Significant Difference - Better Results with Technology 2006 - Sahin, C.
S.Overcoming the "No Significant Difference" Phenomenon in Distance
Education by Internet Andolu University, Turkey http://aof.edu.tr/iodl2006"...
Example:
Freshman Composition 101:
• 1996 : Created freshman reading and
composition course for distance
learning
• Faculty demands:
• Maintain course standards
• Administration requirements:
• Cannot change course outline
• Only add Independent Study
Traditional Primary Learning
Objectives
• Analyze Fiction Objectively
• Present Research Paper
• Demonstrate Use of Literary Terms in
Deductive Essay
• Utilize MLA Formatting
Distance Education
Objectives
• Maintaining the Flexibility of Distance
Education
• Providing Students with Quality
Educational Experience
• Keeping Distance Ed Classes
Legitimate in the Eyes of Academia
Traditional Assessment Tools
•
•
•
•
•
Timed, Proctored, Deductive Essay
Analytical Research Paper
In-class Lecture
In-class Discussion
Topical Presentation
Primary Hurdle
• Primary concern among English faculty when
presented with possibility of distance
education Freshman Comp online course
was synchronous class contact.
• Theory was that in order to understand the
application of literary terms in the analysis of
fiction, one must be able to practice through
class conversation
Other factors . . . .
• Ensuring rigidity of testing
• Time
• Security
• Content
• Format
• Maintaining quality and quantity of written
research essays as required by established
college standards and IGETC agreement
(Intersegmental General Education Transfer Curriculum)
Solutions…
• Classroom time / Synchronous Contact
• Replaced with weekly chat room meetings
• Synchronous conversation with students
• Chat room transcript archived on web site
for review and retrieval
• All students required to attend or to read
archive and respond
Solutions…continued
• Testing Protocol:
• Establishment of Proctored Testing Center
• Testing center controls:
•
•
•
•
Allowed materials
Allotted time
Identification of test-taker
Adapted Testing needs
More Solutions
• Term Papers:
• Strict adherence to established guidelines
• Content
• Length
• Format
• Peer editing asynchronously through
message board
• Turnitin.com
• SafeAssign
Solutions…
(almost done)
• Asynchronous Contact
•
•
•
•
•
Message board
E-mail
Web page
Archived chat room
Web links
Results
• Comparing ENGL 101 and ENGL 103
Freshman Composition Courses
• Retention is higher in on-campus
courses
• Traditional Avg: 22.5/30
• Distance Learning Avg: 15.3/30
Results…continued
• Grade range reflects a standard curve
in traditional courses (frown)
• Traditional Delivery Averages:
•
•
•
•
A: 18%
B: 36%
C: 24%
D: 22%
More Results…
• Distance education grade range is wider
(smile)
• Distance Education Grade Averages:
•
•
•
•
A: 38%
B: 23%
C: 8%
D: 31%
FACTORS
(as determined by observation, surveys and interviews)
• Population
•
•
•
•
Age (older is better)
Experience (older is still better)
Distractions (less is better)
Motivation (more is better)
• Delivery Method
• Technology (BB, attachments, chat room)
• Influence of real, live person
What It Means . . . .
• Delivery methods create some different outcomes
BUT…
• Assessments tools and criteria are the same
• Testing center more uniform than individual
classroom practices (now being used for traditional
courses as well)
• Technical Support would help many students
succeeded in distance education
• The distance education version of freshman
composition at Citrus College has as much, if not
more, student learning outcome assessment and
verification than the on-campus version of the same
course.
Art 199
Motion Picture Appreciation
Distance Ed vs. Traditional Deliveries
Content Of The Course
• Social/technical factors of film
production
• Genres
• Aesthetics
• Narrative and non-narrative
construction
• Film form and history
• Ability to make sophisticated small talk
at cocktail parties.
Concerns
• Logistical concerns re: films
• Rigor
• Quality and Quantity of education to
maintain IGETC agreement
• I am the only faculty member who
teaches this course.
Methods of Instruction
(from course outline)
•
•
•
•
•
Lecture
Lecture/Discussion
Independent Study
Out of class film viewing
Directed study
Assessment
• Discussion
• Large Paper
• Quizzes
• Taken online
• Tests
• Taken in testing center under controlled
conditions until this semester
What it means
• No one flies under the radar
• No one can sleep through a film and
pass my class
• Testing is controlled
• Criteria are unchanged
• At least the same learning outcomes
as a traditional course.
Overall Midterm Comparison, Fall
2006
16
14
12
10
8
Distance Ed
6
Traditional
4
2
0
As
Bs
Cs
Ds
Fs
Final Comparison, Fall 2006
14
12
10
8
Distance Ed
6
Traditional
4
2
0
As
Bs
Cs
Ds
Fs
CVC Online Success & Retention
Online Course Success and Retention
100%
%
80%
60%
Overall Success
40%
Overall Retention
20%
0%
Spring 2005
Fall 2005
Semester
Spring 2005
Fall 2005
Overall Success
58%
56%
Overall Retention
81%
79%
Enrollments
1939
2066
CVC Spring 2006 Distance Learning and On Campus Grade Distributions
Course
Instr. Method
ACCT-2301 Prin Accounting I
ACCT-2301-3427
INET
BIOL-1408 Biology Non-Sci Maj I
College
Success
Retention
On Campus
47%
70%
Distance Learning
50%
75%
On Campus
58%
79%
BIOL-1408-3460
INET
Distance Learning
17%
50%
BIOL-1408-3461
INET
Distance Learning
53%
89%
BIOL-1408-3462
INET
Distance Learning
58%
75%
BIOL-1408-3463
INET
Distance Learning
49%
78%
BIOL-1408-3464
INET
Distance Learning
80%
100%
On Campus
70%
84%
BIOL-1409 Biology Non-Sci Maj II
BIOL-1409-3460
INET
Distance Learning
71%
100%
BIOL-1409-3461
INET
Distance Learning
56%
78%
BIOL-1409-3462
INET
Distance Learning
54%
85%
On Campus
60%
81%
BMGT-1303 Prin of Management
BMGT-1303-3426
INET
Distance Learning
100%
100%
BMGT-1303-3427
INET
Distance Learning
42%
82%
On Campus
54%
84%
BUSI-1301 Intro to Business
BUSI-1301-3426
INET
Distance Learning
100%
100%
BUSI-1301-3427
INET
Distance Learning
50%
78%
On Campus
64%
82%
COSC-1300 Computer Literacy
COSC-1300-3426
INET
Distance Learning
67%
93%
COSC-1300-3428
INET
Distance Learning
60%
84%
On Campus
77%
89%
COSC-1401 Microcomp. Conc./Apps.
COSC-1401-3426
INET
Distance Learning
82%
88%
COSC-1401-3427
INET
Distance Learning
44%
80%
COSC-1401-3428
INET
Distance Learning
39%
78%
COSC-1401-3429
INET
Distance Learning
100%
100%
CVC Spring 2006 Distance Learning and On Campus Grade Distributions
ECON-2301 Prin Economics
On Campus
63%
80%
ECON-2301-3426
INET
Distance Learning
33%
33%
ECON-2301-3427
INET
Distance Learning
49%
72%
On Campus
66%
78%
Distance Learning
61%
61%
On Campus
58%
78%
ECON-2302 Prin Economics
ECON-2302-3426
INET
ENGL-1301 Composition I
ENGL-1301-3420
INET
Distance Learning
45%
55%
ENGL-1301-3421
INET
Distance Learning
50%
91%
On Campus
55%
80%
HIST-1301 History U S
HIST-1301-3420
INET
Distance Learning
63%
73%
HIST-1301-3421
INET
Distance Learning
57%
57%
On Campus
70%
85%
Distance Learning
59%
84%
On Campus
71%
81%
Distance Learning
36%
62%
On Campus
62%
83%
Distance Learning
56%
80%
On Campus
68%
84%
HIST-1302 History U S
HIST-1302-3420
INET
NUTR-1322 Prin of Nutrition
NUTR-1322-3426
INET
PSYC-2301 Intro Psychology
PSYC-2301-3420
INET
SOCI-1301 Intro Sociology
SOCI-1301-3420
INET
Distance Learning
81%
95%
SOCI-1301-3421
INET
Distance Learning
63%
92%
On Campus
71%
85%
Distance Learning
73%
87%
SPCH-1311 Intro Speech Comm
SPCH-1311-3420
INET
What Makes DE Work? . . .
INFRASTRUCURE!
A viable Distance Learning program
requires changes to the infrastructure of
campus to sustain learning outcomes
mandated by:
• College District
• State Regulations
• Accrediting Commission
• Federal Government—Title V
Infrastructure Changes
Testing Center
• Grew out of need to offer timed, proctored
testing for DE students to establish controls
for test taking to insure quality
• Meets regular effective contact requirements
to authenticate outcomes
• Now serves DE as well as the larger campus
community--academic makeup testing,
assessment testing, adapted testing
accommodation, job applicant testing,
computer skills competency testing.
Infrastructure—
Supporting Services of SLO’s in DE
• Distance Education Department
• Administrative, Clerical, and Technology
Support Staff
• Training for Students and New DE Instructors
• Orientation of New Students and New Faculty
• Online Services including registration and
tutoring
• DE Student Support staff (between technical &
counseling)
The End?
• DE was implemented long before and is
keeping pace with if not exceeding SLO
requirements for traditional courses
• Distance Education has clearly prompted the
college to reevaluate outcomes, has affected
how we teach in the traditional classroom,
and has effected a change in infrastructure to
meet needs in regards to outcomes.