Strengthening Student Success
Jerry Rudmann, Irvine Valley College
Pat Arlington, Coastline Community College
October 2007

 Technology Uses
 Technology Tools
Expected Outcome: Be able to select and use technology-based approaches to assess student learning outcomes

 Assessing Students in Large Classes
 Assessing Performance at a Distance
 Minimizing Subjectivity in Assessment
 Creating Authentic Assessments
 Engaging Students in Self-Evaluation
 Accommodating Diverse Learning
Styles
 Assessing Conceptual Thinking

 CCC Confer (Web Conferencing)
 Online Rubric Builders
 eLumen (SLO Assessment/Tracking)
 Calibrated Peer Review (CPR)
 Classroom Responders (“Clickers”)
 Scannable and Online Tests
 Cmap Tools (Concept Maps)
 ePortfolio

 Developed through faculty dialog
 Behavioral/measurable
 Real-world
 Higher-level
 Conditions
 Performance Criteria
 Global, over-arching
 Scored with rubric

 Small-group work in project-based learning
 Assessment of DL students

Foreign language classes

Speech/oral communication

Information literacy
 Score using discussion or presentation rubric http://www.cccconfer.org

CCC Confer Screen Shot

Characteristic/
Primary Trait
Rubric for Online Presentation logical sequence, around; audience making content had difficulty
Subject
Knowledge
Graphics
Technology
Poor
1 difficult to understand
Student does not demonstrate grasp of information and cannot answer questions about subject
Student uses superfluous or inappropriate graphics or no graphics
Student had significant problems with the presentation technology
Fair
2 following the presentation
Student appears uncomfortable with information and is able to answer only rudimentary questions
Student occasionally uses graphics that rarely support text and presentation
Student had some problems using the technology
Satisfactory
3
Presentation followed logical sequence that audience could follow
Student appears at ease with expected answers to all questions but fails to elaborate
Student’s graphics relate to text and presentation
Student used the technology well
Excellent
4
Presentation was well organized with logical, interesting sequence that audience could easily follow
Student demonstrates full knowledge (more than required) by answering all class questions with explanations and elaboration
Student’s graphics explain and reinforce screen text and presentation
Student used the technology with ease and to excellent effect
TOTAL
Score
0
Excerpted and adapted from Evaluating Student Presentations rubric by Education Development Center, Inc.: http://www.internet4classrooms.com/doc/PBL_germs.doc

 Tools

Integrated discussion forum in CMS or standalone (e.g., WebBoard by
O’Reilly)
 Uses

Practice/enrichment


Small-group project-based learning
To assess understanding – score with rubric

http://www.uas.alaska.edu/sitka/IDC/resources/onlineDiscussionRubric.pdf

 Rubrics to guide and measure learning
 Tools

Rubistar

Landmark Rubric Machine

Coastline Rubric Builder

Rubistar Art History Rubric

Coastline Rubric Builder

 Plagiarism
 Grading ( GradeMark )
 Peer Review
 GradeBook http://www.turnitin.com

Return to Turnitin
~ Next Slide

 Reduce Time Spent Creating Reports
 Assess Course, Program, and/or
Degree-Level Outcomes
 Share Assessment Rubrics Across
Classes and Programs
 View Individual or Aggregated Results
 Use Online or Offline http://www.elumen.info

 Rubrics are attached to each SLO
Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

 Writes prose clearly
Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

 “Link” a specific student activity
(e.g. project, paper, test) to the
SLO(s) for which the activity can serve as evidence
Student activity Student learning outcome(s)
Rubric
Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

 Here, one assignment stands as evidence for 3 different SLOs
Writes prose clearly
Rubric
Read and write a response paper for the novel A Lesson Before
Dying
Critically analyzes a text
Rubric
Considers ethical aspects of a situation or text
Rubric
Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Science and Gen Ed SLOs/Rubrics from the Science committee from the Biology Department from the faculty committee on critical thinking
Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007 from the faculty committee on communication skills

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

Excerpted from eLumen: A Brief Introduction by David Shupe, July 2007

 Web-based program that enables frequent writing assignments with minimal impact on instructor time
 Uses peer review
 Promotes deeper learning http://cpr.molsci.ucla.edu/

Calibrated Peer Review in Psychology 101

in Introductory Psych Course
SLO on Pseudoscience skepticism :
Students will correctly identify nonscientific explanations of human behavior and explain why those explanations are not based upon science and do not provide reliable or valid explanations of behavior or predictions of future behavior.

The Pseudoscience Belief Test
Please rate how much you believe the following statements. Use the 7-point scale provided.
1 – Do not believe in this at all.
2 – I doubt very much that this is real.
3 – I doubt that this is real.
4 – I am unsure if this is real or not.
5
6
7
– I believe that this may be real.
– I believe that this is real.
– I strongly believe this is real.
__ 1. A person’s personality can be easily predicted by their handwriting.
__ 2. A person can use their mind to see the future or read other people’s thoughts.
__ 3. A person’s astrological sign can predict a person’s personality and their future.
__ 4. An ape-like mammal, sometimes called Bigfoot, roams the forests of America.
__ 5. The body can be healed by placing magnets on to the skin near injured areas.
__ 6. Healing can be promoted by placing a wax candle in your ear and lighting it.
__ 7. A dinosaur, sometimes called the Lock Ness Monster, lives in a Scottish lake.
__ 8. Sending chain letters can bring you good luck; ignoring them can bring you bad luck.
__ 9. The government is hiding evidence of alien visitation at places such as Area 51.
__ 10. Voodoo curses are real and have been known to kill people.
__ 11. A broken mirror can bring you bad luck for many years.
__ 12. Houses can be haunted by the spirits of people who have died in tragic ways.
__ 13. Water can be accurately detected by people using “Y” shaped tree branches.
__ 14. Animals, such as cats and dogs, are sensitive to the presence of ghosts.
Adapted from…Walker, Hoekstra, & Vogl, (2002). Science education is no guarantee of skepticism, Skeptic, vol 9, no 3.

Critical Thinking Experiment
Using a SLO as the Dependent Variable
Calibrated Peer
Review Lesson on Graphology
Pseudoscience
Belief Pre-test
Randomly
Assigned 90
Students
Calibrated Peer
Review Lesson on Different
Topic
Pseudoscience
Belief Post-test

 Students read assignment
 Students read resource materials
 Students wrote a short essay in response to the materials: Why or why I believe graphology is a reliable, valid way to measure and predict personality.
 Students are “calibrated” – prepared to score essays written by their peers.
 Students receive a detailed grade report for the assignment.

Treatment
Group
Graphology
Conditioning
Statistical Summary
Pre-test
Average
4.41
4.12
t(51) = 0.67
p = ns
Post-test
Average
2.33
3.69
t(46.7) = 2.93
p < .01
Paired t-tests
t(26) = 6.40
p < .01
t(25) = 1.31
p = ns

Mean Pre and Post-Test Scores on Graphology Belief Question
3
2.5
2
1.5
1
0.5
5
4.5
4
3.5
0
Pre-test Post-test
Graphology
Conditioning

Example Essay
The Detection of a Pseudoscience: Graphology
Elaine Quigley’s posting on the website www.businessballs.com is littered with “red flags” that expose graphology as the pseudoscience/pseudopsychology that it is. While an attempt to promote graphology, Quigley’s posting fails to measure up to several of Cotton and Scalise’s guidelines for “baloney detection.” This paper will examine four areas in which graphology fails to live up to its claim of being “science.”In an attempt to display graphology’s validity, Quigley cites the notion that it is “a very old and respected science.” The fact that it has existed for approximately 3,000 years is used to justify Quigley’s notion that graphology is a science. However, one educated in the definition of science knows that the age of a theory is not a factor used to determine its validity. In fact, there are many beliefs that have been around for thousands of years that cannot be tested and therefore cannot be deemed as scientifically reliable. Graphology is just one of many ideas that cannot be justified despite their age. Quigley also fails to tell how the “science” of graphology has been tested and proven. Instead, she simply states that graphology is a “reliable indicator of personality and behavior” and expects her readers to accept this statement as fact. She also mentions that “the science is still being researched and expanded.” This is the extent to which she approaches the issues related to the research of graphology. Without explaining the testing that was done to prove the methods reliability, how is one to know that graphology is indeed reliable? Indeed, the answer is simple. It is impossible to be sure of the reliability of a measure of personality if the measure itself cannot be tested. In addition to not presenting methods for testing the claims of graphology, Quigley also fails to present evidence in support of its validity. Instead, she simply states that “it is not easy to explain how and why graphology works, nevertheless it continues to be used, respected and appreciated by many.” Could it be that the only “evidence” for the reliability of graphology is the satisfaction that its users experience? Unfortunately, being
“used” and “accepted” characteristics required of a science. Finally, the vast majority of information provided by
Quigley is anecdotal and leads up to a sales pitch for her services. She provides vague stories about how graphology has been used to produce more successful hiring processes and personal relationships. The information is presented more as an advertisement than a scientific work. Quigley goes into more detail on her experience as a graphologist than she does on the aspects of graphology that would qualify it as a science. In conclusion, it is quite clear that based on the evidence presented in this paper, graphology qualifies as a pseudoscience rather than a science. The claims of graphologist Elaine Quigley fail to show that graphology is indeed a science. Instead, she relies on the age of graphology and anecdotal evidence in support of graphology while ignoring issues related to methods for testing graphology’s claims and the results that have resulted in tests of its validity. Looking critically at “discoveries” is no doubt a useful tool that extends beyond the subject of graphology. The methods for recognizing pseudosciences compiled by Cotton and Scalise are certainly tools that would empower all people and prevent them from being fooled by pseudoscientific claims.

Questions and Answers for CPR Peer Reviewers
1. Did the essay begin with a topic sentence?
2. Was the essay free of spelling and grammatical errors?
3. Did the essay present at least four (4) different reasons for supporting or denying the validity of graphology (or handwriting analysis) as a method of assessing personality and/or predicting behavior?
4. Did the essay have balance? Although this may seem subjective, do you feel that it provided a balance among each of the points made? For example, was each point was explained in the same amount of detail.
5. Did the author's arguments seem convincing to you?
6. Did the author conclude with any reflection about whether this assignment was or was not helpful to his or her learning? In other words, did the author indicate that this assignment might help him or her judge the validity explanations of behavior encountered in the popular media (newspaper, radio, TV, magazines, etc.)?
7. How would you rate this text? (Scale of 1 – 10)

Student’s Screen: Detailed Results

Instructor Screen: Student Progress

Instructor Screen: One Student’s Results

Instructor’s Screen: Student Results
SLO Data

 Engage students
 Monitor student understanding
 Quickly and easily collect and store assessment data
 Use publisher item banks or create your own

PBS Demo

for clicker training resources http://www.renlearn.com

 Embedding Questions in Multiple
Sections and Classes
 Basing Multiple-Choice Questions on
Case Studies and Complex Scenarios http://www.scantron.com
and http://www.renlearn.com

 Online or Scannable
 Surveys

Pre and post surveys of student self evaluation of progress

Faculty and business community or advisory groups related to expected learning outcomes

Student satisfaction (indirect outcomes)
 Quizzes/Tests
Cross
Match

Practice and graded

http://cmap.ihmc.us/

 Provides graphical means for organizing and representing knowledge
 Addresses meaningful learning rather than simply rote learning
 Allows students to construct their own understanding of the content
 Provides opportunities for collaboration
 Provides alternative to essays or multiple-choice tests

High school student’s concept map about birds from http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMaps.htm

 Advantages

Document artifacts of learning

Support diverse learning styles

Authentic assessment

Course, program, or degree-level tracking

Job skill documentation
 Proprietary or Open Source
 ePortfolio and Open Source Portfolio

Lock Assignments after submission
Random selection of assignments by learning objective
Anonymity of the student who produced the assignment and the instructor
Access to the work and the scoring rubrics
Reports to aggregate scores; generate frequencies/means
Ability to download raw data which can be analyzed in another format http://www.eportfolio.org

 Aligned with
Sakai
 Admins or
Faculty can structure and review work
 Learning matrix documents levels of work http://www.osportfolio.org

 Calibrated Peer Review: http://cpr.molsci.ucla.edu/
 CCC Confer: http://www.cccconfer.org
 Class Climate: http://www.scantron.com
 Cmap Tools: http://cmap.ihmc.us
 eListen: http://www.elisten.com
 eLumen: http://www.elumen.info
 ePortfolios:

 ePortfolio.org: http://eportfolio.org
Open Source Portfolio: http://www.osportfolio.org/

For others, see EduTools ePortfolio product comparison: http://eportfolio.edutools.info/item_list.jsp?pj=16
 Hot Potatoes : http://hotpot.uvic.ca/
 Online Rubric Builders

Rubistar: http://rubistar.4teachers.org

Landmark Rubric Machine: http://landmark-project.com/rubric_builder/index.php

Coastline Rubric Builder: http://rubrics.coastline.edu
 Remark Survey Software: http://www.principiaproducts.com/web/index.html
 Renaissance Classroom Responders: http://www.renlearn.com/renresponder/
 Turnitin.com: http://www.turnitin.com

 Dr. Jerry Rudmann, Professor of Psychology
Irvine Valley College jrudmann@ivc.edu
 Pat Arlington, Instructor/Coordinator
Instructional Research
Coastline Community College parlington@coastline.edu