David Allbritton
Elizabeth Arnott
Peter Wiemer-Hastings
Jesse Efron
DePaul University http://reed.cs.depaul.edu/peterwh/RMT

Talk presented at the annual meeting of the
Society for Computers in Psychology,
November 2003, Vancouver BC, Canada.

The concepts and skills taught in psychology research methods courses are fundamental to our field, yet are often difficult for students to master.
RMT (Research Methods Tutor, a descendent of
Autotutor) is an intelligent tutoring system that will supplement classroom instruction by engaging students in conversation about course topics, using LSA (latent semantic analysis) to understand student responses. Pedagogical strategies implemented in RMT are described, along with plans for evaluation and deployment.

• The skills taught in research methods are fundamental to psychological science
• Students have difficulty mastering these fundamental concepts in the classroom
• Tutoring can help

• Flexibility
• Effectiveness
– 2 Sigma (Bloom, 1984)
– the “Holy Grail” of ITS Research

• Practicality
• 24/7 Availability
• Lowering Social Barriers
• Low Cost ( after development, that is)

• Representation of Domain Knowledge
– Deep or Shallow?
• Student Model
– Deep or Shallow?
• Interface
– Constrained (Menus) or Natural Language?

• Didactic vs. Socratic
• Conceptual vs. Problem-Oriented
• Conversational vs. Pedagogical
• Collaborative vs. Competitive
• Authentic vs. Artificial Task Setting

• Comprehension (Clarity)
• Motivation
• Appropriately Challenging (ZPD)

• Architecture
• Pedagogy
• Implementation
• Evaluation
• Deployment

• LSA for semantic representation
– Word meanings are vectors in a semantic space
– Sentences are “bags of words”
– Cosine between vectors defines similarity of meaning
• Dialog Transition Network to guide dialog moves
–
Example of DTN
• Shallow knowledge of domain
• Shallow student model

• Conversational
• Authentic Task
– “You have just been hired as a research assistant. Your first task is to design a study..”
• Three Levels of Questions / Problems
– Conceptual
– Analytic
– Synthetic

• Why are studies conducted outside of the lab more likely to have problems with internal validity?
• Model Answer:
– Because the experimenter has more control over conditions in the lab.
– Unanticipated confounds are more likely in a field setting.

• Now I’ll give you a description of an experiment, and ask you to evaluate the validity of the experiment.
(..description of experiment…)
• The concept of internal validity in research means that changes in the dependent variable are a direct result of changes in the independent variable, and not of some other variable. What are some ways that other things besides the lighting may have influenced the test scores?

(Cont.)
• Model Answer
– Students took the exam at different times of the day.
– One group is older than the other.
– The two grades have different teachers.

• Now, I am going to describe an experiment, and you are going to help me set it up.
• An experimental psychologist believes that plant owners who interact with their plants will be happier than plant owners who do not. She identifies individuals who have never owned a plant and tests their happiness level using the Happiness Inventory. She then divides the non-plant owners into two groups: treatment and control. Describe how she should assign the participants to the two groups, what she should do to each group, and what each group should be told to do.

(Cont.)
• Model Answer
– Participants should be randomly assigned to two groups.
– The treatment group should receive a plant.
– The treatment group should be instructed to interact with the plant.
– Control group should receive a plant but not be instructed to interact with it.
– Both groups should be asked to return after a period of time to be re-tested on the Happiness Inventory.

• Interface
– Tutor is animated avatar
– Text-only version for comparison
– Student types responses in a text box
• Curriculum Script

• Topics
– Validity, reliability, ethics, types of research, experimental design
• XML schema defines structure of script
– Questions
– Model Answers
– Prompts, hints, and splices
– Dependencies

• Usability and Pilot Testing
• Experimental Evaluation of Tutor Features
– RMT with avatar vs. text-only interface
– RMT with authentic task vs. traditional tutoring
– RMT vs. web-based tutorials

• One of our avatars’ voice needs work:
– “The animation was fine, but the woman’s voice was annoying.” (“Miyako” avatar)
• The “Merlin” avatar was well-liked:
– “I liked his facial expressions.”
– “He was funny looking. Relieved stress in a way.”
– “The animation was cool. The speech was monotonous.”

• “It was easy because he asked a question and in case you got it wrong, he will help you to get to the right answer.”
• “It didn’t just give you the right answer if you got it wrong.”
• “It was creative and forces you to be clear with your ideas.”

(N=26)
• Things they particularly liked:
– The feedback
– The hints and prompts
– Multiple chances to get the right answer
– Various ways of asking questions
– Being led to the right answer
– The Merlin animation
• Suggestions for improvement:
– Mostly technical problems (esp. speech, agents)

• Usability Testing (now)
• Experimental implementation in selected course sections (soon)
• Available for all research methods courses at DePaul (eventually)

• Lower development cost than expert system
• “Cheap” way of doing NLP
• Works well for lists, on an all-or-nothing basis

• No domain knowledge – only as good as the curriculum script
• Natural language understanding is only approximate
– No syntax: “IV causes DV” = “DV causes IV”
– “Heavy” words can overpower rest of sentence
• Not good at figuring out how much two sets overlap (only good at all-or-nothing matches)

http://reed.cs.depaul.edu/peterwh/RMT