Dorothy Chun
UC Santa Barbara
 Why
do it?
1. To be able to say with greater
certainty, beyond anecdotal
evidence, that your efforts are
having an effect (hopefully
positive!).
2. To conduct formative and
summative evaluation.
 Today’s
presentation is framed in
terms of my experience with 2
research projects:
◦ ICE (UCSB-Uni Kassel, Germany)
based on Cultura
◦ Chin 332 project w/ Stephen & David
 Both
began w/o research
questions.
 1st
round: ICE added to curriculum
so only online questionnaires and
forums
 In writing a chapter for Teaching
with Technology we discovered
that students were missing a lot
of subtle differences
 E-mail exchanges vs. forums
 2nd
round: different partner
group with unequal numbers
 Small forum groups with 6/group
 Students reported back to the
class: mismatch between what
they said in forums and in class
 Analysis of classroom discourse
using Byram’s ICC disappointing
 3rd
round: different format (ABC)
due to partner’s request
 Teacher-chosen topics did not
work as well as student-chosen
topics
 1.
What would you like your
students to learn from the cafés?
 2. What kinds of evidence would
show that they have learned it?
 3. How can you find or obtain
this kind of evidence?
 What
are the 3-4 most important
things you want your students to
learn from this exchange?
 Take a moment to write them
down.
 How will you define success?
 Build
a community
◦ Intercultural exchange (including
business culture)
◦ Linguistic community
 Affective:
Sense of solidarity
 Linguistic: Cohesion in the
discourse
 Expanded
vocabulary
 Grammar practice
 Additional chances to use the L2
 Language variation
 Authentic language in the real
world, beyond the textbooks
 How
others think
 How others act/behave in social
contexts
 How things work in daily life in
the other’s culture
 Openness, awareness, curiosity
about other culture
 Examine
cultural assumptions
about the other culture
 Local adaptation of the other
culture
 Reflect
on your own culture
 Grapple with identity issues
 Qualitative
vs. Quantitative
 Qualitative:
◦ Often, but not always longitudinal,
descriptive, e.g., pedagogical best
practices
 Quantitative:
◦ Often, but not always crosssectional, “treatment” studies
 Teacher
journals: record
successes as well as frustrations
(e.g., use the facilitator forums as
data)
 Questionnaires: ask students what
they thought they learned, how
they liked the project
 Data
analysis: examine actual
“data” produced by the students,
e.g., in the forums
 Learning outcomes: linguistic
progress, intercultural
competence
 Be
specific about the outcomes
you are most interested in (and do
not try to solve the world’s
problems!).
 Different methods are needed for
different goals/questions.
 Collect data systematically.
 Collect
as much demographic
data as possible:
◦ Age
◦ Gender
◦ First language (L1)
◦ L2 exposure
◦ Extent of language use (L1 and L2)
◦ Contexts of language use (family,
school, work)
about all of the other
variables that could affect
learning outcomes (level, prior
knowledge, motivation, time on
task, etc.).
 Try to find a way to ensure that
these other variables are kept the
same or are noted.
 Think
It’s better to have more data than
you think you need than less (save
everything!).
 Combine qualitative + quantitative:
Study learning outcomes in addition
to students’ perceptions about what
they thought they learned or their
attitudes towards the cafés.

 During
the project, by keeping
good records, including recording
communication between both
sides, you can identify what works
and what doesn’t.
 You can then change/improve
your strategies
 Experiment
with sequencing
 Experiment with prompts and
instructions
 Compare online vs. offline tasks
 Compare use of different media
 Compare use of different online
tools
 For
what specific purpose are
each of the following used for?
 Is the use of the medium effective
for your goal?
◦ Text
◦ Graphics
◦ Video
◦ Audio
 For
what specific purpose are
each of the following used for?
 Is the use of the tool effective for
your goal?
◦ BRIX activities
◦ Wikis
◦ Skype, videoconferencing
 Which
types of teacher behavior
are successful?
 Where (in which activities), when (at
what point in the activity), how often
(whenever there is a “lull”?) should
teacher participate?
 Look
at forums to see whether:
◦ Students address each other
◦ Students address the teacher
◦ Students respond to others’ postings
(cohesion)
◦ Students use different types of
speech acts (state opinions, inquire,
apologize, request clarification,
discuss)
 Look
at forums to see if/whether:
◦ Students only state their opinions
◦ Students only talk about L2 and C2
◦ Students reflect on their L1 and C1
◦ Students express awareness of
similarities and differences between
C1 and C2
◦ Students express understanding of
C2
 Look
at data to see how
◦ Students demonstrate understanding
of the other culture
◦ Students’ attitudes, perceptions,
interactions have changed over the
course of the exchange (note that it’s
not always easy to know whether
anything has changed)
 Look
at data to see
◦ How knowledge is co-constructed
and what it looks like
◦ What causes interaction
◦ Whether students speculate and
hypothesize (rather than just make
statements or ask questions)
 Action
research can be done in
the classroom and with the online
cafés.
 You do not have to do a rigorous
experiment with treatment groups
(and statistical analyses).
 You
do need to have specific
research questions (targeted
outcomes).
 You do want to try and control for
as many other contributing
variables as possible.
 Doing
action research can help
both in formative evaluation
(seeing what works and what
doesn’t during the project) and in
summative evaluation (seeing
whether the original learning
goals have been met at the end of
the project).
 Have
fun with your impressive and
exciting projects!
 Mahalo
nui loa