Integrating ICALL into
synchronous CMC
Markus Dickinson, Rebecca Sachs, Yunkyoung Kang,
Soojeong Eom, & Chong Min Lee
CALICO/IALLT Conference
March 20, 2008
San Francisco, CA
Intelligent CALL (ICALL)

Intelligent CALL, using natural language processing
technology, provides many promising means of
facilitating L2 development:
– Detailed information about learners’ L2 production errors can
foster awareness of language and encourage cognitive
comparisons
– Feedback can be precisely tailored to learners’ proficiency
levels, common mistakes, activity goals, learning styles,
cognitive abilities, etc.
– Specific improvements can be tracked across exercises
through the use of learner models
(Amaral & Meurers, 2006; Heift & Schulze, 2007; Nagata, 2002)
Limitations of ICALL

However, there is a tension between
The ability of an ICALL system to
provide meaningful, accurate
feedback
The flexibility an ICALL system
allows for in terms of meaningful,
communicative interaction
– Contextualized language use is increasingly emphasized in
ICALL, and with increasing success
(e.g., Amaral et al., 2006; Amaral & Meurers, 2006; Nagata, 2002)
– But actual communicative interaction remains relatively
unexplored in systems capable of providing feedback
(though see Petersen, 2006)

In order to manage computational complexity:
– Exercises are often restricted to the sentence level
– Activities often do not simulate true communication
Synchronous CMC

Synchronous CMC (computer-mediated
communication) between L2 learners can also
provide beneficial contexts for language development:
– Activities can approximate ‘target tasks’ relevant to real-life
communicative situations
– Learners can function as language users as opposed to
simply ‘displaying’ language or manipulating L2 forms
systematically without attending to meaning (Ellis, 2003)
– Possibly ‘optimal psycholinguistic environment’ for several
interactional features linked to L2 development (Doughty &
Long, 2003)
Pros of CMC

Potentially beneficial features
– Negotiation of lexical meaning (Blake, 2000)
– Attention to form, monitoring of linguistic output, selfcorrection (Salaberry, 2000)
– Incorporation of others’ feedback (Pellettieri, 2000)
– Use of more complex language (Warschauer, 1996)
– Drawing of form-meaning connections (Doughty & Long, 2003)
– Less pressing time constraints and less ephemeral language in
the written modality, reducing processing demands (Payne &
Whitney, 2002)
– Comfortable environment for expression of ideas (Warschauer,
1996)
Limitations of CMC

Concerns regarding the quality of learner-to-learner
interactions: The blind leading the blind?
– Without feedback from a trusted authority, learners might…



reinforce each other’s errors
not have the resources necessary for correcting each other
naturally tend to focus on vocab without attending to grammar
(Blake, 2000; Kern, 1995)
 Teachers may need to set explicit expectations for
grammatically correct language, while simultaneously trying
to balance this with a primary focus on meaningful
communication (Lee, 2001; Pellettieri, 2000)
Combining the benefits

Can ICALL and synchronous CMC be integrated in a
way that exploits the unique benefits of each while
avoiding their limitations?
Intelligent computer-generated feedback in synchronous,
task-based, computer-mediated learner-learner interactions
A balancing act

We think so, but… ICALL has certain limitations
– We want learners’ interactions to be as free as possible,
promoting authentic and meaningful communication
… But must constrain the communicative situation somehow
in order to make processing manageable
(cf. discussion in Amaral et al., 2006)
– We want to allow learners to make errors and then provide
them with detailed, informative computer-generated
feedback on nontargetlike language use
… But must reduce the amount of variability in learner input
so that accurate feedback can be given
Proposed study

Questions:
1. What design features will allow an ICALL
system to provide detailed, accurate,
individualized metalinguistic feedback on L2
errors in meaningful learner-to-learner CMC?
2. How can we make the system as user-friendly
as possible?
3. How effective will this sort of ICALL system be
in terms of promoting L2 development?
We would argue…

Feedback on grammar can be provided in
synchronous CMC without sacrificing an undue
amount of communicative authenticity, as long as
we can design natural ways of…
1. Controlling the activity specification
2. Controlling the range of learner input


Participants: English-speaking university students in
first-year Korean classes
Target of feedback: Korean postpositional particles
Activity specification

Dyadic picture-based spot-the-differences task
– Each participant will see one version of a house and have to
exchange information in the L2 in order to find similarities and
differences between the two pictures
– A ‘game record’ will provide additional guidance and
potentially increase motivation through including an element
of competition

Each participant must…
– Record the activities and locations of all characters in his/her
partner’s house using a provided chart
– Indicate whether each of these represents a similarity or difference

Following the activity, each dyad will be able to compare their
score to the average scores of other dyads
Picture 1
Picture 2
Activity specification
(cont.)

Guided and goal-oriented
– Constrains the vocabulary and domain, thereby
reducing many of the complexities involved in
generating feedback (computationally speaking)

‘Interactionally authentic’
– Perhaps not so authentic in terms of real-world
relevance, but the sort of task often used in
interaction research to target specific areas of
language and promote negotiation and L2 learning
Target of feedback:
Korean particles

Korean has relatively free word order, so postpositional
particles are used to indicate grammatical functions,
thematic roles (e.g., who is doing what to whom), and
the locations of people and objects.
Subject
I
나 가
Subject
Verb
Object
ate
pizza
어제 밤 에
Time
Location
Time
at a restaurant
음식점
Location
에서
last night
피자 를
Object
먹었다
Verb
Target of feedback:
Korean particles


Particles must be used even in simple sentences;
thus, they are taught from the beginning of L2
Korean study
However, the system is quite complex and difficult to
master for adult learners of Korean:
– Korean particles make distinctions not made in English
– Some verbs have different argument structures across the
two languages
– Several particles are ambiguous
 Particle errors account for a substantial proportion of
the mistakes made by beginning learners (Ko et al.,
2004), and errors persist even at advanced levels
Sources of difficulty for
native speakers of English

No one-to-one correspondence between
Korean locative particles and English
prepositions
Location of a
에 (‘e’) static object
에 (‘e’)
English ‘to’
에게 (‘ege’)
께 (‘kke’)
한테 (‘hante’)
English ‘in’
Location of a
에서 (‘eseo’) dynamic activity
Location an
을/를 (‘eul/reul’)* action moves
through
(*also an object marker)
Sources of difficulty for
native speakers of English


Korean has a topic marker, best translated as “as for” or
“speaking of X”, which English does not have.
The distinction between subject and object markers can be
confusing because some English transitive verbs are used as
intransitive verbs in Korean.
I (subject)
나는
I (topic)
need
a book (object)
책이
book (subject)
iss-ta (have)
pilyoha-ta (need)
choh-ta (like)
shil-ta (do not like)




필요해요
need
exist
be needed
be liked
not be liked
Examples of targetlike and
non-targetlike particle use

Intransitive verbs need a subject marker
주방-에 뭐-가 있어요?
주방-에 뭐-*를 있어요?
kitchen-LOC what-SUBJ is
kitchen-LOC what-OBJ is
‘What is in the kitchen?’

Transitive verbs need an object marker
아버지-가 고기-를 구어요.
아버지-가 고기-*가 굽고 있다.
father-SUBJ meat-OBJ grill
father-SUBJ meat-SUBJ is grilling
‘Father is grilling meat.’

A static location must be marked with a static locative marker
고양이-가 거실-에 있어요.
고양이-가 거실-*에서 있다.
cat-SUBJ living room-LOC is cat-SUBJ living room-LOC is
‘A cat is in the living room.’
(TL examples are on the left with correct particles ; non-TL are on the right with asterisks)
Other expected error types

Missing particles
내ø
어제 밤 ø
I

last night
고기 ø
meat
먹었다.
ate
Incorrect particles (morphology)
subject - i/ka
object - eul/rul
topic - un/nun
comitative - wa/kwa

음식점 ø
restaurant
sister (dongsang) + i/ *ka
rice (bab) + eul/*rul
elephant (kokkiri) + *un/nun
rice (bab)+ *wa/kwa +meat (gogi)
Incorrect particle order
음식점에는 (*는에)
누가
restaurant-Loc-Top
Who-Sub
‘As for a restaurant, who works (there)?’
일해요?
work-Ques
Can beginning learners
use CMC?



We can expect problems with particles regardless of
the communicative situation
Unconstrained tasks might be stressful or
frustrating for beginning learners, making it
important to sequence, guide, and scaffold tasks
appropriately (cf. Doughty & Long, 2003)
AND… Beginning learners of Korean do not yet
know how to type in Korean
– Since we wish to provide communicative practice with
particles, we need to ensure that the focus of the task
does not become that of simply inputting Korean
A possible solution

Word and particle banks
– Learners can select the tokens they wish to
use simply by clicking on words and
particles
– For some morphophonological alternations,
the system will transform some adjacent
characters where necessary
Misspellings will be less of an issue
 Learners will be given positive evidence of
particle attachment in Korean

The interface







Spot-the-differences picture
Word and particle banks
Sentence drafting area
‘Check’ and ‘Send’ buttons
Feedback-providing avatar
Chat window
Game record
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
CHARACTER
오빠가 소파를 책을 읽어요.
SEND
Word Bank
Particles
침대
화장실
에
는
이
의자
거실
을
책
읽었어요
에서 가
…
소파
울었어요
ACTIVITY
Mother
S
D
S
D
Father
S
D
S
D
주방 S
D
읽다 S
D
S
D
S
D
Grandma
CHECK
LOCATION
…
In your sentence, 소파 is
marked with the particle
를, which suggests that
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
TASK
PICTURE:
[ParticipantA]:
오빠가
어디에 있어요?
[ParticipantB]:
거실에 있어요.
Partners
have
[ParticipantA]: 오빠가 거실에서 뭐해요?
slightly different
[ParticipantB]: …
versions and must
communicate to find
differences. They can
scroll over the
picture to enlarge it.
오빠가 소파를 책을 읽어요.
CHARACTER
SEND
Word Bank
Particles
침대
화장실
에
는
이
의자
거실
을
책
읽었어요
에서 가
…
소파
울었어요
ACTIVITY
Mother
S
D
S
D
Father
S
D
S
D
주방 S
D
읽다 S
D
S
D
S
D
Grandma
CHECK
LOCATION
…
In your sentence, 소파 is
marked with the particle
를, which suggests that
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
CHARACTER
오빠가 소파를 책을 읽어요.
SEND
Word Bank
Particles
침대
화장실
에
는
이
의자
거실
을
책
읽었어요
에서 가
…
소파
울었어요
ACTIVITY
Mother
S
D
S
D
Father
S
D
S
D
주방 S
D
읽다 S
D
S
D
S
D
Grandma
CHECK
LOCATION
…
In your sentence, 소파 is
marked with the particle
를, which
suggests
that
WORD
& PARTICLE
소파 is an object.
BANKS:
Instead, you need the
To를create
a to
sentence,
particle
attached
소파 inparticipants
order to indicateclick on
that 소파
is theand
location
words
particles…
of a dynamic activity.
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
CHARACTER
…which
Mother
오빠가 소파를 책을 읽어요.
CHECK
SEND
Word Bank
Particles
침대
화장실
에
는
이
의자
거실
을
책
읽었어요
에서 가
…
소파
울었어요
LOCATION
then
S D
appear in the
Father
S D
sentence drafting
Grandma
주방 S D
area. S D
…
In your sentence, 소파 is
marked with the particle
를, which suggests that
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
ACTIVITY
S
D
S
D
읽다 S
D
S
D
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
CHARACTER
LOCATION
Mother
S D
If they want
help with
Father usage,
S D
Korean particle
오빠가 소파를 책을 읽어요.
Grandma
they can
request 주방 S D
S D
… on their
CHECK
SENDfeedback
sentences
before
In your
sentence, 소파 is
Word Bank
Particles
marked
the particle
entering them
intowith
the
를, which suggests that
침대
화장실
에
는 이conversation.
의자
거실
책
읽었어요
소파
울었어요
에서 가
…
을
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
ACTIVITY
S
D
S
D
읽다 S
D
S
D
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
CHARACTER
오빠가 소파를 책을 읽어요.
Word Bank
SEND
Particles
FEEDBACK AREA: Here,
침대participants
화장실
에
는 이
receive
에서
가 을
의자metalinguistic
거실
feedback
advice on …
particle
책 with읽었어요
울었어요
소파usage.
ACTIVITY
Mother
S
D
S
D
Father
S
D
S
D
주방 S
D
읽다 S
D
S
D
S
D
Grandma
CHECK
LOCATION
…
In your sentence, 소파 is
marked with the particle
를, which suggests that
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
CHARACTER
오빠가 소파를 책을 읽어요.
CHECK
Particles
침대
화장실
에
는
이
의자
거실
을
책
읽었어요
에서 가
…
소파
울었어요
ACTIVITY
Mother
S
D
S
D
Father
S
D
S
D
When주방
theyS
Grandma
SEND
Word Bank
LOCATION
are
D ready,
읽다 S
they click
to S
S SEND
D
…
enter their utterance
In your sentence, 소파 is
into the conversation.
marked with the particle
를, which suggests that
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
D
D
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
CHARACTER
오빠가 소파를 책을 읽어요.
LOCATION
Mother
S
D
S
D
Father
S
D
S
D
주방 S
D
읽다 S
D
S
D
Grandma
S D
CHAT WINDOW:
They can
scroll up…and down
CHECK
SEND
to review the conversation so far. In your sentence, 소파 is
Word Bank
Particles
침대
화장실
에
는
이
의자
거실
을
책
읽었어요
에서 가
…
소파
울었어요
ACTIVITY
marked with the particle
를, which suggests that
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
[ParticipantA]: 오빠가 어디에 있어요?
[ParticipantB]: 거실에 있어요.
[ParticipantA]: 거실에서 뭐해요?
[ParticipantB]: …
GAME RECORD: When
participants find
similarities or differences,
they drag the relevant
words for locations and
activities
to record
오빠가
소파를here
책을 읽어요.
information about their
partners’
pictures, then
CHECK
SEND
click on ‘S’ or ‘D’ to
Word Bank
indicate
whether theParticles
침대
화장실
에
는 이
pictures
match in those
respects
or not. 에서 가 을
의자
거실
책
읽었어요
소파
울었어요
…
CHARACTER
LOCATION
ACTIVITY
Mother
S
D
S
D
Father
S
D
S
D
주방 S
D
읽다 S
D
S
D
S
D
Grandma
…
In your sentence, 소파 is
marked with the particle
를, which suggests that
소파 is an object.
Instead, you need the
particle 를 attached to
소파 in order to indicate
that 소파 is the location
of a dynamic activity.
Is processing feasible?

Learners’ sentence construction is guided by…
– The nature of the picture-based task
– Instructions and the game record
– Word and particle banks, which…




Limit the types of argument structure by limiting the
verbs used
May be necessary for beginning learners who can’t type
in Korean
May serve as a scaffold for using receptive vocab in
conversation
Intensive feedback is provided on one
particular error type
Upshot

Synergy between pedagogical and
computational constraints
– Beginning learners will feel comfortable
communicating meaningfully in the L2 (with
familiar content and sufficient guidance)
– The learners can still make mistakes in the L2
while attempting to express themselves
– ICALL processing can focus just on detecting
particle errors in a known domain
How can we detect illformed sentences?

A combination of techniques will ultimately be used to
feed into an error diagnosis module
– Linguistic processing will be kept separate from error
detection/diagnosis and feedback generation
– Since general relations between elements of the task pictures
are fixed, fairly traditional anticipation-based pattern
matching (i.e., regular expressions) could be used
– This will need to be augmented with basic linguistic
abstraction (part-of-speech tags and syntactic chunks)


Partial parsing methods are extremely robust & provide
information even when a full syntactic parse is not possible
Linguistic abstraction ensures applicability to new exercises
Opportunity to experiment
with different techniques

Particle errors will often result from a mismatch
between the argument relations of the sentence and
the morphological forms used by the learners
 Could use multiple parsing models to check for mismatches
(cf. Metcalf & Boyd, 2006)




One parser captures particle usage patterns from real language
Another parser captures general argument structure patterns
between words, irrespective of particles
Currently exploring this & other techniques (Dickinson & Lee,
2008)
Regardless of the techniques, generating learner data
will provide evaluation material to help advance the
state-of-the-art in processing Korean learner input
Is focused feedback
beneficial?

Some have argued that intensive feedback on one
pre-selected error type may be more effective in
certain contexts than wide-ranging incidental feedback
on a variety of errors
(e.g., Lyster, 1998; Nicholas, Lightbown, & Spada, 2001)

In our study, we will inform learners that they will be
receiving feedback only on particles
– Important for meaning (i.e., communicating who is doing
what to whom) in Korean
– Will hopefully prevent them from mistaking non-feedback for
correctness
– Leaves open the possibility of providing other feedback, if
needed
Is meaningful
communication promoted?

Does this set-up truly represent ‘synchronous
CMC’ as it is commonly conceptualized?
How much will the learners focus on meaningful
communication if it is clear that the feedback is
focusing exclusively on Korean particles?

Particles are crucial to expressing and understanding
meaning in Korean sentences; thus, the ostensibly
grammar-oriented feedback should facilitate
communication
How can the word and particle banks be made
sufficiently rich for the participants’ communicative
purposes?
The importance of piloting




Picture-based tasks have been used successfully in
other experimental CMC research as a means of
guiding content and controlling amount/type of
feedback (e.g., Sachs & Suh, 2007), but learners were
already proficient typists in the L2.
Will beginning learners be capable of interacting
smoothly in the current context?
What sorts of scaffolding will they actually need?
What can we do to make the banks as easy to use
(and as facilitative of L2 development) as possible?
Future directions

Pilot the tasks and competitive game component with
L2 learners
– Get a clearer sense of what to expect in learner input
– Test how the word and particle banks are actually used

Develop the system in modular fashion, ensuring it
will be extendible to other Korean language activities
– Activity model (indicating expected constructions and words
for the word bank)
– Expert model (for linguistic analysis)
– Error diagnosis module
– Feedback module
(cf. TAGARELA: Amaral & Meurers, 2006)
Future directions (cont.)



Develop activities to target more areas of language
Make the tasks more complex, meaningful, and
relevant to real-life communicative situations
Use this set-up to test questions of SLA theory and
language pedagogy
– Assess L2 development under different feedback conditions
(e.g., metalinguistic info vs. recasts) with pre-test/post-test
experimental designs
– Investigate optimization of feedback for different areas of
language, proficiency levels, aptitude profiles, etc.

Integrate this system with the Korean language
curriculum at Georgetown
Questions?
Comments?
Please email:





Markus Dickinson
Soojeong Eom
Yunkyoung Kang
Chong Min Lee
Rebecca Sachs
md7@indiana.edu
se48@georgetown.edu
yk95@georgetown.edu
cml54@georgetown.edu
rrs8@georgetown.edu
References


Amaral, L., & Meurers, D. (2006). Where does ICALL fit into foreign language
teaching. CALICO 2006. University of Hawai’i.
Amaral, L., Metcalf, V., & Meurers, D. (2006). Language awareness through reuse of NLP technology. Pre-conference Workshop on NLP in CALL – Computational
and Linguistic Challenges. CALICO 2006. University of Hawai’i.

Blake, R. (2000). Computer-mediated communication: A window on L2 Spanish
interlanguage. Language Learning & Technology, 4(1), 120-136.

Chapelle, C. (2003). English language learning and technology. Philadelphia: John
Benjamins.

Dickinson, M. & Lee, C.M. (2008). Korean Particle Error Detection via Probabilistic
Parsing. Workshop on “Automatic Analysis of Learner Language” at CALICO-08.
San Francisso.

Doughty, C.J., & Long, M.H. (2003). Optimal psycholinguistic environments for
distance foreign language learning. Language Learning & Technology, 7(3), 50-80.

Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford
University Press.

Heift, T., & Schulze, M. (2007). Errors and intelligence in computer-assisted
language learning: Parsers and pedagogues. Routledge.
References

Kern, R.G. (1995). Restructuring classroom interaction with networked computers:
Effects on quantity and characteristics of language production. The Modern
Language Journal, 79, 457-473.

Ko, S., Kim, M., Kim, J., Seo, S., Chung, H., & Han, S. (2004). An analysis of
Korean learner corpora and errors. Hanguk Publishing Co.

Lee, L. (2001). Online interaction: Negotiation of meaning and strategies used
among learners of Spanish. ReCALL Journal, 13(2), 232-244.

Lyster, R. (1998). Form in immersion classroom discourse: In or out of focus?
Canadian Journal of Applied Linguistics, 1, 53-82.



Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and
research synthesis. In A. Mackey (Ed.), Conversational interaction in second
language acquisition: A series of empirical studies (pp. 407-452). Oxford: Oxford
University Press.
Metcalf, V., & Boyd, A. (2006, December). Head-lexicalized PCFGs for verb
subcategorization error diagnosis in ICALL. Workshop on Interfaces of Intelligent
Computer-Assisted Language Learning. The Ohio State University.
Nagata, N. (1995). An effective application of natural language processing in
second language instruction. CALICO Journal, 13(1), 47-67.
References

Nicholas, H., Lightbown, P.M., & Spada, N. (2001). Recasts as feedback to language
learners. Language Learning, 51, 719-758.

Payne, J.S., & Whitney, P.J. (2002). Developing L2 oral proficiency through synchronous CMC:
Output, working memory, and interlanguage development. CALICO Journal, 20(1), 7-32.





Pellettieri, J. (2000). Negotiation in cyberspace: The role of chatting in the development of
grammatical competence in the virtual foreign language classroom. In M. Warschauer & R. Kern
(eds.), Network-based language teaching: Concepts and practice (pp. 59-86). Cambridge: CUP.
Petersen, K. (2006, December). Measuring L2 development in an ICALL context. Workshop on
Interfaces of Intelligent Computer-Assisted Language Learning. The Ohio State University.
Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of
L2 grammar. In J.M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and
teaching. Amsterdam: John Benjamins.
Sachs, R., & Suh, B-R. (2007). Textually enhanced recasts, learner awareness, and L2
outcomes in synchronous computer-mediated interaction. In A. Mackey (Ed.), Conversational
interaction in second language acquisition: A series of empirical studies (pp. 197-227). Oxford:
Oxford University Press.
Warschauer, M. (1996). Comparing face-to-face and electronic discussion in the second
language classroom. CALICO Journal, 13(2), 7-26.