𝑇ℎ𝑒𝑠𝑖𝑠 𝑃𝑟𝑜𝑝𝑜𝑠𝑎𝑙:
"𝐶𝑜𝑛𝑛𝑒𝑐𝑡𝑖𝑜𝑛𝑖𝑠𝑡 𝑚𝑜𝑑𝑒𝑙𝑙𝑖𝑛𝑔 𝑜𝑓 𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐 𝐿𝑎𝑛𝑔𝑢𝑎𝑔𝑒 𝐼𝑚𝑝𝑎𝑖𝑟𝑚𝑒𝑛𝑡"
𝑇ℎ𝑒𝑚𝑒𝑙𝑖𝑠 𝐾𝑎𝑟𝑎𝑚𝑖𝑛𝑖𝑠
𝑆𝑢𝑝𝑒𝑟𝑣𝑖𝑠𝑜𝑟𝑠: 𝐷𝑟. 𝑀𝑖𝑐ℎ𝑎𝑒𝑙 𝑇ℎ𝑜𝑚𝑎𝑠 & 𝐷𝑟. 𝑅𝑖𝑐ℎ𝑎𝑟𝑑 𝐶𝑜𝑜𝑝𝑒𝑟
𝐽𝑢𝑙𝑦 9 , 2008
1. Introduction
The primary objective of this thesis is to investigate language development in children with
Specific Language Impairment using connectionist modelling. Specific Language Impairment
(SLI) is a developmental disorder diagnosed when children fail to develop age-appropriate
language in the absence of factors which are usually concomitant with language learning
problems (such as hearing impairment, frank neurological damage or low non-verbal IQ test
scores) (Leonard, 1998). SLI has a prevalence of ~7% in the children population (Tomblin,
Records, Buckwalter, Zhang, Smith & O’Brien, 1997); language development in SLI has been
studied extensively for English, and to a lesser extent for other languages. Studies report a
wide range of deficits in language use and learning (in phonology, morphology, grammar,
syntax, semantics or pragmatics), indicating that the disorder is characterised by a greatly
heterogeneous profile. This is possibly a confounding factor in empirical investigations of
the causes of SLI, as different studies offer rather different theoretical explanations for SLI.
As we will see, for some researchers, SLI (or certain subtypes of SLI they consider) stems
from a deficit in the brain systems specifically involved in language processing. For others,
SLI should be attributed to a more general processing deficit, which is not specific to
language. There is, so far, no consensus on the question of whether SLI originates from a
language-specific deficit, and no unified account for the many aspects of the heterogeneous
profile of the disorder.
This research project addresses main issues in SLI research with the connectionist
modelling methodology. Connectionist or artificial neural network models are based on
computations performed by neuron-like units which are interconnected with modifiable
weighted connections. The most interesting property of this class of models is their ability to
learn. For example, when a supervised learning regime is used, connectionist models can be
trained to learn complex mappings between the input and the output (target)
1
representations of a training set. Furthermore, the learning trajectory in connectionist
models is traceable. Connectionist models allow monitoring of various features of the
learning process, such as activation and weight patterns (representations), accuracy and
error types on the mappings of the training set, as well as the ability of the model to
generalise its knowledge on novel mappings. Finally, as connectionist models are
parameterisable, one can study how different parameter settings affect the learning
trajectory. The ability to demonstrate learning, as well as the possibility to manipulate the
parameter settings and trace the learning process are considered to be powerful features of
the connectionist modelling methodology. These features render connectionist models a
valuable computational tool for addressing issues of (language) development (Thomas &
Karmiloff-Smith, 2002, 2003).
The current project proposes a systematic investigation of language development in SLI
using connectionist modelling. In particular, this project seeks connectionist explanations for
SLI, so as to contribute towards a better understanding of the impairment. For this purpose,
the project will contrast typical and atypical language development using a series of
connectionist models for different aspects of language development. Search into the
parameter space will be performed in order to detect parameter settings under which the
models simulate the corresponding behavioural data (accuracy rates, error patterns, etc).
Manipulations of the parameter settings that cause the models to shift from a typical
developmental trajectory to an atypical one will designate possible connectionist
explanations of SLI, which will be contrasted to theoretical accounts of SLI.
A second aim of this project is to address the heterogeneous profile of SLI. This will be
achieved by focusing on a wide range of issues of language development. A series of
models, addressing phenomena in phonology, morphology, syntax and semantics for both
typical and atypical development will be implemented. The modelling work will therefore
endeavour to capture a significant portion of the heterogeneity of symptoms in SLI. The
intended outcome is a unified connectionist explanation for the variability in SLI.
Finally, this project aims to adopt a cross-linguistic perspective, considering SLI in
English, and also in (Modern) Greek. The project will examine whether the same
connectionist models could account for atypical language development in two languages
which possess different characteristics (e.g. in their morphological richness). In order to
achieve this, the certain models initially developed for English will be subsequently
expanded to Greek.
This document briefly reviews the relevant SLI literature (behavioural data, theoretical
accounts) and introduces connectionist modelling from the perspective of the study of
atypical language development. This background information provides the wider context
and the rationale for the specific modelling work to be undertaken. In the research
methodologies section, a series of models to study various aspects of language
development in SLI is proposed. Finally, a detailed timetable of how this work will be
accomplished is presented.
2
1.1. SLI literature
In the literature, SLI has also been referred as ‘developmental dysphasia’,
’developmental aphasia’, ‘word deafness’, ‘delayed speech’, ‘deviant language’, ‘language
disorder’, ‘delayed language’, ‘developmental language impairment’, and ‘specific language
deficit’ (Leonard, 1998). The variety of these terms reflects the shaping of the diagnostic
criteria for the impairment along the history of the study of SLI (cf. Leonard, 1998, pp.5-8),
as well as the existence of different views of the profile of SLI. For example, a ‘delay’ view of
SLI suggests that the impairment is characterised by “late emergence of language and ...
slower than average development of language from the point of emergence to mastery”
(Leonard, 1998, p.32). By contrast, a ‘disorder’/’deviance’ view proposes qualitative
differences between the linguistic profiles of children with SLI and typically developing
children. The term ‘Specific Language Impairment’ is nowadays most broadly used. As
Bishop (1997, p. 21) comments, this term denotes that, apart from language, (nonverbal)
cognitive development falls in the normal range, while being neutral regarding the
delay/disorder question.
SLI is diagnosed based on both inclusion and exclusion (Leonard, 1998, p.10). The
linguistic and non-linguistic skills of children are assessed on standardised tests. Exclusionary
criteria (e.g. sensory/ hearing/ oral motor deficits, frank neurological damage, behavioural/
emotional problems, environmental causes), are considered to rule out cases in which the
impaired linguistic profile may be due to other factors.
1.1.1. Variability of deficits in SLI
Numerous studies have investigated the cognitive profile of children with SLI in
greater detail than that provided by the diagnostic tests. Such studies often use linguistic
and non-linguistic tasks to compare the performance of Children with SLI and typically
developing children (controls) matched on chronological age, mental age, mean length of
utterance (MLU), expressive/receptive vocabulary or other criteria (Leonard, 1998).
Extensive listing of findings from behavioural studies of SLI are provided in Leonard (1998,
chapter 2) and Bishop (1997).
Auditory and phonological deficits are prevalent in SLI, as poor performance has
been reported in tone discrimination (Tallal & Piercy, 1973a, b), phoneme discrimination
and identification of phoneme constancy (Bird, Bishop, & Freeman, 1995), or non-word
repetition tasks (Gathercole & Baddeley, 1990). Morphological/grammatical deficits, such as
omission of noun and verb inflections and auxiliary verb forms in obligatory contexts are
characteristic of most of children with SLI and are often considered as behavioural hallmarks
for the impairment (cf. Rice, 2000). Children with SLI present immature sentence structure
(Johnston & Kamhi, 1984) and many grammatical/syntactical deficits, for example in whquestion formation (e.g. Ingram,1972; van der Lely & Batel, 2003). Grammatical/
syntactical comprehension is also impaired, with deficits in anaphoric pronoun resolution
3
(van der Lely & Stollwerk, 1997), comprehension of passives (Bishop, 1979; van der Lely &
Harris, 1990; Dick, Wulfeck, Krupa-Kwiatkowski, & Bates, 2004). Lexical abilities are limited,
and are characterised by word-finding difficulties (e.g. McGregor & Leonard, 1995; German,
1987), naming errors (e.g. Casby, 1992 ; McGregor & Waxman, 1995), impoverished worddefinitions (Dockrell, Messer, George, & Ralli, 2003) and problems in the learning of new
words (Oetting, Rice, & Swank, 1995). Furthermore, certain studies have suggested
pragmatics-related difficulties, such as deficits in producing speech acts (Gallagher & Craig,
1984) or reduced conversational participation (Craig, 1993). Finally, weaknesses in certain
non-linguistic tasks (e.g. in mental imagery tasks (Johnston & Ramstad, 1983)) were also
identified.
Therefore, a wide range of deficits, which spread along all the dimensions of
language, characterises SLI. When individual performance is considered, it is not the case
that all children present the same deficits. On the contrary, it is generally agreed that there
is a great degree of individual variability within SLI. SLI is commonly characterised as a
developmental impairment with a “heterogeneous” or “uneven” profile (e.g. Ulman and
Pierpont, 2005; Thomas, 2005).
The heterogeneous profile of SLI, has led certain researchers to consider subgroups
within SLI, using statistical sorting procedures or based on clinical judgements (e.g. Aram &
Nation, 1975; Rapin & Allen, 1987; Wilson & Risucci, 1986). Leonard (1989) comments that
although certain subtypes of SLI were consistently identified by most of these studies,
further investigation is needed for them to be validated. Of particular interest is perhaps
Grammatical-SLI (abbreviation: G-SLI), a subtype of SLI proposed by van der Lely (e.g. van
der Lely, 1996). As its name implies, this subtype refers to deficits centred in the
grammatical comprehension and production of language. Based on a coherent pattern of
grammatical errors in this subtype, van der Lely (1996) has argued that this brand of SLI
should be attributed to a selective deficit of grammar. Finally, Bishop (2000) suggested an
overlap between Semantic-Pragmatic Disorder, SLI and the autistic spectrum.
1.1.2. Theoretical Accounts of SLI
Although the cognitive profile of SLI has been extensively studied, there is still no
widely accepted account for the aetiology of the impairment.
Certain researchers consider SLI to be due to a language-specific deficit, in particular
a deficit of the brain systems involved in the processing of grammar. Rice, Wexler, and
Cleave (1995), propose the Extended Optional Infinitive (abbreviation: EOI) account for SLI.
According to this account, the Optional Infinitive stage, i.e. a stage in language development
characterised by inconsistent application of tense-marking in obligatory contexts, is
protracted in SLI. Other language-specific hypotheses address a wider range of grammatical
deficits in SLI. The ‘feature blindness hypothesis’ (Gopnic & Crago, 1991) suggests that the
grammar of Children with SLI lacks morphophonological rules. van der Lely’s (1996)
Representational Deficit for Dependent Relationships (abbreviation: RDDR) hypothesis
4
considers that an “underlying deficit in the computational syntactic system” accounts for
the grammatical difficulties presented in G-SLI). Although supported by the languagespecific profile of SLI, grammar-specific hypotheses are often criticised for not being able to
accommodate linguistic deficits beyond grammar or address cross-linguistic data (cf.
Leonard, 1998, p.235).
For other authors, language limitations in SLI are due to a general (non-specific-tolanguage) deficit. These authors consider that deficits like slow processing (e.g. Bishop,
1994) or capacity limitations in processing (Kail, 1994; Leonard et al., 1992) affect language
development in SLI. Such accounts have the potential to address a wide range of linguistic
(and non-linguistic, e.g. deficits in mental imagery tasks (Johnston & Ramstad, 1983))
deficits in SLI. Whether these accounts could address variability in SLI in a coherent manner
needs to be further investigated (cf. Leonard, 1998, p.268).
Another group of studies also adopt an explanation of SLI which is not based on a
linguistic-processing deficit. However, this explanation does not consider a general
processing deficit but a specific deficit, localised in the phonological system, which spreads
along development to other aspects of language processing, such as syntax. Tallal and Piercy
(1973a, b) suggest that a perceptual/temporal processing deficit (evidenced from difficulties
in tone discrimination tasks under rapid conditions) underlies SLI. Gathercole and Baddeley
(1990) propose limitations in the phonological working memory in SLI, based on poor
performance of children with SLI in non-word repetition. Such accounts were also supported
by the connectionist modelling work of Joanisse (2004) and Joanisse and Seidenberg (2003).
Finally, Ulman and Pierpont (2005) propose the Procedural Deficit Hypothesis (PDH).
This hypothesis considers that language relies differentially on two different memory
systems in the brain: the procedural and the declarative one. A general deficit of the
procedural system (which is mainly involved in the learning and the execution of grammar)
explains the wide range of linguistic and non-linguistic deficits in SLI. The uneven profile of
SLI is also a result of the attempt of the declarative system to compensate for the
impairment of the procedural system. A connectionist implementation of this compensation
mechanism was considered in Thomas (2005).
1.1.3. Greek SLI
Greek is a language with a rich inflectional morphology. For example, nouns have
three genders and are inflected with respect to number and case, depending on the singular
form of the noun in the nominative case (Leonard, 1998). Verb inflections include features
for person, number, tense, and aspect (Clahsen & Dalalakis, 1999).
Behavioural studies on Greek SLI are relatively sparse. Various morphological/
grammatical deficits are reported, such as problems in marking subject-verb agreement,
(although past-tense is correctly marked) (Clahsen & Dalalakis, 1999), and omissions of
5
definite articles, clitic object pronouns, interrogative pronouns, and the case marker of
masculine noun phrases (Tsimpli & Stavrakaki, 1999). Stavrakaki (2001, 2002, 2006) has
identified problems of Greek children with SLI in the comprehension of reversible relative
clauses and the production of wh-questions. The same author reports limited verb
vocabulary and problems in verb-retrieval (Stavrakaki, 2000).
1.2.Connectionist modelling
Connectionist models are architectures of interconnected units, able to perform
parallel distributed computation (Rumelhart, McClelland, & PDP Research Group, 1986).
Based on a simplified model of the real neuron (cf. O’Reilly & Munakata, 2000, pp. 23-27),
each unit accumulates (integrates) weighted input activation signals from other units. An
activation function quantifies the unit’s output level of activation (firing), which is in turn
input to proceeding units. The (synaptic) weights of the connections are modifiable: a
learning algorithm determines changes in their values, so as to achieve certain criteria (for
example, the minimisation of the output error, in supervised learning with Back-Propagation
(for a detailed description of the algorithm, see O’Reilly & Munakata, 2000, pp. 158-162).
Therefore, when connectionist models are exposed to an environment (training set),
representations of aspects of this environment are developed in the weight patterns.
Offering the possibility to trace these representations during learning, connectionist models
are a powerful tool for addressing phenomena of cognitive development.
For example, in the domain of language development, the influential model of
Rumelhart and McClelland (1986) showed that a single-route connectionist architecture can
learn both regular and irregular past-tense inflections and demonstrate developmental
patterns observed in children (e.g. the U-shaped curve for the learning of irregulars). The
model argued against dual-route symbolic accounts (e.g. Pinker, 1991), which consider
separate pathways for the learning of rules and exceptions. Subsequent connectionist
models addressed the acquisition of morphology in further detail (e.g. incremental training
and type/token frequency effects (Plunkett & Marchmann, 1990, 1993), acquisition of noun
and verb morphology (Plunkett & Juola, 1999)). Other connectionist models studied
acquired language disorders, considering trained networks to which changes corresponding
to lesions (e.g. pruning of connections, noise) were introduced. For example, the model of
Joanisse and Seidenberg (1999) simulated the “double dissociation” between past-tense
formation for irregulars and nonce words reported for Parkinson’s Disease/Anterior Lesion
and Alzheimer’s Disease/Posterior Aphasia patients (correspondingly). Finally, some
connectionist models addressed developmental language disorders (e.g. Thomas &
Karmiloff-Smith 2002, 2003). These models examined how a typical learning trajectory was
changing under certain constraints, introduced in the parameter settings. Such
6
connectionist models for studying SLI are discussed separately and in more detail in
subsection 1.2.1. .
It is true that connectionist models could not claim a significant degree of biological
plausibility. A great deal of biological detail is omitted, as the architecture of the brain is too
complex to mimic, given current computational power. Scalability issues also apply.
Moreover, it is implausible that learning in the brain happens through the back-propagation
algorithm (O’Reilly & Munakata, 2000, p.162). Therefore, connectionist models could be
better described as biologically-flavoured models of cognition, which demonstrate statistical
learning, i.e. learning based on the regularities embedded in the training set.
The statistical nature of learning in connectionist networks often raises criticism,
mainly coming from symbolic-processing point of views. Connectionist models are often
considered statistical learners which afford ”a huge number of additional ‘degrees of
freedom’ ” (e.g. Green, 1998). They are thus able to fit to any data (training set), after
proper parameterisation. Marcus (1998) presents examples of connectionist networks
whose ability to learn certain tasks (involving variable-instances relations or recursive
structures) is strongly contingent on the types of representations or the training regimes
used. For him, and for other authors (e.g. Fodor & Pylyshyn, 1988), connectionist models are
successful in learning cognitive tasks when they are essentially implementations of symbolic
systems. They therefore challenge the appropriateness of performing inferences about
cognitive development based on the learning patterns of connectionist.
Being aware of the limitations of the connectionist methodology, this project still
posits that a systematic investigation of SLI with connectionist modelling would be
beneficial for a better understanding of the impairment.
1.2.1. Connectionist models of Specific Language Impairment
Connectionist modelling studies on SLI are relatively sparse. The acquisition of verb
morphology1 (and especially the acquisition of the past-tense) in SLI has been addressed in
Hoeffner and McClelland (1993), Joanisse (2003), and Thomas (2005)/Thomas and
Karmiloff-Smith (2003). Joanisse and Seidenberg (2003) also proposed a model for the
processing of sentences with anaphoric pronouns in SLI. Finally, Thomas and Redington
(2004) proposed a model for syntax comprehension in SLI. In this section, the existing
connectionist approaches to SLI are presented.
Hoeffner and McClelland (1993) addressed SLI deficits in the production of
uninflected and inflected (past tense, past participle, 3rd singular and progressive) verb
forms. Their model, a network with bidirectional connections (attractor network), learned
mappings between representations of the semantics of words and representations of their
1
For a non connectionist explanation of aspects of morphological acquisition, such as optional infinitive errors
in typical and atypical development, see MOSAIC (e.g. Freudenthal, Pine, & Gobet , 2006).
7
phonological forms. The model considered lower phonetic saliency for word-final stops and
fricatives; therefore less strong phonological representations were used for these
phonemes. Furthermore, SLI was simulated as a non-language specific perceptual deficit,
based on experimental findings (e.g. Tallal and Piercy, 1973 a,b). Thus, the impaired model
employed much weaker phonological representations for all phonemes (and even more
weak representations for word-final stops and fricatives). This simple manipulation captured
major phenomena of the acquisition of verb morphology in SLI, described in behavioural
studies: slower and poorer learning compared to typical development, increased
percentages of zero-marking errors, and different degrees of impairment for different
inflections. The model also predicted a greater degree of impairment for regular past-tense
forms than for irregular ones; this prediction was challenged by the modelling work of
Joanisse (2003) for past tense formation in SLI presented next.
A model by Joanisse (2003) also considered the perceptual (phonological) deficit
hypothesis for SLI in an attractor network trained on mappings between representations of
the semantics and phonological forms. However, here SLI was simulated by the addition of
small amounts of random noise to the phonological representations. In line with data from
behavioural studies of SLI, the impaired model was weaker in learning both regular and
irregular past-tense forms, while performance in generalising the rule to novel words was
very low. Therefore, the model of Joanisse (2003) challenged dual-route accounts of SLI,
which consider the disorder a rule-leaning deficit. Instead, the model demonstrated the
importance of phonology in the acquisition of the past-tense, especially in the
generalisation to novel forms.
Another model, in Thomas (2005), and Thomas and Karmiloff-Smith (2003),
addressed past-tense acquisition in the context of the Procedural Deficit Hypothesis (Ullman
& Pierpont,2005). The Procedural Deficit Hypothesis, which considers SLI as a deficit in the
procedural systems (rules), proposes a compensation mechanism to account for the
existence of overgeneralisation errors for irregulars or the ability to generalise the past
tense rule to novel strings in SLI. The authors examined whether a similar compensatory
pattern could be demonstrated in a connectionist network. A feed-forward three-layered
architecture was trained on mappings between phonological and lexical-semantics
representations of the input and phonological representations of the output. Interestingly,
when a low-discriminality activation function was used, the model qualitatively fitted the SLI
profile in past-tense production (low overall inflection rates, reduced overgeneralisation
errors for irregulars, poor generalisation to novel strings and increased frequency effects on
regular verbs). Therefore, it was a domain-general deficit that simulated the compensation
mechanism posited by the domain-specific Procedural Deficit Hypothesis. However, this
model did not capture the prevalence of zero-marked errors in SLI, while it suggested a
delayed rather than an atypical developmental profile for SLI.
In the domain of syntax comprehension, a model by Joanisse and Seidenberg (2003)
examined anaphoric resolution, an aspect of syntax processing in SLI. A three layered feedforward network was trained on mappings between sequences of phonological words to
their meanings (semantics). SLI was simulated as a perceptual deficit, with the addition of
8
random noise to the phonological representations. The domain-general perceptual deficit
replicated the gradient pattern of performance for different types of sentences reported in
behavioural studies. For example, resolution of anaphoric sentences with reflexive pronouns
(e.g. himself) was impaired to a greater extent than resolution of sentences with bound
pronouns (e.g. him), while the degree of impairment was less, when gender information was
provided by the context (e.g. Peter Pan says Wendy is tickling herself). The model suggested
that the phonological-deficit account can explain grammatical/syntactic deficits in SLI.
However, other authors note that not all children with SLI present a phonological deficit (cf.
Ullman & Pierpont, 2005, p. 400).
Finally, a model by Thomas and Redington (2004) addressed the comprehension of
complex syntactic structures (actives, subject clefts, passives and object clefts). The model
was a Simple Recurrent Network (abbreviation: SRN) (Elman, 1990) which was trained on
identifying an agent-patient or a patient-agent structure in a set of sentences presented
sequentially (word-by-word) in the input. Except from an acquired deficit (random
connection pruning), which was implemented to address behavioural data from Dick et al.
(2001), a developmental deficit (fewer hidden units) was also considered. The
developmental deficit, which corresponded to a ‘processing limitations’ account for SLI,
produced a pattern of performance qualitatively different from the performance pattern of
the acquired deficit case. In particular, the model generated the prediction that passives
structures would be relatively less vulnerable in a developmental deficit. Interestingly, this
prediction was verified by a subsequent study of Dick, Wulfeck, Krupa-Kwiatkowski, and
Bates (2004).
In conclusion, although successful in capturing many aspects of atypical language
development in the domains they addressed, connectionist studies on SLI were limited in
the most prevalent deficits in SLI, i.e. the deficits in morphology and syntax. Findings from
the existing modelling approaches could not be combined to address the heterogeneous
profile of SLI. This project therefore considers that SLI is “undermodelled” and proposes
further investigation of SLI with connectionist modelling.
2. Proposed research methodologies
As outlined above, this project proposes a systematic investigation of SLI with the
connectionist methodology, aiming to consider a wide range of phenomena on language
development, to adopt a cross-linguistic perspective, and to seek a unified explanation that
could address the heterogeneous profile of the impairment. This section summarises
connectionist work already underway or completed and sketches out directions for future
research.
9
2.1. Multiple infections Generator- A model of Morphology (Completed)
The first model that this project proposes is a model for the acquisition of inflectional
morphology. As discussed in the introduction, deficits in this domain (e.g. omission of
inflections) are prevalent in SLI, and are often considered a behavioural hallmark for SLI
(Rice, 2000). Furthermore, theoretical accounts of SLI (eg. the EOI account, Rice et al., 1995),
as well as the consideration of G-SLI subtype (van der Lely, 1996), were based on such
deficits. A detailed description of the acquisition of inflectional morphemes is provided in
the literature for both typical development (e.g. Brown, 1973 ; de Villiers & de Villiers, 1985)
and SLI (e.g. van der Lely, 1996).
The proposed model, the Multiple inflections Generator (abbreviation: MIG)
combines features of previous models of morphology in order to implement a generalised
inflectional system. Previous models have shown that a single connectionist architecture
can accommodate different inflection types (different Part-of-Speech (POS) categories)
(Plunkett & Juola, 1993) and different inflections within types (e.g. noun plural, noun
genitive) (Hoeffner & McClelland, 1993). Based on these findings, MIG was developed as a
model for the acquisition of noun (base forms, plural, genitive), verb (base forms,
progressive, 3rd singular, past tense), and adjective (base forms, comparative, superlative)
inflections.
The architecture of MIG is depicted in Fig.1. It is a three-layered feed-forward
architecture, which learns mappings between multiple types of input information (multiplecues model: lexical-semantics, grammar, phonology, and target inflection) and output
phonological representations.
Fig.1. The architecture of Multiple Inflections Generator (MIG)
10
Results2
The model (implemented in MATLAB) was trained using a training set that consisted
of 2000 words of an artificial language, encapsulating basic features of English. The training
set was constructed based on measurements obtained from the Tagged Brown Corpus
(abbreviation: TBC) (Francis & Kucera, 1979) with computational linguistics methods
(Python). Although, the TBC is distant from child language, using measurements from this
corpus was to a degree appropriate, as children are also exposed to adult language.
Frequencies were measured for the following: POS categories, inflections types, allophones,
irregular mappings types, and phonologically overlapping items of different POS categories.
Approximations of the obtained measurements were used for constructing the training set
of MIG. A table that summarises the basic features of the training set is provided in the
Appendix. The training set also considered low and high frequency tokens.
MIG was trained3 until it reached ceiling performance (accuracy: 99.85%). The model
captured several phenomena described in the literature for acquisition of inflectional
morphology. For example, the order with which different inflections were acquired (Fig.2)
was consistent to that described in the studies of Brown (1973), and de Villers and de Villers
(1981). Accuracy in subtypes of regular inflections (e.g. past tense allophones, Fig.3), was
depending on their frequency within the inflection. Furthermore, differences in the
acquisition of low and high frequency items (Fig.4) were more pronounced in irregular
inflections. Finally, the model predicted that zero-marking errors occur mainly in verb
inflections (Fig.5).
An impaired version of MIG, which employed fewer hidden units, was also
considered as an implementation of the general-processing- limitations account for SLI. As
depicted in Fig.6 this version performed significantly worse than the normal model.
Different inflections were affected differentially (Fig. 7), in line with the model of Hoeffner
and McClelland (1993). Furthermore, the impaired model produced more zero-mark errors
(omissions of inflections) than the normal model. Zero-mark errors of the two models in the
past-tense inflection are plotted in Fig. 8. The pattern presented is qualitatively consistent
with the EOI account of SLI (Rice et al., 1995).
A generalisation set consisting of items which rhymed with items of the training set
(in two degradations: low and high similarity), and strings violating the phonotactic rules
was used to assess the ability of the model to generalise the inflection on novel items.
Generalisation results for the past-tense rule are illustrated in Fig. 9. Although accuracy
(Fig.9, upper left) in generalising the past-tense rule was low for non-phonotactic novel
words (blue line), the errors that the model made were mostly incorrect replications of an
unfamiliar stem (Fig.9, bottom left). This yields that although the model could not reproduce
2
Pilot results from simulations were presented in the ‘Alston Child Language Meeting’, 9-11 May 2008.
The exact parameters of training were: 250 epochs, BackPropagation with Cross-Entropy Error, learning rate
001, 75 hidden units, 95 units for input and output phonology, 2000 units for lexical semantics, 3units for
grammatical category, and 10 units for the targeted inflection.
3
11
the stem accurately, it learned to apply the rule (i.e. to apply a past-tense suffix) to most
(>80%) of the novel items (Fig.9, upper right). These high rates of generalisation that MIG
achieved are consistent with behavioural data, and are an interesting result, not
demonstrated by previous connectionist models of morphology. Furthermore, the low rates
of generalisation that connectionist models of morphological processing exhibited, was a big
criticism of Marcus to connectionism (e.g. Marcus, 1996).
Future directions
The pilot results from simulations with MIG are promising. Further work with MIG
will aim to seek parameters under which a quantitative match to the typical development
and SLI data (accuracy, error types) is achieved. Training under other atypical conditions
(impairments in phonology and semantics, low-discrimination activation function) will also
be considered. Finally, the model will be extended to explain acquisition of inflectional
morphology in a morphologically rich language (Greek).
12
Fig.2. Normal model: Order of emergence of different inflection types
Fig.3. Normal model: Accuracy in regular past tense allophones
Fig.4. Normal model: Frequency effects in irregular past tense past-tense
13
Fig.5. Normal model: No-mark errors in different POS categories
Fig.6. Overall accuracy of normal and impaired model
14
Fig.7. Differential effect of impairment in different inflection types
Fig.8: No-mark errors in past-tense
15
Fig.9. Normal model: accuracy/errors in the generalisation of past-tense
2.2. Models proposed as future work
In this section, a series of connectionist models is proposed as future work for this
project. These models are just sketched out, so as to demonstrate the empirical phenomena
that will be targeted and the architectures that will be used. A great deal of implementation
details is omitted.
2.2.1. Model of phonology
The second model proposed by this project is a model for phonological processing in
SLI. As already discussed, deficits in phonology (e.g. in tone discrimination (Tallal & Piercy,
16
1973a, b), phoneme discrimination and identification of phoneme constancy (Bird et al.,
1995) or non-word repetition (Gathercole et al., 1995)) are prevalent in SLI. According to the
phonological-deficit account, such deficits affect many other aspects of language.
Fig.10. Proposed architecture for investigating phonological processing
To address these phenomena, this project proposes a model that learns phonological
words, i.e. mappings between input and output phonological representations (Fig.10). The
architecture is similar to a model proposed by Harm and Seidenberg (1999), which studied
phoneme discrimination in dyslexics. The proposed model will investigate neighbourhood
density and frequency effects in the acquisition of phonology and will also consider training
under constraints of atypical development. Finally, the model will also be extended to
Greek.
2.3. Model of semantics
Another major area of deficits in SLI is lexical semantics. As outlined above, children
with SLI present word-finding difficulties (McGregor & Leonard, 1995; German, 1987),
naming errors (Casby, 1992 ; McGregor & Waxman, 1995), impoverished word-definitions
(Dockrell et al., 2003) and problems in the learning of new words (Oetting et al., 1995).
The connectionist architecture of Fig. 11 is proposed to target these empirical
phenomena. The model is based on the model for semantic cognition of Rogers and
McClelland (2004). The model will employ distributed feature-based semantics
representations for a set of entities. The architecture will be trained on mappings between
the input and the output representations Differences in the acquisition of prototype and
atypical entities and differences in the ease of acquisition of different semantic features will
be investigated.
17
Fig.11. Proposed architecture for investigating semantics.
Finally, training under conditions of reduced resources or other learning constraints
will be considered, to simulate SLI.
2.4. Model of Syntax Comprehension
Another model proposed in this project will address SLI deficits in syntax
comprehension. This model will focus in the comprehension of active, subject-cleft, passive,
and object-cleft sentences. In particular, the model will expand on the previous modelling
work of Thomas and Redington (2004), in which an SRN (Elman, 1990) simulated empirical
phenomena on syntax comprehension from Dick et al (2001). This model generated
predictions which were in turn verified by a subsequent study by Dick et al. (2004).
The proposed model will therefore use a similar recurrent architecture (Fig. 12) to
target the empirical data from Dick et al.(2004) on the comprehension of complex sentence
structures by typically developing children, children with SLI, and children with focal lesions.
Due to the recurrent loop of hidden units SRNs are capable of processing sequential
information and consequently sentence syntax. This model will therefore be trained to
assign an AP (agent-patient) or a PA (patient-agent) tag (correspondingly) to active/subjectcleft and passive/object-cleft sentences presented sequentially in the input. Prediction of
the next work - a common task for SRNs- will also be investigated.
Again, the aim will be to fit the behavioural data and to identify parameter settings
which simulate the SLI profile. The model will be also extended to address relevant Greek
data (Stavrakaki, 2002).
18
Fig.12. Proposed architecture for investigating syntax comprehension
2.5. Model of Syntax Production
Finally, a model for investigating syntactic production will be developed. The
phenomenon that will be targeted is the formation of Wh-questions. In SLI, Wh-question
formation is characterised by deficits (e.g. “What under the table?” “What John eat
something?”) (Ingram, 1972 ; van der Lely & Batel, 2003).
To address the domain of question formation, an architecture (Fig.13) which learns
to map the input semantic scheme (e.g. in an agent/patient/action format) and the input
type information (e.g. declarative/interrogative) of sentences, to a sequence of output
representations for the words (the targeted question) is proposed. This architecture is based
on a Jordan network (Jordan, 1986) and employs recurrent connections from the output to
the input, which enable a dynamic output sequence to be produced from a static input
representation.
Training under limited processing resources or other learning constraints will be
considered for simulating SLI. The model will pursue the thesis that error patterns in SLI
represent the interference between canonical/ high frequency syntactic forms (e.g.
declarative) and lower frequency forms (e.g. interrogative) under suboptimal conditions.
Extension to Greek data (Stavrakaki, 2006) will also be considered.
19
Fig.13. Proposed architecture for investigating syntax production
20
3. Timetable
Time period
Winter term
07/08
(Completed)
Spring term
07/08
(Completed)
Summer term
07/08
Task
-Preliminary review of the literature
Winter term
08/09
-Development of the model of phonology, generation / processing of
results
-Development of the model of semantics, generation /processing of
results
- Extension of the model of phonology to Greek, generation/processing
of results
-Development of the model of syntactical comprehension, generation /
processing of results
-Extension of the model of syntactical comprehension to Greek,
generation / processing of results
-Development of the model of syntactical production, generation/
processing of results
Spring term
08/09
Summer term
08/09
Winter term
09/10
Spring term
09/10
Summer term
09/10
-Development of the model for inflectional morphology (MIG)
-Production and processing of pilot results
-Completion of investigations with MIG (English)
-Extension of MIG to Greek
-Extension of the model of syntactical production to Greek, generation /
processing of results
-Writing up of thesis
-Writing up of thesis
Table.1. Timetable allocating research time
21
References
Aram, D., & Nation, J. (1975). Patterns of language behavior in children with developmental
language disorders. Journal of Speech and Hearing Research, 18, pp. 229-241.
Bird, J., Bishop, D.V.M., & Freeman, N.H. (1995). Phonological awareness and literacy
development in children with expressive phonological impairments. Journal of Speech and Hearing
Research, 38, pp. 446-462.
Bishop, D.V.M. (2000). Pragmatic language impairment: a correlate of SLI, a distinct subgroup, or
part of the autistic continuum? In D. V. M. Bishop & L. B. Leonard (Eds.), Speech and Language
Impairments in Children: Causes, Characteristics, Intervention and Outcome Hove, UK: Psychology
Press.
Bishop, D.V.M. (1997). Uncommon understanding: Development and Disorders of language
comprehension in children. Hove, England: Psychology Press.
Bishop, D.V.M. (1979). Comprehension in developmental language disorders, Developmental
Medicine and Child Neurology 21, pp. 225–238.
Brown, R. (1973). A first language: The early stages. London: George Allen & Unwin Ltd.
Casby, M.W. (1992). An intervention approach for naming problems in children. American
Journal of Speech-Language Pathology, 1, pp. 35-42.
Clahsen, H., & J. Dalalakis 1999. Tense and agreement in Greek SLI: A case study. Essex Research
Reports in Linguistics, 24, pp. 1-25.
Craig, H. (1993) Social skills of children with specific language impairment: Peer relations.
Language, Speech, and Hearing Services in Schools, 24, pp. 206-215.
de Villiers, J.G., & de Villiers, P.A. (1985). Acquisition of English. In D. Slobin (Ed.), The
crosslinguistic study of language acquisition: Vol. 1. The data (pp. 27-140). Hillsdale, NJ: Lawrence
Erlbaum.
Dick, F., Wulfeck, B., Krupa-Kwiatkowski, & Bates (2004). The development of complex sentence
interpretation in typically developing children compared with children with specific language
impairments or early unilateral focal lesions. Developmental Science, 7(3), pp. 360-377.
Dick, F., Bates, E., Wulfeck, B., Aydelott Utman, J., Dronkers, N., & Gernsbacher, M.A. (2001).
Language deficits, localization and grammar: evidence for a distributive model of language
breakdown in aphasics and normals. Psychological Review, 108(4), pp. 759-788.
Dockrell, J.E., Messer, D., George, R., & Ralli, M. (2003). Beyond naming patterns in children with
WFDS – definitions for nouns and verbs. Journal of Neurolinguistics, 16, pp. 191-211.
Elman, J.L. (1990). Finding structure in time. Cognitive Science, 14, pp. 179-211.
Fodor, J. A., & Pylyshyn, Z. (1988). Connectionism and cognitive architecture: A critical analysis.
Cognition, 28, pp. 3-71.
Francis, W.N., & Kucera, H. (1979). Manual of Information to Accompany a Standard Sample
22
of Present-day Edited American English , for Use with Digital Computers. Original ed. 1964, revised
1971, revised and augmented 1979. Department of Linguistics, Brown University, Providence, R.I.
Freudenthal, D., Pine, J.M. & Gobet, F. (2006). Modelling the development of children's use of
Optional Infinitives in English and Dutch using MOSAIC. Cognitive Science, 30, pp. 277-310.
Gallagher, T.M., & Craig, H.K. (1984). Pragmatic assessment: Analysis of a highly frequent
repeated utterance. Journal of Speech and Hearing Disorders , 49 , pp. 368-377.
Gathercole, S. E., & Baddeley, A. D. (1990). Phonological memory deficits in language disordered
children: Is there a causal connection? Journal of Memory and Language 29, pp. 336–360.
German, D. J. (1987). Spontaneous language profiles of children with word-finding problems.
Language, Speech, Hearing Services in Schools, 18, pp. 217-230.
Gopnik, M., & Crago, M. (1991). Familial aggregation of the developmental language disorders.
Cognition, 39, pp. 1–50.
Green, C.D. (1998) Are connectionist models theories of cognition? Psycoloquy 9(4).
Harm, M.W. & Seidenberg, M.S. (1999). Phonology, Reading Acquisition, and Dyslexia: Insights
from Connectionist Models. Psychological Review, 106(3), pp. 491-528.
Hoeffner, J. H., & McClelland, J. L. (1993). Can a perceptual processing deficit explain the
impairment of inflectional morphology in developmental dysphasia? A computational investigation.
In E. V. Clark (Ed.), Proceedings of the 25th Child Language Research Forum. Palo Alto, CA: Stanford
University Press.
Joanisse, M.F. (2004) Specific language impairments in children: Phonology, semantics and the
English past tense. Current Directions in Psychological Science, 13, pp. 156-160.
Joanisse, M.F., & Seidenberg, M.S. (1999). Impairments in verb morphology following brain
injury: A connectionist model. Proceedings of the National Academy of Science USA, 96, pp. 75927597.
Joanisse, M.F., & Seidenberg, M.S. (2003). Phonology and syntax in specific language
impairment: Evidence from a connectionist model. Brain and Language, 86, pp. 40-56.
Johnston, J., & Kamhi, A. (1984). Syntactic and semantic aspects of the utterances of languageimpaired children: The same can be less. Merrill-Palmer Quarterly, 30, pp. 65–85.
Johnston, J., & Ramstad, V. (1983). Cognitive development in preadolescent language impaired
children. British Journal of Disorders of Communication, 18, pp. 49-55.
Jordan, M.I. (1986). Attractor dynamics and parallelism in a connectionist sequential machine.
Proceedings of the Eighth Annual Conference of the Cognitive Science Society, Englewood Cliffs, NJ:
Erlbaum, pp. 531-546.
Ingram, D. (1972). The acquisition of questions and its relation to cognitive development in
normal and linguistically deviant children: A pilot study. U Papers and reports in child language
development (br. 4, 13). Committee on linguistics Stanford University.
Kail, R. (1994). A method for studying the generalized slowing hypothesis in children with
specific language impairment. Journal of Speech and Hearing Research, 37, pp. 418-421.
23
Leonard, L.B. (1998). Children with specific language impairment. Cambridge, MA: MIT Press.
Leonard, L.B., Bortolini, U., Caselli, M.C., McGregor, K.K., & Sabbadini, L. (1992). Morphological
deficits in children with specific language impairment: The status of features in the underlying
grammar. Language Acquisition, 2 , pp. 151-179.
Marcus, G.F. (1996) Why do children say “breaked”? Current Directions in Psychological
Science, 5, pp. 81-85.
Marcus, G. F. (2001). The Algebraic Mind: Integrating Connectionism and Cognitive Science.
Cambridge, MA: MIT Press.
McGregor, K.K., & Leonard, L.B. (1995). Intervention for word-finding deficits in children. In M.
Fey, J. Windsor & S. Warren, (Eds.), Language intervention: Preschool through the elementary years ,
(pp. 85-105). Baltimore, MD: Paul H. Brookes.
McGregor, K.K., & Waxman, S.R. (1995). Multiple level naming abilities of children with wordfinding deficits. Paper presented at the Boston University Conference on Language Development,
Boston, MA.
O'Reilly, R.C., & Munakata, Y. (2000). Computational Explorations in Cognitive Neuroscience:
Understanding the Mind by Simulating the Brain. Cambridge: MIT Press.
Oetting, J., Rice, M., & Swank, L. (1995). Quick incidental learning (QUIL) of words by schoolage
children with and without SLI. Journal of Speech and Hearing Research, 38 , pp. 434–445.
Pinker, S. (1991). Rules of language. Science, 253, pp. 530-535.
Plunkett, K., & Juola, P. (1999). A connectionist model of English past tense and plural
morphology. Cognitive Science, 23, pp. 463-490.
Plunkett, K., & Marchman, V. (1991). U-shaped learning and frequency effects in a multi-layered
perceptron: Implications for child language acquisition. Cognition, 38, pp. 1-60.
Plunkett, K., & Marchman, V. (1993). From rote learning to system building: Acquiring verb
morphology in children and connectionist nets. Cognition, 48, pp. 21-69.
Rapin, I. & Allen, D. (1983). Developmental language disorders: Nosologic considerations. In
U.Kirk (Ed.), Neuropsychology of language, reading, and spelling (pp. 155–184). New York: Academic
Press.
Rice, M.L. (2000). Grammatical symptoms of specific language impairment. In: D.V.M. Bishop &
L.B. Leonard (Eds.), Speech and Language Impairments in Children: Causes, characteristics,
intervention and outcome (pp.17-34). Hove, England: Psychology Press.
Rice, M, Wexler, K. & Cleave, P. (1995). Specific Language Impairment as a period of extended
optional infinitive. Journal of Speech and Hearing Research, 38, pp. 850-863.
Rogers, T. T. and McClelland, J. L. (2004). Semantic Cognition: A Parallel Distributed Processing
Approach. Cambridge, MA: MIT Press.
24
Rumelhart, D. E. & McClelland, J. L. (1986). On learning the past tense of English verbs. In J. L.
McClelland, D. E. Rumelhart and PDP Research Group (Eds), Parallel distributed processing: Volume
2: Psychological and Biological models. (pp. 216-271). Cambridge, MA: MIT Press.
Rumelhart, D. E., McClelland, J. L., & PDP Research Group (Eds.) (1986). Parallel distributed
processing: Volume 1: Foundations. Cambridge, MA: MIT Press.
Stavrakaki, S. (2006). Developmental perspectives on Specific Language Impairment: Evidence
from the production of wh-questions by Greek SLI children over time. International Journal of
Speech-Language Pathology, 8( 4) , pp. 384–396.
Stavrakaki, S. (2002). Sentence comprehension in Greek SLI children. In L. Kelly, F. Windsor & N.
Hewlett (Eds.), Advances in Clinical Linguistics and Phonetics. Hillsdale, NJ:Erlbaum. Chapter 5, pp.
57-72.
Stavrakaki, S. (2001). Comprehension of reversible relative clauses in Specifically Language
Impaired and Normally Developing Greek children. Brain and Language, 77, pp. 419-431.
Stavrakaki, S. (2000). Verb lexicons in SLI: some experimental data from Modern Greek. Journal
of Greek Linguistics, 1, pp. 93-129.
Tallal, P., & Piercy, M. (1973a). Defects of non-verbal auditory perception in children with
developmental aphasia. Nature, 241, pp. 468-469.
Tallal, P., & Piercy, M. (1973b). Developmental aphasia: Impaired rate of non-verbal processing
as a function of sensory modality. Neuropsychologia, 11, pp. 389-398.
Tomblin, J.B., Records, N.L., Buckwalter, P., Zhang, X., Smith, E., & O’Brien, M. (1997).
Prevalence of specific language impairment in kindergarten children. Journal of Speech, Language
and Hearing Research, 40, 1245-1260.
Thomas, M. S. C. (2005). Characterising compensation. Cortex, 41(3), pp. 434-442.
Thomas, M.S.C. & Karmiloff-Smith, A. (2003). Modelling language acquisition in atypical
phenotypes. Psychological Review, 110 (4), pp. 647-682.
Thomas, M.S.C. & Karmiloff-Smith, A. (2002). Modelling typical and atypical cognitive
development. In U. Goswami (Ed.), Handbook of Childhood Development (pp. 575-599). Blackwells
Publishers.
Thomas, M.S.C. & Redington, M. (2004). Modelling atypical syntax processing. In W. Sakas (Ed.),
Proceedings of the First Workshop on Psycho-computational models of human language acquisition
at the 20th International Conference on Computational Linguistics, pp. 85-92.
Tsimpli, I.M. & Stavrakaki, S. (1999). The effects of a morphosyntactic deficit in the Determiner
System: The case of a Greek SLI child. Lingua, 108, pp. 31-85.
Ullman, M.T., & Pierpont, E. (2005). Specific Language Impairment is not specific to language:
The Procedural Deficit hypothesis. Cortex, 41(3), pp. 399-433.
van der Lely, H. K. J. (1996). Empirical evidence for the modularity of language from
Grammatical SLI children. Proceedings of The 20 annual Boston University conference on language
development, Volume 2. Somerville, MA: Cascadilla Press, pp. 781-791.
25
van der Lely, H. K. J., & Battell, J. (2003). Wh-movement in children with grammatical SLI: a test
of the RDDR hypothesis. Language, 79, pp. 153–181.
van der Lely, H. K. J., & Harris, M. (1990). Comprehension of reversible sentences in specifically
language impaired children. Journal of Speech and Hearing Disorders, 55, pp. 101-117.
van der Lely, H. K. J. & Stollwerck, L. (1997). Binding theory and specifically language impaired
children, Cognition, 62, pp. 245–290.
Wilson, B.C., & Risucci, D.A. (1986). A model for the clinical quantitative classification
Generation 1—Application to language disordered preschool children. Brain and Language, 27 , pp.
281–309.
26
Appendix: The training set of MIG: features that approximate the
measurements on the tagged Brown Corpus.
level0
level1
level2
level3
level4
level5
SINGULAR
(600)
PLURAL (150)
NOUNS
(800)
REGURAL
(770)
130
30
/z/
(500)
500
/ez/
(150)
150
BASE FORMS
(130)
130
/s/
(130)
130
/z/
(200)
200
(70)
70
/ez/
80
PAST TENSE
(120)
IRREGULAR
(70)
BASE FORMS
(320)
SUPERLATIVE
(40)
500
150
REGULAR
(330)
COMPARATIVE
(40)
ADJECTIVES
(400)
140
(150)
PROGRESSIVE
(80)
VERBS
(400)
/s/
(140)
/z/
(500)
/ez/
(130)
/s/
3rd
SINGULAR
(70)
TOKEN
TYPES
(1600)
600
IRREGULAR
(30)
GENITIVE (50)
counts
/t/
(65)
65
/d/
(180)
180
/ed /
(85)
85
IDENTITY
(10)
VOWEL_CHANGE
(50)
ARBITRARY
(10)
10
50
10
320
REGULAR
(380)
IRREGULAR
(20)
REGULAR
(380)
IRREGULAR
(20)
27
380
20
380
20