Readability Assessment for Text
Simplification
Sandra Aluisio1, Lucia Specia2,
Caroline Gasperin1, Carolina Scarton1
1University
of São Paulo, Brazil
2University of Wolverhampton, UK
The 5th Workshop on Innovative Use of
NLP for Building Educational Applications
Motivation
• Develop technology to benefit low literacy readers
INAF levels
Illiterate
Rudimentary
Basic
Advanced
47
34
27
26
36
37
38
38
26
25
26
25
26
28
25
12
13
12
11
9
7
2007
2009
2001-2002 2002-2003 2003-2004 2004-2005
25
21
68
%
˗ Rudimentary: studied up to 4 years; can find explicit information in short and familiar texts
˗ Basic: studied between 4 and 8 years; can read and understand texts of average length, and
find information even when it is necessary to make some inference
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
2
Readability Assessment
• To assess the readability level of a text
– Three levels of readability: INAF levels
Rudimentary – Basic – Advanced
• To supplement our text simplification technology
– Two levels of simplification: degree of application of
simplification operations
• STRONG: operations are applied to all complex syntactic
phenomena present  RUDIMENTARY
• NATURAL: operations are applied selectively, only when the
resulting text remains “natural”  BASIC
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
3
Text Simplification Scenario
• Authoring tool for creating simplified texts
SIMPLIFICA
1. Author inputs text
2. Author receives suggestions of possible
simplifications: may accept or not
•
•
Lexical substitutions
Syntactic simplification
3. Author does not know if the text is simple
enough for his audience
• Feedback: Readability assessment
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
4
Readability Assessment System
• Machine learning
– Classes = 3 INAF levels
– Trained on corpus of manually simplified texts
• Original text + natural and strong simplifications
– Extensive set of features
• Cognitively-motivated: Coh-Metrix [Graesser et al., 2004]
• Syntactic: occurrence of complex phenomena
• Language model: up to trigrams
– 3 paradigms: Classification, Ordinal Classification,
Regression [Heilman et al., 2007]
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
5
Corpora
• Training and testing corp
– General news: Zero Hora (ZH) newspaper
– Popular science news: Caderno Ciencia (CC)
– 3 versions for each text: original, natural, strong
Corpus
Documents Sentences
ZH original
ZH natural
ZH strong
CC original
CC natural
CC strong
CH
104
104
104
50
50
50
130
2184
3234
3668
882
975
1454
3624
Words
Avg. words p. text
(std. deviation)
Avg. words p.
sentence
46190
47296
47938
20263
19603
20518
95866
444.1 (133.7)
454.7 (134.2)
460.9 (137.5)
405.2 (175.6)
392.0 (176.0)
410.3 (169.6)
737.4 (226.1)
21.1
14.6
13.0
22.9
20.1
14.1
26.4
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
6
Features
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Number of words
Number of sentences
Number of paragraphs
Number of verbs
Number of nouns
Number of adjectives
Number of adverbs
Number of pronouns
Average number of words per sentence
Average number of sentences per
paragraph
Average number of syllables per word
Flesch index for Portuguese
Incidence of content words
Incidence of functional words
Raw Frequency of content words
Minimal frequency of content words
Average number of verb hypernyms
Incidence of NPs
Number of NP modifiers
Number of words before the main verb
21
22
23
24
25
26
27
28
29
30
Number of high level constituents
Number of personal pronouns
Type-token ratio
Pronoun-NP ratio
Number of “e” (and)
Number of “ou” (or)
Number of “se” (if)
Number of negations
Number of logic operators
Number of connectives
41
42
43
44
45
46
47
48
49
50
Adverb ambiguity ratio
Adjective ambiguity ratio
Incidence of clauses
Incidence of adverbial phrases
Incidence of apposition
Incidence of passive voice
Incidence of relative clauses
Incidence of coordination
Incidence of subordination
Out-of-vocabulary words
31
32
33
34
35
36
37
38
39
40
Number of positive additive connectives
Number of negative additive connectives
Number of positive temporal connectives
Number of negative temporal connectives
Number of positive causal connectives
Number of negative causal connectives
Number of positive logic connectives
Number of negative logic connectives
Verb ambiguity ratio
Noun ambiguity ratio
51
52
53
54
55
56
57
58
59
LM probability of unigrams
LM perplexity of unigrams
LM perplexity of unigrams, no line break
LM probability of bigrams
LM perplexity of bigrams
LM perplexity of bigrams, no line break
LM probability of trigrams
LM perplexity of trigrams
LM perplexity of trigrams, no line break
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
7
Feature Analysis
• Pearson correlation between features and
literacy levels
1
2
3
4
5
6
7
8
9
10
Feature
Words per sentence
Incidence of apposition
Incidence of clauses
Flesch index
Words before main verb
Sentences per paragraph
Incidence of relative clauses
Syllables per word
Number of positive additive connectives
Number of negative causal connectives
Correlation
0.693
0.688
0.614
0.580
0.516
0.509
0.417
0.414
0.397
0.388
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
8
Predicting readability Levels
Features
All
Language
Model
Basic
Syntactic
CohMetrixPORT
Flesch
Class
original
natural
strong
original
natural
strong
original
natural
strong
original
natural
strong
original
natural
strong
original
natural
strong
F
Corr. MAE
0.913 0.84 0.276
0.483
0.732
0.669 0.25 0.381
0.025
0.221
0.846 0.76 0.302
0.149
0.707
0.891 0.82 0.285
0.32
0.74
0.873 0.79 0.290
0.381
0.712
0.751 0.52 0.348
0.152
0.546
Classification
Weka SVM
Features
All
Language
Model
Basic
Syntactic
CohMetrixPORT
Flesch
Class
original
natural
strong
original
natural
strong
original
natural
strong
original
natural
strong
original
natural
strong
original
natural
strong
F
Corr. MAE
0.904 0.83 0.163
0.484
0.731
0.634 0.49 0.344
0.497
0.05
0.83 0.73 0.231
0.334
0.637
0.891 0.81 0.180
0.382
0.714
0.878 0.8 0.183
0.432
0.709
0.746 0.56 0.310
0.489
0
Ordinal Classification
Weka Pairwise SVM
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
9
Predicting readability Levels
Features
All
Language Model
Basic
Syntactic
Coh-Metrix-PORT
Flesch
Corr.
0.8502
0.6245
0.7266
0.8063
0.8051
0.5772
MAE
0.3478
0.5448
0.4538
0.3878
0.3895
0.5492
Regression
Weka SVM-reg, RBF Kernel
• Best correlation: Regression
• Lowest MAE: Ordinal Classification
• Combination of all features consistently yields better
results: more robust
• Syntactic features achieve the best correlation scores
• Language model features performed the poorest
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
10
Conclusions
• It is possible to predict with satisfactory
performance the readability level of texts
according to our three classes of interest
• Ordinal Classification seems to be the most
appropriate model to use
– High correlation, lowest error rate (MAE)
• Combination of all features is best
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
11
SIMPLIFICA Tool
• Integration of classification model
– Simplest model, highest F-measure, comparable
correlation scores
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
12
Future Work
• Add deeper cognitive features, e.g. semantic,
coreference, latent semantics metrics
• User evaluation: authors
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
13
Thanks!
•
project
http://caravelas.icmc.usp.br/
The 5th Workshop on Innovative Use of NLP for Building Educational Applications
14