The Czech National Corpus
Looking Back, Looking Forward
Věra Schmiedtová
Institute of the Czech National Corpus, Charles University Prague
vera.schmiedtova@ff.cuni.cz
While the Institute of the Czech National Corpus (ICNC) was founded in 1994 as part of the
Faculty of Arts, Charles University in Prague, the Czech National Corpus project took off only
in 1996, when the Institute could begin its activity two years later thanks to its first grant.
Although the ICNC was an integral part of Charles University, the Faculty of Arts had no
financial means to support its operation. From the very beginning the Institute has been able to
exist only due to special-purpose grants provided by the Czech Science Foundation (Grantová
agentura České Republiky) or due to special-purpose financial contributions made by the
Ministry of Education of the Czech Republic. In its initial stages the Institute’s existence was
also dependent on sponsorship from Czech banks, especially the Commercial Bank. Why should
we mention this? It is important to remember that the existence of the ICNC has never been
simple, nor is its future going to be entirely free of uncertainty.
At the outset the Institute’s primary aim was to compile a material database for a new
monolingual dictionary of the Czech language. The material is ready (text databases, frequency
lists, etc.), the compilation of a new monolingual dictionary of contemporary Czech has not
started yet.
In any case the Institute has set itself a number of other targets. It compiles various types of
corpora which make it possible to trace the development of Czech since the 13th century. the
main focus, however, is on the synchronic stage of both the written and the spoken language. It
organizes an intensive collection of data in a number of locations all over the country, i.e.
recoridng and transcription of spontaneous speech.
To date, the Institute has built the following types of corpus which are accessible to users via the
Internet:
1. Corpora of synchronic written language;
Two representative corpora have been made public, SYN2000 (100 million words) and
SYN2005 (100 million words). One other corpus, the non-representative SYN2006pub (300
million words), was been released into the public domain. There are plans to make
SYN2009pub (500 million words) accessible to everybody (Křen 2009) in the course of 2009.
This means that during this year the Czech language will have available electronic data of a
billion word forms of contemporary written language.
2. Diachronic corpus;
(It includes texts from the end of 13th century onwards, numbering more than 3 600 000 word
forms.)
3. Corpora of spontaneous spoken language;
There are four corpora of spontaneous spoken language available now - Pražský mluvený
korpus (PMK) [Prague Spoken Corpus], Brněnský mluvený korpus (BMK) [Brno Spoken
Corpus] and the corpora ORAL2006 (Kopřivová-Waclawičová 2006) and
ORAL2008.(Waclawičová-Křen 2008)
4. Parallel corpora;
In 2005 the project InterCorp was initiated; participating in its compilation are linguistic
departments of the Faculty of Arts, Charles University, Akademie věd ČR and some other
departments at the Masaryk University in Brno. The project includes 23 of mostly European
languages contrasted with Czech. At present these corpora are comprised of some 4 million
word forms in fiction texts (Čermák 2009).
5. Single-author corpora;
(The Institute produced single-author corpora of two outstanding 20th-century Czech writers –
Karel Čapek and Bohumil Hrabal. One of the corpora has already resulted in a frequency
analysis of Čapek’s language Slovník Karla Čapka [The Dictionary of Karel Čapek] (2007)
(for detailed reports see Čermák 2008; Křen 2008); a similar analysis of Bohumil Hrabal’s
language Slovník Bohumila Hrabala [The Dictionary of Bohumil Hrabal] will be published
during 2009.)
We may take a closer look at the corpora now:
Corpora of synchronic written language
SYN2000 Corpus
contains some 100 million word forms. It consists of whole texts that were included in it on the
basis of research into written language reception so as to cover as wide a spectrum of genres of
the Czech language as possible. It contains most texts originating between 1990 and 1999. The
corpus also includes important works of Czech literature appearing before 1990 such as the
novel Krakatit (1924) by Karel Čapek (1890-1938) or the novel Zbabělci [Cowards](1958) by
Josef Škvorecký (1924). Inclusion of such earlier texts is conditioned by the author‘s year of
birth preceding 1880.
The corpus is lemmatized and part-of-speech tagged. It can be searched using the corpus
manager Bonito which lets the user see a word in context, search for a string of words, search by
part-of-speech tags and canonical word forms, resort a concordance, see information on the
origin and type of text in which the word occurred, save selected concordance lines on the harddisk of the computer used, see statistical data and create subcorpora.
Composition of the SYN2000 Corpus:
60 % journalistic texts
25 % technical texts
15 % fiction
SYN2005 Corpus
is also a synchronic corpus representative of present-day written Czech, containing 100 million
word forms (tokens). It differs from the previous corpus in many respects. Their comparison is
attempted in articles by Křen and Hlaváčová (2006, 2008). None of the texts included in the
SYN2005 Corpus was previously used in the SYN2000 Corpus, the two corpora are disjunct,
and together they contain some 200 million words (tokens).
Composition of the SYN2005 Corpus:
40
27
33
% fiction
% technical texts
% journalistic texts
The representativeness ((balanced composition) of the SYN2005 Corpus is based on a
recent study of written language reception; its composition therefore differs from that of the
SYN2000 Corpus considerably in many ways. A comparison of the two corpora according to
the main categories of texts is shown in the following table:
SYN2005 SYN2000
fiction
40 %
15 %
technical texts
27 %
25 %
journalistic texts 33 %
60 %
Other differences can be found within the main genres: while the thematic subcategories
of technical writing have changed only little, by contrast the composition of journalistic writing
changed substantially. All journalistic texts date from 2000-2004, with each year having the
same proportion of journalistic texts. Compared to the SYN2000 Corpus, though, the
representation of individual titles is different. Technical writing dates from 1990-2004, fiction
texts are occasionally even older than that, but in both cases care was taken to include as little
earlier texts as possible.
Unlike the SYN2000 Corpus, this corpus uses an improved version of lemmatization and
part-of-speech tagging. The actual system of part-of-speech tags had remained the same, only
one more was added, that of verbal aspect.
Corpus SYN2006PUB
is a synchronic corpus of journalistic writing of 300 million word forms (tokens). It exclusively
contains journalistic texts from November 1989 to the end of 2004, i.e. from the same period as
covered by the SYN2000 Corpus and the SYN2005 Corpus. As regards the actual texts included
there is no overlap between the three corpora whatsoever. Altogether the three corpora,
SYN2000, SYN2005 and SYN2006PUB, contain some 500 million words (tokens).
Compared to the SYN2005 Corpus, the lemmatization and part-of-speech tagging had
again been upgraded, although the difference is not as marked as that between the SYN2000 and
SYN2005 corpora.
Corpora of spoken Czech
ORAL2006, ORAL2008
contain transcripts of conversations taking place exclusively in informal situations. ORAL2008
is the first spoken corpus of the ICNC which is fully balanced in terms of the basic
sociolinguistic variables of the speakers (gender, age, education, and the area of residence in
childhood, i.e. at a time when their idiolect was formed; these areas are defined according to the
traditional classification of regional dialects).
The ORAL2008 Corpus is based on the same type of data as ORAL2006, however none
of the transcripts included in ORAL2008 are part of the ORAL2006 Corpus.
The corpora are composed of recordings made in different parts all over the Czech Lands
(but not in Moravia, which is a region of the Czech Republic where a different group of Czech
dialects are spoken). All these recordings were made in informal settings; the speakers knew
each other well and had friendly relations. They were not informed in advance about the purpose
of the recording; they were told only after it had been made. All of them subsequently agreed to
the recordings being used for the purposes of the Czech National Corpus. The recording,
transcription and annotation were made in keeping with the general principles applied in the
compilation of the previous spoken corpora as part of the Czech National Corpus, and especially
the ORAL2006 Corpus. All corpora use identical system of annotating the three basic binary
sociolinguistic categories of speakers.
ICNC publishing activities
The list of the ICNC’s publishing achievements is quite extensive. The SYN2000 Corpus
was the source for Frekvenční slovník češtiny [A Frequency Dictionary of Czech] (2004) (for a
report see Čermák-Křen 2005), while the PMK was the source for Frekvenční slovník mluvené
češtiny [A Frequency Dictionary of Spoken Czech] (2007). The Institute has so far published
eight volumes in the series Studie z korpusové lingvistiky [Corpus Linguistics Studies],
presenting monographs based on the ICNC’s corpora. The first volume to appear in the Corpus
Lexicography Series was The Dictionary of Karel Čapek mentioned above. It is to be followed
by The Dictionary of Bohumil Hrabal later this year. The Institute also intends to publish here
Slovník totalitního jazyka [A Dictionary of Totalitarian Language] for the purposes of which a
corpus of texts from this period has already been compiled (see Schmiedtová 2006).
The ICNC’s future is assured until 2011 when the Ministry of Education’s research grant
which provides for the Institute’s operation expires. Before the end of this research grant period
the Institute plans to go on developing all lines of research. It aims to publish more monographs
in the Corpus Linguistics Studies series. (In the immediate future there are plans for monographs
on: Valency of abstract nouns; Contemporary declination of one specific type of noun; The
perfect in present-day Czech.)
This year the ICNC is going to publish a book entitled Statistika češtiny [The Statistics of the
Czech Language], and has organized a conference on parallel corpora.
Conclusion
Inevitably the Institute’s activities and work on such labour-intensive projects are very costly.
Therefore we hope that the Institute will continue to receive financial support that will enable it
to carry on with the work, particularly as the value and importance of these projects increases if
language development can be traced systematically and continuously.
A brief information on some of the books published the ICNC
Frekvenční slovník češtiny [A Frequency Dictionary of Czech]
František Čermák - Michal Křen (eds.)
It was published in November 2004 by the NLN Publishing House in Prague.
Based on the sufficiently large SYN2000 Corpus, carefully balanced so as to be
representative of contemporary written language, analyzed automatically with
subsequent extensive manual corrections, the dictionary is guaranteed to offer
highly reliable data. The main body of the dictionary includes: 50 000 most
frequent common nouns provided with their frequencies, frequency-based
ranking and also their typical distribution in the main genres (fiction, technical
and journalistic texts) expressed in percentages; 2 000 most frequent proper
names; 1 000 most frequent abbreviations. The appendices provide information
in the most frequent punctuation marks, letters in the Czech texts, and the
proportion of words forms in text of the lemmas listed by the dictionary. The dictionary comes with a CD
which allows browsing the word list, its sorting and searching according to various criteria and,
naturally, saving the selected entries for further analysis.
Frekvenční slovník mluvené češtiny [A Frequency Dictionary of Czech]
František Čermák (ed.)
The first dictionary ever to present authentic spoken Czech in contrast to
both standard and spoken Czech. It is derived from the Prague Spoken
Corpus based on sociolinguistically representative recordings. The
included CD contains the whole corpus complete with corpus extraction
software.
Jak využívat Český národní korpus [How to use the Czech National
Corpus]
František Čermák - Renata Blatná (eds.)
The manual for users interested in work with corpora provides information
on how to search in a corpus and makes use of the data collected from them.
It is intended as a study guide for those want to explore Czech in a different
way from that used in traditional textbooks. It includes exercises in
phonetics, morphology, lexicon and word combinations.
Slovník Karla Čapka [The Dictionary of Karel Čapek]
František Čermák (ed.)
This volume opens a new series of corpus-based monographs on language
called Korpusová lexikografie [Corpus Lexicography]. It will present a
succession of dictionaries relating to a specific crucial historical period or
single-author dictionaries presenting the language of important and well-known
personalities of national culture who significantly shaped the era in which they
lived and the language of that time.
A list of monographs derived from the ICNC corpora:
Corpus linguistics:
The state of the art and
model approaches
Collocations
Multiword prepositions in
contemporary Czech
The valency of Czech
adjectives
Aspectual morphology of
the Czech verb
Morphology of spoken Czech:
Frequency analysis
Czech in the spoken corpus Regulation of language and
the Concept of minimal
intervention
The valency of the Czech
substantives
References:
Čermák, F.-Křen, M. (2005): New Generation Corpus-Based Frequency Dictionaries: The Case
of Czech. In: International Journal of Corpus Linguistics,10, 4, 453 - 467. John Benjamins,
Amsterdam - Philadelphia 2005. ISSN 1384-6655
Čermák, F. (2008): An Author´s Dictionary: The Case of Karel Čapek, In Proceedings of the
XIII Euralex International Congress, Institut Universitari fr Linguistica Aplicada Universitat
Pompeu Fabra, Barcelona 2008, pp.323-332
Čermák, F. (2009): Parallel Corpora: The Case of InterCorp, Corpus Linguistics Conference
2009, Liverpool, Great Britain
Čermák, F. (2009): Spoken Corpora design. In: International Journal of Corpus Lingvistics
14,1, 113-122. John Benjamins, Amsterdam - Philadelphia
Kocek, J.-Kopřivová, M.-Schmiedtová, V. (2000): The Czech National Corpus. Proceedings of
the 9th EURALEX International Congress, Heid U., Evert S., Lehmann E., Rohrer Ch. (eds.),
Stuttgart 2000, s. 127 – 132
Kopřivová, M.-Waclawičová, M. (2006): Representativeness of Spoken Corpora on the
Example of the New Spoken Corpora of the Czech Language In: Proceedings of the
international conference „Corpus linguistics – 2006“ St. – Petersburg University Press. St.
Petersburg pp.174-181
Křen, M. (2006): Frequency Dictionary of Czech: A Detailed Processing Description. In: Insight
into Slovak and Czech Corpus Linguistics, Šimková, Mária (ed.), pp. 16 - 25. Veda, Bratislava
2006.
Křen, M. (2006): SYN2000 vs. SYN2005: Comparing the Large Synchronic Corpora of Czech.
In: Труды международной конференции "Корпусная лингвистика - 2006", с. 182 - 189.
Издательство СПбГУ, Санкт-Петербург 2006.
Křen, M. (2008): Compilation of the Dictionary of Karel Čapek. In: Corpus Linguistics,
Computer Tools, and Applications - State of the Art, Barbara Lewandowska-Tomaszczyk
(ed.), pp. 469 - 481. Peter Lang, Frankfurt am Main.
Křen, M.-Hlaváčová, J. (2008): Corpus as a Means for Study of Lexical Usage Changes. In:
Proceedings of the 13th EURALEX International Congress, Bernal, Elisenda - DeCesaris,
Janet (eds.), pp. 437 - 447. Barcelona 2008.
Křen, M. (2009): The SYN Concept: Towards One-Billion Corpus of Czech, Corpus Linguistics
Conference 2009, Liverpool, Great Britain
Schmiedtová, V. (2006): What did the totalitarian language in the former socialist
Czechoslovakia look like? Conference of The Slavic Linguistics Society 2006,
http://www.indiana.edu/~sls2006/page2/page2.html, downloaded http://ucnk.ff.cuni.cz/english/stahni.php#schmiedt
Waclawičová, M. (2007): Spoken Corpus ORAL2006, Information It Provides ang General
Caracteristics of Spoken Text. In: Computer Treatment of Slavic and East European
Languages, eds. J. Levická, R. Garabík. Tribun, Bratislava, Brno 2007, pp. 283-289
Waclawičová, M.-Křen, M. (2008): ORAL2008: New Balanced Corpus of Spoken Czech. In:
Труды международной конференции "Корпусная лингвистика - 2008", pp. 105 - 112.
Издательство СПбГУ, Санкт-Петербург 2008. ISBN 978-5-288-04769-5
Publications based on ICNC corpora:
Cvrček, V. 2008: Regulace jazyka a Koncept minimální intervence. Nakladatelství Lidové
noviny, Praha, ISBN 978-80-7106-600-2
Čermák, F. (ed.) 2007: Slovník Karla Čapka. Nakladatelství Lidové noviny, Praha, ISBN 97880-7106-915-7
Čermák, F. (ed.) 2007: Frekvenční slovník mluvené češtiny. Karolinum, Praha, ISBN 978-80246-1425-0
Čermák, F. - Blatná, R. (eds.) 2005/2007: Jak využívat Český národní korpus. Nakladatelství
Lidové noviny, Praha .
ISBN 80-7106-736-9
Čermák, F. - Blatná, R. (eds.) 2006: Korpusová lingvistika: Stav a modelové přístupy.
Nakladatelství Lidové noviny, Praha.
ISBN 80-7106-861-6
Čermák, F. - Šulc, M. (eds.) 2006: Kolokace. Nakladatelství Lidové noviny, Praha 2006, ISBN
80-7106-863-2
Čermák, F. - Křen, M. (eds.) 2004: Frekvenční slovník češtiny. Nakladatelství Lidové noviny,
Praha, ISBN 80-7106-676-1
Čermáková, A. 2009: Valence českých substantiv. Nakladatelství Lidové noviny, Praha 2009,
ISBN 978-80-7106-426-8
Esvan, F. 2007: Vidová morfologie českého slovesa. Nakladatelství Lidové noviny, Praha,
ISBN 978-80-7106-913-300
Kopřivová, M. - Waclawičová (eds.) 2008: Čeština v mluveném korpusu. Nakladatelství
Lidové noviny, Praha, ISBN 978-80-7106-982-9
Kopřivová, M. 2006: Valence českých adjektiv. Nakladatelství Lidové noviny, Praha, ISBN 807106-862-4
Šonková, J. 2008: Morfologie mluvené češtiny: Frekvenční analýza. Nakladatelství Lidové
noviny, Praha, ISBN 978-80-7106-956-0
Appendix
Corpora writen language (contemporary)
corpus
size
(number of
words)
lemmatizazion
tags
corpus characterization
SYN2006PUB
300 mil.
Yes
Yes
corpus of newspapers texts from 1989 to 2004
SYN2005
100 mil.
Yes
Yes
gendres balanced corpus, predominate texts from
2000 to 2004
SYN2000
100 mil.
Yes
Yes
gendres balanced corpus, predominate texts from
1989 to 1999
FSC2000
100 mil.
Yes
Yes
adapted SYN2000, a reference source for the
Frequency Dictionary of the Czech Languge
KSK
800.000
No
No
transcripted private letters from1990 to 2004
years
ORWELL
80.000
Yes
Yes
manually tagged corpus the novel "1984" by
Orwell
Corpora writen language (contemporary)
corpus
size
(number of
words)
lemmatizazion
tags
corpus characterization
ORAL2008
1 mil
No
No
sociolinguisticly balanced corpus of informal
spoken Czech
ORAL2006
1 mil.
No
No
corpus of informal spoken Czech
PMK
675.000
No
No
Prague Spoken Corpus
BMK
490.000
No
No
Brno Spoken Corpus
Corpora writen language (diachronic)
corpus
size
(number of
words)
lemmatizazion
tags
corpus characterization
DIAKORP
1,6 mil.
No
No
diachronic corpus
size
(number of
words)
lemmatizazion
tags
corpus characterization
Yes
(partly)
(partly)
paralellel corpora
Parallel corpora
corpus
InterCorp
31 mil.Czech
part
34,5 other
languages