Sharing Resources in
Jan Odijk, Arjan van Hessen
Chiang Mai, Thailand,
12 Nov 2011
Long Term Preservation
National project in the Netherlands
Budget: 9.01 m euro
Funding by NWO (National Roadmap Large
Scale Infrastructures)
• Coordinated by Utrecht University
• 24 partners (universities, royal academy
institutes, independent institutes, libraries, etc.)
• Dutch National contribution to the Europe-wide
CLARIN infrastructure
• Prepared by CLARIN preparatory project (20082011)
– Also coordinated by Utrecht University
• From Dec 2011 to be coordinated by the
– ERIC: a legal entity at the European level
specifically for research infrastructures
CLARIN infrastructure (NL)
• An technical research infrastructure in
which a humanities researcher who
works with language-related resources
– Can find all data relevant for the research
– Can find all tools relevant for the research
– Can apply the tools to the data without any
technical background or ad-hoc adaptations
– Can store data resulting from the research
– Can store tools resulting from the research
via one portal
CLARIN infrastructure (NL)
• This requires systematic sharing of
resources (=data, tools, web services, …)
• Systematic Sharing requires
Long Term Preservation
of resources
CLARIN-NL subprojects
• Resource curation projects
– Curate an existing resource
• Demonstrator projects
– Curate an existing tool and supply a
demonstration scenario
• #subprojects 21 (12-14 in 2012)
• Data Curation Service
– Offers the service of curating existing data
• Where curation includes
– Documentation, Visibility, Referability, Accessibility,
Long Term Preservation, Interoperability
• CLARIN infrastructure is virtual and distributed
– CLARIN-Centres work together to implement the infrastructure
– Each stores and makes available a part of the resources
– Some also provide computational facilities
– Centres must meet a list of requirements and be certified by
• Candidate CLARIN Centres in NL
Institute for Dutch Lexicology (INL)
Max Planck Institute for Psycholinguistics (MPI)
Meertens Institute (MI)
Huygens ING Institute (HI)
Data Archiving and Networked Services (DANS)
Infrastructure Implementation
• Implementation of basic infrastructure functionality
– setting up authentication and authorizations systems
– several registries (e.g. ISOCAT, RELCAT, Metadata Registry)
– various other infrastructure services
• Search Facilities
– In resource descriptions (`metadata’)
• Centralized after metadata harvesting
– In the data themselves
• Via federated search
• Using Webservices in Workflow systems
– Cooperation with Flanders
– Based on work done in the STEVIN-programme
– (as a severe test for interoperability)
• Is always necessary, so hardly any additional effort
• Partly in natural language
• Partly formalized
– Described under a particular formally identifiable attribute
– With an explicit type for the value of the attribute
– Possibly with further restrictions on the values (patterns, finite
lists of values, constraints, etc.)
– Represented formally and unambiguously
• Any piece of documentation that can be formalized must
be formalized, and must be put in the resource
description (metadata of the resource)
• Resource Descriptions
– Component-based MetaData Infrastructure (CMDI)
– One can define resource profiles as collections of components
(which can contain components).
– Many generally useable components are available
– Resource profiles for most common resources are available
– Component-based  flexibility
– Flexibility: danger: diversity, no interoperability
– Controlled by semantic interoperability (see below)
– Not yet available but needed: profile(s) for tools
• Supported by tools
– Component and profile editors
– Component and profile registries
– Metadata editor
• Each resource and its resource description must be
stored at a CLARIN-centre
• CLARIN-centres make resource descriptions available
for metadata harvesting (using OAI-PMH)
• Via harvesting the metadata, the metadata become
available in the CLARIN resource catalogue
– browsing via the Virtual Language Observatory (VLO) using
faceted browsing
– Search via a search interface (under development)
• In the metadata and in the data
• String search and structured search
• Results if desired collected in a Virtual Collection
• By name or title is not sufficient
– All the problems that natural language poses for communication:
• not always unique (ambiguity)
• language-specific Corpus Gesproken Nederlands
– Variants in other languages: Spoken Dutch Corpus
– limited knowledge of the foreign language  variants: Corpus Spoken Dutch, Dutch Spoken
• Long, too redundant,
– abbreviations/acronyms: CGN
• Invites for errors
– Spoken Dutch Cropus, Spken Dutch Corpus
• URLs
– Still too long/redundant (unless one uses shortened URLs)
– Unstable, volatile
• Persistent Identifiers (PIDs) are needed
• PIDs
• Each CLARIN-Centre
– must assign a PID to each resource (and/or to
– Keep the PID resolution registry up-to-date
• PID systems
– Handle (preferred)
– Perhaps others (e.g. DOI)
• CLARIN infrastructure
– Accessible at any time and from any place
– CLARIN-NL promotes maximal open access of resources
– is working on plans to implement policies and functionality to
properly handle IPR and ethical restrictions
• Researchers’ Mindset
– Many researchers in the humanities are hesitant or even
unwilling to share their resources with others
– How to resolve this? With a carrot and a stick
• CLARIN must accommodate reasonable wishes
• CLARIN must prove benefits for researchers who put their resources there
• Funding agencies must oblige researchers to do so (partially already so)
Long Term Preservation
• Necessary to make sure the resources can be shared
with future researchers (that may be the producer!)
• Each CLARIN-Centre is obliged to ensure long term
• Usually outsources to specialized centres
– MI outsources to DANS
– MPI outsources to internal Max Planck Gesellschaft organisation
• Interoperability of resources is the ability of resources to
seamlessly work together
– No manual ad-hoc adaptations
– Adaptations occur automatically `behind the screens’
• Need for interoperability is high
– Humanities researchers: not the required technical background
• Interoperability
– Syntactic interoperability and Semantic interoperability
• Each subproject must try to achieve interoperability
– Report any problems and make suggestions for adaptations
– So that the resources are adapted to the infrastructure (in some
cases) and vice-versa (in other cases)
• Not easy, but the only way to get further is to actually try
this and learn from it.
Syntactic Interoperability
• the formats of data are selected from a limited set of (de
facto) standards or best practices supported by CLARIN
• software tools and applications take input and yield
output in these formats
Semantic Interoperability
• Focus on the semantics of Data Categories (DCs)
• a privileged data category registry (DCR) is set up containing DCs:
unique persistent identifiers for DCs (PIDs),
their semantics,
a definition,
lexicalizations in various languages.
• Each resource specific DC mapped to DC from the
privileged DCR.
• every researcher can use his/her own DCs
• different DCs from different resources can be
interpreted as identical in meaning, via the DC of the
privileged DCR
• In CLARIN-NL multiple (complementary) privileged
DCRs are allowed. The primary is ISOCAT
Semantic Interoperability
• Achieving semantic interoperability is very hard
– Many DCs are almost identical
(principled/pragmatic/arbitrary reasons)
– Some DCs in ISOCAT are not defined clearly
– There are many similar DCs in ISOCAT
– Relevant DCs are not easy to find in ISOCAT
• Three actions taken
– Held several workshops to discuss problems
– Appointed a coordinator to deal with problems
– Decided to implement RELCAT registry to
specify relations between DCs
• CLARIN-NL requires systematic sharing of resources
• Therefore requires researchers to work on
– Documentation
– Visibility
– Referability
– Accessibility
– Long Term Preservation
– Interoperability
Of resources
• For certain aspects this is relatively easy but it must be done
• For other aspects this is very hard but it must be done so that we
can learn
• The approach described here may be a model for other countries
working on the CLARIN-infrastructure
• It may be a model for other resource sharing facilities (e.g. METASHARE)
Thanks for your attention!