RDA’s Recently Endorsed Outputs
September 16, 2015
Agenda






Introduction
Data Foundation and Terminology –
PID Information Types
Practical Policy
Data Type Registries
Questions
2
Data Foundation and Terminology
- talking the same language –
Peter Wittenburg, Gary Berg-Cross, Raphael Ritz
Summary of the Problem
4
 What is the problem?
 Data organizations (DOrg) and ideas about it are all different
 We are all speaking different languages, wasting time and
misunderstanding each other in any project involving data
 Different DOrgs make data discovery and integration very time
consuming, inefficient and thus expensive
 Different DOrgs prevent us developing maintainable support software
 Who is impacted (specific domains, professions, etc.)?
 All efforts to integrate data, for example in federations, BDA projects, etc.
 What are the ramifications of not having the problem
resolved?
 Combining data of all sorts across different origins (projects, repositories,
disciplines, etc.) is a nightmare and requires a lot of curation and
transformation before the actual scientific analysis can start
Highlights of the Effort and Deliverables
 Working Group structure (how many members, diversity
of experience, geographies, etc.)
 DFT WG had 60 members coming from almost all regions
 Members came from different types of institutions and disciplines
 DFT WG included relative newcomers up to members with much
experience from data intensive projects
 DFT WG produced
 a list of core terms essential to harmonize conceptualization of data
organizations
 a graphical model relating the terms
 a set of auxiliary documents including many use cases to demonstrate the
bottom-up approach and research of the WG
 a Term Tool (using Semantic Media Wiki) to store definitions and allow
editing, classification and discussion of terms (which is also open for other
groups)
5
Active Contributors to the Work
Institute/Project
Country/ Region
Domain
CNRI
US
IT Research and Systems
U Cardiff
UK
IT Research and Systems
AWI
DE
Oceanography & Environment
MPG
DE
Research Organisation
EUDAT
EU
Data Infrastructure
CLARIN
EU
Linguistic Research Infrastructure
EPOS
EU
Earth Observation Res. Infrastructure
ENES
Int
World Climate Res. Infrastructure
ENVRI
EU
Environmental Res. Infrastructure
DataOne
US
Environmental Infrastructure
ESSD/RENCI
US
Earth Science System Data
NCGEN/RENCI
US
Clinical Genomics
Europeana
EU
Humanities Infrastructure
DataCite/EPIC
Int
PID Infrastructures
DICE
US
IT Research and Systems
CAS
CN
Earth Science Model
ADCIRC/RENCI
US
Ocean and Storm modeling
6
Impact of the Deliverable
7
 Who was impacted by deliverable?
 The European data infrastructure EUDAT is federating data from many
discipline repositories where each data collection has a different data
organization. If integration is not simply done at physical level (file
structures), this heterogeneity makes it very costly to integrate all data to
enable re-purposing and to make it accessible at different repositories.
 The Technology Director of the international CLARIN project said:
 Very handy to have a lingua franca when discussing research infrastructure
architectures
 It was good to be involved as adopting community from the start of the work
 Similar experiences are made by US, Chinese etc. colleagues that work
on large scale data integration. Integration work is special and thus does
not scale. Even the integration of a simple database of animal voices of
the world (11 TB) requested the development of special scripts to extract
metadata, relations, rights etc. in addition to the data files
 Harmonization would reduce integration time by large factors and had
already great effects on interaction efficiency and integration.
Endorsements/Adopters and how have they used
the deliverable
8
 Our adopters
 The early adopters are to a certain extent those who have these
dramatic problems in data integration such as EUDAT, CLARIN, etc.
 Their approach was aligned with the progress of the WG discussion. All
their repository setups adhere now to the DFT model and their
interaction with different communities are based on it: central is the
Digital Object, that is described by metadata, is associated with a
Persistent ID and whose instances are stored in trustful repositories (see
simplified diagram)
persistent ID
digital
object
bitstream
repository
metadata
 Also several other projects, for example from humanities, health,
bioinformatics, neuroinformatics and atmosphere research adopted the
basic & simple model and the terminology.
Endorsement/Adoption
9
Institute/Project
Country/ Region
Domain
CNRI
US
IT Research and Systems
U Cardiff
UK
IT Research and Systems
MPG
DE
Research Organisation
EUDAT
EU
Data Infrastructure
CLARIN
EU
Linguistic Research Infrastructure
EPOS
EU
Earth Observation Res. Infrastructure
ENES
Int
World Climate Res. Infrastructure
ENVRI
EU
Environmental Res. Infrastructure
ESSD/RENCI
US
Earth Science System Data
NCGEN/RENCI
US
Clinical Genomics
DICE
US
IT Research and Systems
ADCIRC/RENCI
US
Ocean and Storm modeling
Deep Carbon Project
US
Environmental/Athmospheric Research
Note: There may be more projects/institutes that have endoresed
or adopted the DFT model without noticing us.
How You Can Endorse
10
 Who could use the DFT Terminologies?
 The vocabulary is openly available for everyone who wants to run a project
including those with large data collections
 The organization should be strictly compliant to the model to guarantee
independence and thus easy re-purposing of all components
 The vocabulary is openly available for everyone who is working in a data
federation project integrating data from different sources or who wants to
re-purpose data for data intensive science
 Projects could use the DFT WG model as a common reference model to design
transformations
 Projects could use the suggested terminology to achieve quick, mutual
understanding
 Software developers can adopt this basic model to make sure that their
software can be used by almost everyone adhering to state of the art
principles
How You Can Endorse
11
 How to access and use them
 Take the “Core Terms and Model” document which provides the
final model and the corresponding terms and apply it in your
project
 In case of questions
 Read the supplementary documents to understand
conceptualization and background for choices
 Meet the WG co-chairs and experts at a plenary
 Contact the WG co-chairs
 Contribute to the now functioning DFT IG (email, wiki, Term Tool)
 Send a request to the RDA Europe support team (email, wiki)
(references see last slide)
Next Steps
 Are there plans to further evolve this deliverable?
 Yes, since the WG just focused on the basic set of core terms, and
additional RDAS WGs are completing work so there is much more out
there where terminology harmonization would help substantially
 We also see the need to consider the dynamics of the field and to be
ready to adapt current definitions and perhaps even the model
 Is there an IG or WG that individuals can join on a
related topic?
 Yes, a follow-up DFT Interest Group has been established and will
meet at Plenary 6
 A larger scope of integrated work is being discussed as part of the
Data Fabric IG
12
Contact Information
 Who can individuals contact to learn more about this
deliverable?
 DFT WG:
https://rd-alliance.org/groups/data-foundation-and-terminology-wg.html
 DFT IG:
https://rd-alliance.org/groups/data-foundations-and-terminology-ig.html
 TeD-T Term Definition Tool:
http://smw-rda.esc.rzg.mpg.de/index.php/Main_Page
 RDA EU Support Team:
dmp@europe.rd-alliance.org
13
PID Information Types:
Towards PID interoperability
Tobias Weigel (DKRZ / University of Hamburg)
Tim DiLauro (Data Conservancy / Johns Hopkins University)
Summary of the Problem
 Move from management of files towards management
of objects
 How does object management scale with increasing numbers?
 How do we further automate our processes?
 Issues independent from particular disciplines, repositories,
management approaches
 Understanding the most elemental characteristics of
digital objects – for machine agents and human users
 Facilitate interoperability across PID systems and
simplify PID record usage
 Not addressing these key challenges is likely to lead to
insular solutions and reiteration of efforts
15
Highlights of the Deliverables
16
 More than 50 group members from EU/US/AU
 A lot of technical expertise and community experience
 Key deliverables (cf. summary report):




Conceptual insights on types and their possible structures
Practical type examples geared towards diverse use cases
Openly licensed API specification and Java-based prototype
Approach for using a general type registry
IDENTIFIER
Verification service
properties
size
checksum
timestamps
aggregation
version
license
format
Size:
Format:
Checksum:
Date:
Size:
Checksum:
Format:
License:
Impact of the Deliverable
 Some initial types have been registered, making it
possible to explore further applications
 Information on how to register new types available in the report
 Registration relies on the Type Registry
 Incited plans in communities and projects about
concrete applications
 PIDs and typing increasingly seen as a crucial
component to decouple management of objects from
contents
 Simplify client access to data across domains, implementations and
changes in information models
 More lightweight access to information on less accessible objects
17
Endorsements/Adopters
18
 Adopters can be:
 Communities who can use existing types and share custom types, as
well as build tools and services that exploit them
 PID service providers who can offer a typing service as added value
beyond registration and resolution, increasing PID interoperability
Adopter
Category
Country
Scope / Goal
ENES
Community
Int.
IPCC AR6 data management
DCO-DS/RPI
Community
US
Enhancing existing PID usage
EUDAT
Community/Service
provider
EU
Added-value service to various
disciplinary communities
MGI/NIST
Community
US
Automation of data type conversions
EPIC
Service provider
EU
CNRI
Service provider
US
DONA
Service provider
Int.
Generic added-value service
How You Can Endorse
19
 Make use of existing types, invent your own and please
tell us about it!
 Follow-up RDA WGs on Collections and Data Typing will continue the
work on concrete types. The PID Interest Group is also a good place to
provide general feedback.
 Specification and prototype source code are openly
available
 Possible development by EUDAT, DCO, ENES and others as
interested adopters
 Offer by PID service providers as a service beyond
registration and resolution
 Contribution to a unified type registry is encouraged
Next Steps and Contact Information
 PID Information Types WG
 https://rd-alliance.org/groups/pid-information-types-wg.html
 PID Interest Group
 https://rd-alliance.org/groups/pid-interest-group.html
 PID Collections candidate WG
 https://rd-alliance.org/groups/pid-collections-wg.html
 https://rd-alliance.org/pid-collections-p6-bof-session.html
 Data Typing BoF
 https://rd-alliance.org/data-typing-p6-bof-session.html
 personal contact: weigel@dkrz.de
20
Working Group
Practical Policy
based on slides and latest documents from the PP WG
chaired by Reagan Moore, Rainer Stotzka
Summary of the Problem
Computer actionable policies are used to
 enforce data management
 automate administrative tasks
 validate compliance with assessment criteria
 automate scientific data processing and analyses
Practical Policy
Assertion or assurance that is enforced about a (data) collection
(data set, digital object, file) by the creators of the collection
Users motivated by issues related to scale, distribution
22
Policy Templates
23
 Practical Policy members represented
 11 types of data management systems
 30 institutions
 2 testbeds
 iRODS
Renaissance Computing Institute,
DataNet Federation Consortium – DFC
 GPFS
Institute of Physics of the Academy of Sciences, CESNET
Garching Computing Centre – RZG
 Published two documents
 Moore, R., R. Stotzka, C. Cacciari, P. Benedikt, “Practical Policy Templates”
February, 2015, http://dx.doi.org/10.15497/83E1B3F9-7E17-484A-A466B3E5775121CC.
 Moore, R., R. Stotzka, C. Cacciari, P. Benedikt, “Practical Policy Implementations”,
February, 2015, http://dx.doi.org/10.15497/83E1B3F9-7E17-484A-A466B3E5775121CC.
Production Environments
24
 Computer actionable rules to enforce:
 Preservation standards
 Authenticity, integrity, chain of custody, arrangement
 Data management plans
 Collection creation, product generation, publication, storage,
archives
 Data distribution
 Replication, content distribution network
 Publication
 Descriptive metadata, time dependent access controls
 Processing pipelines
 Workflow execution
Endorsements/Adopters
 Distributed data management environments
 EUDAT Data Policy Manager
 B2SAFE use case




International Neuroinformatics Coordinating Facility
Institut national de physique nucléaire et de physique des particules
New Zealand BESTGRID
DataNet Federation Consortium






NSF data management plans
Odum Institute preservation archive
The iPlant Collaborative genomics data grid
Science Observatory Network digital library
SILS LifeTime Library
HydroShare
 NOAA National Climatic Data Center
 NASA Center for Climate Simulations
25
Applications
26
 Policy-based collection management






Purpose for assembling the collection
Properties required to support the purpose
Policies that control when and where the properties are enforced
Procedures that execute operations controlled by the policies
Persistent state information that is generated by the procedures
Periodic assessment criteria that verify compliance
 RDA Publications
 Policy templates
 Constraints, operations, required state information
 Policy implementations
 Computer actionable rules to automate policy enforcement
Next Steps and Contact Information
27
 Data Fabric Interest Group
 Policies to support
 Federation
 Interoperability
 Data Foundations and Terminology Interest Group
 Vocabulary for policy management
 Interoperability testbeds
 EUDAT
 http://eudat.eu/data-access-and-reuse-policies-darup
 National Data Service
 http://www.nationaldataservice.org
 DataNet Federation Consortium
 http://datafed.org
Data Type Registries
Larry Lannom, CNRI
Daan Broeder, Meertens Institute, KNAW
Summary of the Problem





29
Data sharing requires that data can be parsed, understood, and reused by
people and applications other than those that created the data
How do we do this now?
 For documents – formats are enough, e.g., PDF, and then the
document explains itself to humans
 This doesn’t work well with data – numbers are not self-explanatory
 What does the number 7 mean in cell B27?
Data producers may not have explicitly specified certain details in the data:
measurement units, coordinate systems, variable names, etc.
Need a way to precisely characterize those assumptions such that they
can be identified by humans and machines that were not closely involved
in its creation
Affects all data producers and consumers
Goal of the DTR Effort: Explicate and Share
Assumptions using Types and Type Registries
30
 Evaluate and identify a few assumptions in data that can be
codified and shared in order to…
 Produce a functioning Registry system that can easily be evaluated
by organizations before adoption
 Highly configurable for changing scope of captured and shared
assumptions depending on the domain or organization
 Supports several Type record dissemination variations
 Design for allowing federation between multiple Registry instances
 The emphasis is not on
 Identifying every possible assumption and data characteristic
applicable for all domains
 Technology
Highlights of the Deliverable





31
Confirmation that detailed and precise data typing is a key consideration in
data sharing and reuse and that a federated registry system for such types
is highly desirable and needs to accommodate each community’s own
requirements
Deployment of a prototype registry implementing one potential data model,
against which various use cases can be tested
Involvement of multiple ongoing scientific data management efforts, across
a variety of domains, in actively planning for and testing the use of data
types and associated registries in their data management efforts
Integration with one additional RDA WG (Persistent Identifier Types) and at
least one Interest Group (RDA/CODATA Materials Data, Infrastructure &
Interoperability IG)
Development of a set of questions that require further consideration before
a detailed recommendation on data typing can be issued
Impact of Use Case: Process Use Case
32
3
Users
2
1
Federated Set of Type
Registries
4
ID
ID
ID
ID
Type ID
Type ID
Type
Type
Payload
Type Payload
Type
Payload
Payload
Payload
4
Payload
Typed Data
Terms:…
I Agree
10100
Visualization
11010
Rights
101….
Data Set
Data Processing
Dissemination
Services
1 Client (process or people) encounters unknown data type.
2 Resolved to Type Registry.
3 Response includes type definitions, relationships, properties, and possibly service pointers. Response can be
used locally for processing, or, optionally 4 typed data or reference to typed data can be sent to service provider.
Endorsements/Adopters
 Materials Science Adoption Project
 Demo at P6
 X-ray diffraction use case
 normalize data sets resulting from multiple proprietary instruments
 Enable a homogenous analysis platform for data consumers to perform their
analyses
 Deep Carbon Observatory
 Goal: given a dataset identifier, discover detailed information about the structure(s)
within that dataset, and act accordingly
 DTR is a registry used for explicating structures in the form of type records
 Facilitate norms of behavior relevant to data curation and re-use
 Digital Object Identifier
 Given a DOI, what services are relevant and applicable
 Having chosen a service, how can a client invoke that service?
 Having invoked a service, how can a client process the returned data?
33
How You Can Endorse
 Start a new prototype effort
 Follow existing prototype efforts
 Attend the BOF at P6
 Join the Data Typing WG when it starts
 Try the public prototype at typeregistry.org
34
Next Steps and Contact Information
 A follow-up WG is planned: Data Typing
 Leverage results of DTR
 Collect results from multiple prototypes
 Best practices for federation
 BOF on Data Typing at P6: 24 Sept., Breakout #6
 Proposed Chairs of Data Typing WG
 Giridhar Manepalli, CNRI
 Simon Cox, CSIRO
 Tobias Weigel, DKRZ
 Larry and Daan are still around 
35