5 May2014
Katrin Heinze
Testing of
XBRL files for the
supervisory reporting
Eurofiling Workshop in Rome
Rubric
Objectives
The objective of this presentation is
• to provide an overview on the testing strategy of the SUBA project
• to explain the test plan and sample test cases
• to inform about the next steps
Eurofiling Workshop in Rome, 5th of May 2014
Page 2
www.ecb.europa.eu ©
Rubric
Agenda
Overview
1
SUBA* test strategy
2
Sample XBRL test cases
3
Next steps
* SUpervisory
Eurofiling Workshop in Rome, 5th of May 2014
Page 3
BAnking Data System
www.ecb.europa.eu ©
Rubric
Agenda
Overview
1
SUBA Test Strategy
2
Sample XBRL test cases
3
Next steps
Eurofiling Workshop in Rome, 5th of May 2014
Page 4
www.ecb.europa.eu ©
Rubric
SUBA Test
Strategy
Functional Tests
Non-Functional Tests
 Based on
 User requirements
 Use cases
 Experience of users
 Technical requirements
 Source code (unit tests)
 …
175
 Categorised following
 Load tests (number of users)
 Performance tests (processing time)
 Volume test (quantity of data)
 Stress test
 Test of the (data) security
 Usability tests
 …
Technical Test Execution Status
17.04.2014
Regression Tests
 Done after changes on
 Components
o XBRL taxonomy
o Validation rules
 Interfaces
o EXDI
o RIAD
 Infrastructure
 …
114
33
28
Total
Eurofiling Workshop in Rome, 5th of May 2014
No run
Failed
Page 5
Passed
www.ecb.europa.eu ©
Rubric
SUBA Test
Strategy
HP ALM usage within the SUBA project
Scope
Achievements
Way forward
Eurofiling Workshop in Rome, 5th of May 2014

Planning of releases and test cycles

Enable traceability between requirements, tests, defects and
according code changes

Increase visibility with dashboards on analysis and KPIs

Definition of test cases linked to the defined requirements

Execution of test cases based on defined test data

Defect tracking and documentation of bug fixes

Increase the level of automation on test execution

Customise reports and analysis based on the project needs

Preparation of test cases for the next release
Page 6
www.ecb.europa.eu ©
Rubric
Test plan
structure
1. level: Categories
2. level: Use cases
3. level: Test cases
Structure in
the Test Lab
Eurofiling Workshop in Rome, 5th of May 2014
Page 7
www.ecb.europa.eu ©
Rubric
Tracking
of the test progress in HP ALM
Eurofiling Workshop in Rome, 5th of May 2014
Page 8
www.ecb.europa.eu ©
Rubric
Agenda
Overview
1
SUBA Test Strategy
2
Sample XBRL test cases
3
Next steps
Eurofiling Workshop in Rome, 5th of May 2014
Page 9
www.ecb.europa.eu ©
Rubric
Sample
XBRL test cases
Positive functional test cases
Creation of test data for
 each module of the COREP taxonomy
 which is a valid XBRL instance document.
Valid according to
 XML specification
 XBRL 2.1 specification
 XBRL Dimension 1.0 specification
 XBRL Formula 1.0 specification
 EBA disabled validation rules
 EBA Filing Rules for COREP and FINREP
By using different file sizes
 small
 medium
 large
Eurofiling Workshop in Rome, 5th of May 2014
Page 10
www.ecb.europa.eu ©
Rubric
Sample
XBRL test cases
Negative functional test cases
 Tests that ensure that the application shows a proper behaviour when invalid
files are received.
 Creation of test data where the system is expected to provide correct error
messages.
 Possible test cases : A reporting entity sends a file which
 is not a well-formed XML document.
 has not the correct format, i.e. jpg, txt, xls etc.
 has not the expected encoding.
 includes neither facts nor contexts.
 misses a namespace in the xbrli root element
 includes letters where numbers are expected as fact value
 misses an explicit member in the scenario element
 Includes a fact with no value but also no nil attribute is provided
 Refers to a non-existing module
 Refers to two different modules.
Eurofiling Workshop in Rome, 5th of May 2014
Page 11
www.ecb.europa.eu ©
Rubric
Agenda
Overview
1
SUBA Test Strategy
2
Sample XBRL test cases
3
Next steps
Eurofiling Workshop in Rome, 5th of May 2014
Page 12
www.ecb.europa.eu ©
Rubric
Next steps
Voluntary functional tests with NCAs
Test scope
Test period: 13th to 15th of May
Test activities:
•
Acquisition
 NCA sends data to EXDI via WebUI or standard adapter.
 EXDI system sends a delivery message.
 SUBA system sends an acknowledgement message to NCA with the XBRL
validation result
 NCA receives the messages in the scheduled time periods.
Eurofiling Workshop in Rome, 5th of May 2014
Page 13
www.ecb.europa.eu ©
Rubric
Next steps
Load tests
First estimation about the load expected for the first remittance date
Number of reported entities
128
First remittance date NCAs to EBA/ECB
Modules reported
31.07.2014
Number of files
(per institute)
Max. file size
in MB
Max. number of
observations
COREP consolidated
Own Funds
1
115
177.536
Large Exposures
1
264
355.930
NSFR
1
9
7.510
49.664
69.244.928
Total
*)
512
Estimations based on information of the Deutsche Bundesbank for one big German bank
IT infrastructure for establishing a supervisory function at the
ECB
14
Rubric
Next steps
Organisation and approach – Test case preparation
Specification
workshops
Specification
sign-off
• Development of software
• Definition of test cases (HP
ALM)
Control point A
Control
point
Tests
execution
Implementation
• Tests runs (HP ALM)
• Defects workflow (HP ALM)
• Bug fixing
Control point B
Metric
Control point C
Target (by priority)
Average
High
Medium
Low
A
% reviewed requirements
100%
100%
100%
100%
B
% requirements test coverage
75%
100%
100%
50%
C
% tests run
90%
100%
100%
75%
% tests succeeded
100%
100%
100%
100%
3
3
5
5%
5%
10%
Defect resolution time (working days)
Defects reopened ratio
Eurofiling Workshop in Rome, 5th of May 2014
Page 15
Rubric
Next steps
Key issues and challenges

Supporting the voluntary functional tests with
the NCAs, organized by WG SIM

Defining scenarios for load tests on XBRL processing
via automated test, simulating also peaks
Preparation for
next releases

Defining test cases based on more
precise user requirements
EBA testing

Participating in tests scheduled by EBA
Functional tests
Load tests
Eurofiling Workshop in Rome, 5th of May 2014
Page 16
www.ecb.europa.eu ©