File Analyser test version
A3.1 (FILES)
DPP Compliance Programme
AMWA / UK DPP -- AS-11 UK DPP HD Shim v1.1
PART 3 – File Testing and Reporting (for PART 1 & PART 2)
DPP Lab, BBC R&D, Centre House, 56 Wood Lane, W12 7SB, UK
CONTENTS
1.
2.
3.
4.
5.
DOCUMENT OVERVIEW
FILE TESTING PROCEDURE
DOWNLOAD OF TEST FILES
TESTING INSTRUCTIONS
OTHER REQUIRED INFORMATION
APPENDIX A – MXF Assessment Criteria
APPENDIX B - Overall Analyser Testing Procedure
1 DOCUMENT OVERVIEW
This document is concerned with practical FILE TESTING of devices whose primary
function is File Format Analysis of AS-11 DPP HD files.
It constitutes PART 3 of the overall test procedure for File Analysers (as described
in Appendix B). It includes a set of PART 1 TEST FILES (Testing MXF) and a set of
PART 2 TEST FILES (Testing Descriptive Metadata).
This PART 3 document is also appropriate for any device which is being assessed
for Descriptive Metadata validation testing. In this case only the PART 2 document
and test files will apply, covering Descriptive Metadata.
2 FILE TESTING
PROCEDURE
1. Follow the DOWNLOAD OF TEST FILES instructions below
2. Follow TESTING INSTRUCTIONS below
3. Email the TEST RESULTS, together with OTHER REQUIRED INFORMATION
detailed below, to the DPP at the following:
complianceprogramme@digitalproductionpartnership.co.uk
4. The test results will then be reviewed by the Compliance Programme to
determine if the product correctly identified each file as a Pass or Fail.
5. The OEM will be advised of the outcome.
Page 1 of 5
3 DOWNLOAD OF
TEST FILES
1. PART 1 - TEST FILES (MXF)
A specific URL will be provided for PART 1 (MXF) test files once the required PART 1
document has been reviewed and passed by the Compliance Programme. You should
download all PART 1 test files as we require the device being tested for MXF to assess
all files in this set.
2. PART 2 - TEST FILES (DM)
The following URL will allow you to download PART 2 (DM) test files. You should
download all PART 2 test files as we require the device being tested for descriptive
metadata to assess all files in this set. Go to the following URL for these:
URL:
4 TESTING INSTRUCTIONS
1. Select the specific test plan, or profile, or tick box of a specific version of your
product. This should be what someone buying the product can use operationally.
This is what is being tested.
2. Test every file in the test set, and record the results for each.
3. The recorded information only has to say whether the test plan/profile declares
the file a Pass or a Fail.
4. For ease of assessment by the Compliance Programme, the recorded information
for each file should ideally be presented as a *.csv file, or simply as a comma
separated list. This should be arranged with “File_name,Result” and continue in
that order. For example:
File 1,Pass
File 2,Fail
File 3,Fail
File 4,Pass
File 6,Fail… etc.
5. The list, or csv file should be sent to the DPP along with the other required
information, detailed below.
5 OTHER REQUIRED
INFORMATION
Along with the test results please provide the following general information:
OEM name
Product name
Product version
Product Test Plan, or Profile, or Template
Date of tests
Name & contact details of person carrying out tests
Please also confirm that the test results were obtained using the
specified test plan detailed above, and that they were achieved
when using the product in a configuration which would
reasonably be regarded as normal operational use.
Page 2 of 5
AS-11 UK DPP HD - P3 - Analyser Test Files - A3.1 (FILES).docx (5.11.14)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
APPENDIX A – MXF Assessment Criteria
The Product Test Report will be tied to a specific test plan, or profile, or tick box of a specific version of a specific
product that someone buying the product can use operationally
We want the ‘AS-11 UK DPP HD’ Test Plan that is Certified for an Analyser to be operationally useful – so it:


MUST PASS files that the Lab considers “acceptable” (including any files from Certified Writers)
MUST FAIL files with that the Lab considers “faulty” as per the Features and Test Methods described in
PART 1.
Aims / Approach



The focus is on a set of “core MXF tests” (See below)
However, the Test Plan may include additional tests if the OEM wishes (i.e. they can make it “best the
device can do”)
The Lab only considers the overall PASS / FAIL result output by the Analyser for each MXF file tested
The “core MXF tests” are the tests in this document, and are designed with the following properties in mind:





Test the MXF layer only
Quick to run, and practical to implement
A subset of all the MXF properties of the file
Cover important areas of the specification
Used as part of the Certification criteria
The test files



Will not deliberately include faults beyond those listed above
May vary in ways that explore the allowed scope of the specifications
May include faults listed above
The test process
1. The OEM completes the form regarding the product’s coverage of “core MXF tests”
2. The OEM tests the DPP files provided by the Lab
3. The OEM reports the overall result from the Analyser for each DPP file


Ordinarily the Lab will not read the details of the MXF file test reports generated by the Analyser
The Lab will not be concerned with exactly which errors are reported.
4. The lab assesses the testing results
Assessing Results of File Testing
In cases where these core MXF test results from the Analyser do not match those expected by the lab this will need to
be resolved:


If disagreement is due to "core MXF tests" faults found / not found: these will be resolved between the
Lab and the OEM with reference back to the form completed by the OEM in Step 1.
If the Analyser finds errors in "Files from Certified Writers":
o
If disagreement is due to "other" faults found (e.g. AVC faults) the OEM must:
Page 3 of 5
AS-11 UK DPP HD - P3 - Analyser Test Files - A3.1 (FILES).docx (5.11.14)
 Fix the faulty test
 OR: Provide a comprehensive written assessment of the file and their Analyser's behaviour, with
reference to the relevant standards, in order to prove that the behaviour of their Analyser is
correct
 OR: Remove the problematic test from the test plan / "tick box" being considered for
Certification
o
In addition to the above, there may be areas where the Analyser has to relax its checking in order to
cope with “downgraded” Errors (File Writer Conditional Errors etc) – the Lab will provide guidance to
the OEM on this as required for allowed exceptions.
Certification criteria are met if the following is true:


The Analyser MUST include the entire core MXF tests using one of the prescribed methods, or another
method approved by the DPP Compliance Programme. This is the mandated set.
All mandated tests MUST correctly Pass or Fail the test files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
APPENDIX B - Overall Analyser Testing Procedure
The functionality, defined in a specific Test Plan or Test Profile, is tested and assessed in three parts as part of the
DPP Compliance Programme. These include tests that the Analyser should be carrying out in order to meet the
required Certification Level criteria. The three parts are as follows:
PART 1.
MXF Format Tests for File Analysers (Separate doc)
PART 2.
Descriptive Metadata (DM) Validation Tests (Separate doc)
PART 3.
File Testing and Reporting (for PART 1 & PART 2) (This document)
The Analyser test procedure (Parts 1, 2 and 3) may be carried out by the OEM at any time by following the
instructions detailed in section 3. The range of included tests and capabilities is then assessed by the DPP
Compliance programme against the Certification Level criteria. If these criteria are met then the Analyser will Pass.
File Reader functionality should be tested and recorded separately, following the “File Reader Tests for Analysers”
instructions. This may include other capabilities if present, such as player or transcoder. If the device also writes AS11 DPP HD files then you must complete Writer tests. Instructions for these are available elsewhere and not
covered by this document.
If the Analyser device includes writer functions then for overall product testing all three sets of tests (reader, writer
and analyser) must ‘Pass’ or ‘Pass with Conditions’ in order for a device to qualify for Certification.
Page 4 of 5
AS-11 UK DPP HD - P3 - Analyser Test Files - A3.1 (FILES).docx (5.11.14)
OVERALL PROCEDURE
A)
(Testing to Certification)
OEM to follow TESTING INSTRUCTIONS below
PART 1
a. Complete all the required sections in this document: PART 1 (MXF Format Tests for File Analysers).
b.
c.
d.
e.
f.
g.
h.
i.
Complete the DECLARATION at the end of the document and return the form to the DPP Compliance
Programme (CP) to the email below.
The PART 1 document will then be reviewed.
The CP will Pass (or Fail) the submitted paperwork.
If it is a Pass then the next stage is to test that the product correctly tests MXF files as detailed in PART 1.
The OEM will be given a download link for a set of test files for testing the detection of issues for a range
of MXF features.
The PART 3: File Testing and Reporting (for PART 1 & PART 2) document explains how to record test
results for PART 1 (MXF) and/or PART 2 (DM) tests.
The results of the file testing (Pass or Fail for each file) should be returned.
The Results will then be reviewed by the CP to determine if the analyser correctly identified each file as a
PART 2
a. The companion requirements for Descriptive Metadata are detailed PART 2: Descriptive Metadata (DM)
Validation Tests.
b. The document describes the DM validation rules and includes a download link for a set of test files which
are for testing the detection of a range of DM issues.
c. The PART 3: File Testing and Reporting (for PART 1 & PART 2) document explains how to record test
results for PART 1 (MXF) and/or PART 2 (DM) tests.
d. The results of the file testing (Pass or Fail for each file) should be returned.
e. The Results will then be reviewed by the CP to determine if the analyser correctly identified each file as a
Pass or Fail.
B)
OEM to email completed Form and Results to the DPP:
complianceprogramme@digitalproductionpartnership.co.uk
C)
The outcome of the review of results for PART 1 and PART 2 will be provided to the OEM.
D)
The Test Plan or Test Profile is required to Pass both PART 1 and PART 2 tests.
E)
The product also requires “File Reader Tests for Analysers” to have been completed. If the device also includes
Writer functionality then “Writer Tests” should also have been completed. All tests must be a ‘Pass’ or ‘Pass
with Conditions’ in order for the Product Test Report to be issued.
F)
A copy of the Test Plan or Test Profile should be provide and will be included in the Product test Report.
G)
DPP prepare a Product Test Report, and email this to the OEM
OEM applies to AMWA for Certification and sends the Product Test Report. The test results will appear on
the AMWA Certification Authority web page where the Product Test Report will be viewable and may be
downloaded.
Page 5 of 5
AS-11 UK DPP HD - P3 - Analyser Test Files - A3.1 (FILES).docx (5.11.14)