Euclid
Consortium
OU /SDC
Proposed
Development Strategy
Maurice Poncet (CNES, SDC-FR Lead)
Andrea Zacchei (Inaf – SDC-IT Lead)
Keith Noddle (IfA – SDC-GB Lead)
OU-LE3 KOM Meeting, London
March 22-23, 2012
1
Euclid – Ground Segment Concepts
Euclid
Consortium
The recurring acronyms of the ground segment:
OU-LE3 KOM Meeting, London
March 22-23, 2012
2
Euclid – Ground Segment Concepts
Euclid
Consortium
OUs (Organization Unit) :
OU
Responsible for the definition and prototyping of a given
pipeline (i.e transform scientific req. from SWG into
algorithms or mathematical methods)
Responsible for the validation of a given pipeline (i.e
providing baseline test data)
SDC DEV
SDCs (Science Data Centers) :
Responsible for the S/W development of pipelines : SDC Dev
Responsible for the H/W processing infrastructure : SDC Prod
SDC PROD
Responsible for the pipeline processing operations : SDC Prod
OU-LE3 KOM Meeting, London
March 22-23, 2012
3
SDCs - Development Strategy
•
Euclid
Consortium
SDC-xx DEV Team is a “delocalized” team
–
Distributed over several labs
–
Integration of other countries contributions
•
This team is lead by only one responsible which is at
SDC-xx Dev level
•
SDC-xx DEV Team works through a collaborative web
based development infrastructure
•
A given SDC-xx DEV Team may contribute to SDC of
other countries (under their responsibility)
•
There is one dev team per pipeline
•
SOC should be involved as any other SDC
OU-LE3 KOM Meeting, London
March 22-23, 2012
4
SDCs – Implementation Involvement
Euclid
Consortium
Implementation
LE1
SDC-CH
SDC-DE
SDC-ES
SDC-FR
SDC-GB
SDC-IT
SDC-NL
SDC-FI
SOC
VIS
NIR
SIR
EXT
SIM
L
C
C
L
C
C
L
C
c
MER
SPE
LE3
L
L
C
C
L
L
C
L
L
L
C
SHE
C
C
PHZ
L
c
C
L
L
c
Coord
• L : Lead
• C : Contribution (as corresp. OU Co lead)
• c : other contribution
OU-LE3 KOM Meeting, London
March 22-23, 2012
5
SDCs – SDC Generic WPs
Euclid
Consortium
WP-4-4-nn
SDC-<NAT>
WP-4-4-nn-1000
SDC-<NAT> Management
WP-4-4-nn-2000
SDC-<NAT>-DEV
WP-4-4-nn-3000
SDC-<NAT>-PROD
WP-4-4-nn-2100
SDC-<NAT>-DEV Infra.
WP-4-4-nn-3100
SDC-<NAT>-PROD Infra.
WP-4-4-nn-2200
SDC-<NAT>-DEV Implementation
WP-4-4-nn-3200
SDC-<NAT>-PROD Operation
WP-4-4-nn-22sb
<OU> pipeline Implementation
WP-4-4-nn-32sb
<OU> pipeline operation
Feel free to propose any other set up.
WP-4-4-nn-22sb-st
<OU> pipeline Implementation subtask
WP-4-4-nn-32sb-st
<OU> pipeline operation subtask
WP-4-4-xx-2300
SDC-<NAT>-DEV Integration
WP-4-4-xx-23sb
<OU> pipeline Integration
WP-4-4-nn-23sb-st
<OU> pipeline Integration subtask
OU-LE3 KOM Meeting, London
March 22-23, 2012
6
SDC-DEV – Dev Environment
Euclid
Consortium
SDC
SDC-DEV
Local Infra
Local Infra
Local
LocalInfra
Infra
Dev
Dev
Compilation
Dev
Dev
Compilation
Unit
Test
Compilation
Compilation
Unit
Test
Unit
UnitTest
Tests
S/W int tests
SDC-PROD
e – Infra
Integration
Validation
Deployment
Generation
Int/Val tests
Perf. Test
Quality Check
Dash Board
Conf Mangt
Extraction
Generation
(Preint) tests
Quality Check
Dash Board
Production
Data Ingestion
Processing
Data
Distribution
Labs
Located
Computing
Center(s)
EC level S/W
conf mangt)
OU-LE3 KOM Meeting, London
EMA
March 22-23, 2012
7
SDCs - Development Environment & Context
Euclid
Consortium
• Tailored (i.e. simplified & adapted) ECSS process should be applied
• Linux platform
• Model based Approach : SysML
• Scripting language : Python
• Coding language : C/C++
• Distributed Development (cf. open source best practices)
• Test driven approach
• Iterative & Incremental process (Agile Approach)
• Continuous S/W Integration
• Strong configuration management
• Automation (extraction, build, test, code quality check, …)
• Dashboards (motivating !)
• Online tools
• Collaborative documentation
OU-LE3 KOM Meeting, London
March 22-23, 2012
8
OU / SDC-DEV – Agile Approach (1/2)
Euclid
Consortium
• Iterative & Incremental development
– Sprints (e.g. monthly or less) : sub functions
– Release (e.g. half year or less)
=> S/W available in early phases, ability to adapt the
S/W
OU-LE3 KOM Meeting, London
March 22-23, 2012
9
OU / SDC-DEV – Agile Approach (2/2)
Euclid
Consortium
• OU is “Product Owner”:
– Backlog management : Prioritized list (e.g. priority, maturity, …)
of pipeline “features”,
– Definition of releases and sprints contents in coordination with
SDC-DEV team (e.g. according to SDC-DEV capability),
– Validation of pipeline with SDC-DEV team.
• SDC-DEV team responsible of sprint & release dev.
OU
OU-LE3 KOM Meeting, London
SDC DEV
March 22-23, 2012
10
Continuous Integration Principles
Euclid
Consortium
* Maintain a code repository
* Automate the build
* Make the build self-testing
* Everyone commits to the baseline every day
* Every commit (to baseline) should be built
* Keep the build fast
* Test in a clone of the production environment
* Make it easy to get the latest deliverables
* Everyone can see the results of the latest build
* Automate deployment
OU-LE3 KOM Meeting, London
March 22-23, 2012
11
SDCs - Development Phase
Euclid
Consortium
• Pipeline Development in respect with SGS and
System rules
– ECSS tailoring (documentation, dev cycle)
– UML modeling (common look & feel)
– Pipeline I/F
– IAL principles
– Products, Data modeling with modeling tool
– Quality rules (check should be as much as possible
integrated and automated inside the collaborative
platform)
• Potential common S/W modules should be
submitted at system level
OU-LE3 KOM Meeting, London
March 22-23, 2012
12
SDCs – Integration Phase
Euclid
Consortium
• “target” pre-integration may be done at SDC-DEV side
on virtual machines being representative of target
configuration.
• Target Integration should be done on SDC-Prod infra. in
a dedicated Integration « sand box »
• Integration should be achieved in close cooperation of
SDC-DEV, SDC-PROD and OU teams
• “Technical” tests designed at SDC level
• “Functional” tests designed at OU level
• Integration tests output data should be accessible to
above teams for validation and further investigation
OU-LE3 KOM Meeting, London
March 22-23, 2012
13
SDCs – Validation Phase
Euclid
Consortium
• Validation should be done on SDC-Prod infra. in
a dedicated Validation « sand box »
• Integration should be achieved in close
cooperation of SDC-DEV, SDC-PROD, OU teams,
and as needed SWG.
• “Scientific” tests designed at OU/SWG level
• Validation tests output data should be
accessible to corresponding OU team and SWG
for validation and in case of failure for further
investigation.
OU-LE3 KOM Meeting, London
March 22-23, 2012
14
Euclid
SDCs – Collaborative Development Platform
Consortium
• A Collaborative Development Platform should be set up:
– Aims : verify tools readiness, simulate dev cycle, may be used for SGS
mock-up.
– O/S
: Linux
– Languages
: Python, C++
– Configuration
: SVN
– Generation
: Maven
– Test s
: xUnit
– Integration
: Jenkins
– Quality
: Sonar (with C++ plug-in)
– IDE
: Eclipse, Netbeans, vi, whatever…
– Tracking
: Jira, Redmine
– Doc Gen.
: Doxygen
– Modeling
: Sparx Enterprise Architect
– Deployment
: IzPack
OU-LE3 KOM Meeting, London
March 22-23, 2012
15
Euclid
SDCs - Collaborative Devlopment Platform
Consortium
OU-LE3 KOM Meeting, London
March 22-23, 2012
16
Tools and …. Rules
Euclid
Consortium
But, tools are not sufficient, needs for :
• Rules (e.g. naming conventions, formats,
directory trees, …)
• Guidelines
• Code, Doc templates
• Good practices
• User support
At SGS level
OU-LE3 KOM Meeting, London
March 22-23, 2012
17
SDCs – Typical Product Tree
Euclid
Consortium
SDC-xx
SDC-xx-DEV
SDC-xx-PROD
xxx pipeline
Infrastructure
yyy pipeline (contribution)
Integration Infrastructure
Development Infrastructure
Validation Infrastructure
TL/IAL Plug-ins
Production Infrastructure
TL/IAL Connectors
OU-LE3 KOM Meeting, London
March 22-23, 2012
18
SDCs – Development Inputs
•
Euclid
Consortium
OU-xx outputs :
– Pipeline workflow, breakdown, inputs & outputs and algorithms specifications,
Use cases,…
– Pipeline V&V kit (test data, expected results, documentation)
– Optionally, prototype code : should be reengineered in order to fit SGS rule
•
Formal data modeling of all xx pipeline input (primary and secondary),
output and intermediate data.
•
Code and Data from OU-SIM
•
SGS Level Documentation impacting the software development (RB, TS,
DDF, DJF, MGT, PAF) to which the xx pipeline implementation should be
compliant and/or should derive its own documentation.:
– Architecture
– Development principles
– Common Rules & Tools
– PA/QA
– …
•
Architecture constraints from production infrastructure
OU-LE3 KOM Meeting, London
March 22-23, 2012
19
Euclid
SDCs – Development Deliverables outputs
Consortium
• Pipeline product including Software executables, documented
source code, under strong configuration control
• pipeline documentation (RB, TS, DDF, DJF, MGT, PAF), in
particular :
– RB : Software system specifications (SSS), Interface requirements
document (IRD)
– TS : Software Requirements Specifications (SRS), Interface Control
Document (ICD)
– DDF : Software Design Document (SDD), Software Configuration File
(SCF), Software Release Document (SRelD), Software User Manual
(SUM) including in particular extraction, deployment, installation and
generation procedures.
– DJF : Software V&V Plan (SVVP), Software Test Specifications (STS),
Software Test Report (STR), Software problem reports and nonconformance reports (SPR & NCR)
– MGT : Software Development Plan (SDP)
– PAF : Software Product Assurance Plan (SPAP), traceability & coverage
matrixes (TCM).
OU-LE3 KOM Meeting, London
March 22-23, 2012
20
Euclid
SDCs - Integration Deliverables OutputsConsortium
• Operational VIS pipeline on SDC-FR PROD
infra.
• Verification & Validation documentation :
– Verification & Validation Plan (SValP)
– Verification & Validation Tests Specification (SValS)
– Verification & Validation Tests report (SValR)
• Contribution with OU & PO/System team to
system 2by2 tests report
• Contribution to system E2E tests report
OU-LE3 KOM Meeting, London
March 22-23, 2012
21
SDCs - Operation Deliverables Outputs
Euclid
Consortium
• Data products from pipeline
• Production reports : global metrics ( problems
met during exploitation, performance :
processing, storage,…)
OU-LE3 KOM Meeting, London
March 22-23, 2012
22
SDCs – Versions & Overall Planning
Euclid
Consortium
SGS Global schedule
Phase F
Disposal
Proposed Schedule
OU-LE3 KOM Meeting, London
March 22-23, 2012
23