SPI
Software Process & Infrastructure
http://spi.cern.ch
GRIDPP Collaboration Meeting - 3 June 2004
Jakub MOSCICKI
jakub.moscicki@cern.ch
SPI Services Overview
General Services
CVS service
External Software
Web Portal
Collaborative Facilities
Task Management
Mailing Lists
Software Development
Coding
Development
Analysis and Design
Release
Quality Assurance
Testing
Specifications
Deployment and
Installation
Build systems
Documentation
Provide General Services needed by each project
• CVS repository, Web Site, Software Library
• Mailing Lists, Bug Reports, Task Management, Collaborative Facilities
Provide solutions specific to the Software Development phases
• Tools, Templates, Policies, Support, Documentation, Examples
J.Moscicki
SPI - Software Process & Infrastructure
2
SPI Project Guidelines
• Have different and separated services
- Simple solutions, easy to learn, commonly needed services
- Leave any process for later
•
•
•
•
Work with the users
Develop as little as possible
Establish simple deliverables
Everything is done starting from existing infrastructure
-
LCG and LCG projects (Pool, Seal, etc)
LHC experiments
IT division
Big projects (G4, Root, etc)
• We did not start from tools for requirements, design, etc.
• We started from development-related work
- repository, releases, testing, bug report, etc
 The rest of the talk describes SPI services
J.Moscicki
SPI - Software Process & Infrastructure
3
SPI Services (April 2004)
• External Software
• Savannah Project Portal
• Code Documentation
(doxygen, lxr, viewcvs)
• Testing Frameworks
• Development of LCG
policies, templates
• Automatic Nightly Build
(Nicos)
• QA checklists and reports
• Software Distribution
• LCG Software Configuration
J.Moscicki
• CVS server and AFS
management
for LCG App. Area
• Software Librarian, builds
and releases (new, was Scram
support)
• Documentation and LCG
Workbook
SPI - Software Process & Infrastructure
4
SPI Web Site - http://spi.cern.ch
J.Moscicki
SPI - Software Process & Infrastructure
5
SPI External Software Service
•
•
•
We install software needed by
LCG projects.
Open Source and Public Domain
software (libraries and tools)
like:
• Compilers (icc, ecc)
• HEP made packages
• Scientific libraries (GSL)
• General tools (python)
• Test tools (cppunit, qmtest)
• Database software (mysql,
mysql++)
• Documentation generators
(lxr, doxygen)
• XML parsers (XercesC)
There are currently 50 different
packages, plus others under
evaluation. For more than 300
installations
J.Moscicki
•
•
The LCG projects (SEAL, POOL, PI,
Simulation and SPI) propose what to
install in agreement with LHC needs
The platforms, are decided by the
Architect Forum
• Linux RedHat 7.3 and compilers
- gcc 3.2 and 3.2.3
(rh73_gcc32)
- icc 7.1 (rh73_icc71)
- ecc 7.1 (rh73_ecc71)
• Windows
- Visual Studio .NET 7.1:
(win32_vc7).
• Mac OSX (osx103_gcc33)
• Platforms always been reviewed
• We also provide configuration
for the LCG projects
• A unique AFS location
• Standard structure
package_name/version/
platform/package_ content
SPI - Software Process & Infrastructure
6
External Software – http://spi.cern.ch/extsoft
J.Moscicki
SPI - Software Process & Infrastructure
7
SPI Savannah Portal Service
•
•
•
•
•
•
•
Functionality:
• Bug tracking
• Task management
• Mailing lists, news, faqs
• Access to CVS repository
• Download area, etc
The Web portal for LCG
software projects
Customized from GNU
(SourceForge as origin)
Totally web based
Single entry point to all projects
Uniform access to project
information
Set up common web
infrastructure for a project
without coding
J.Moscicki
• What SPI changed
• installation from GNU, general
bug fixing and improvements
• integration with AFS
authentication
• Integration with standard services
already available
• What SPI does
• administration (project approval)
• maintenance (submitted bugs)
• development (support requests)
• Status
• >80 hosted projects
• >550 registered users
SPI - Software Process & Infrastructure
8
Savannah Service – http://savannah.cern.ch
J.Moscicki
SPI - Software Process & Infrastructure
9
SPI Testing Services
• Software testing should be an integral part of the software
development in the LCG App Area
• The goal was to provide something that can be run automatically
as often as needed (releases, development, etc)
Automated testing
SPI provides
• Test frameworks
• CppUnit, Oval
• Qmtest
• Test support
• Test policies
• Test doc
Software testing
Acceptance test
LHC
experiments
System test
Sw-testing
team
Integration test
Unit test
• Different platforms/compilers
J.Moscicki
SPI - Software Process & Infrastructure
Software
developer
Use in Exp.
Examples
System
Tests
Integration
Tests
Work Package
Test
10
Testing Support – http://spi.cern.ch/testing
J.Moscicki
SPI - Software Process & Infrastructure
11
Quality Assurance Service - http://spi.cern.ch/qa
• The main goal of QA activity help LCG
projects
•
•
•
•
assess and improve the quality of the software
provide tools to collect useful metrics/statistics
which help to
asses quality;
generate reports;
verify if project setup is correct with LCG
policies.
• QA Tools and Focus
•
•
Automatic reports
•
Development/integration of automatic tools
LCG Policies
•
•
agreed and defined by AF
SPI supports them in the tools and
procedures and only helps to work them out
J.Moscicki
SPI - Software Process & Infrastructure
12
Quality Assurance Activities
QA Checklist on each Release
•
•
•
Build the release
Run automatic tests
Statistics
•
•
•
•
•
•
•
•
Test Inventory
Documentation/Examples Inventory
Savannah Statistics
Code Inventory
Rule Checker , Logiscope
LCG Policies
•
•
Configuration of a build system
CVS directory structure
Well-defined, transparent, open
•
•
•
•
QA Reports
•
Automatic reports
•
Generated at every release
•
Published on the SPI web site
Evaluation and usage of external
tools
•
Rule Checker
•
Logiscope, Test coverage
•
SLOC, Valgrind, ignominy
clear rules and checklist of assessed
items
anybody at anytime may see statistics
create reports themselves
anybody may contribute
J.Moscicki
SPI - Software Process & Infrastructure
13
QA Reports
J.Moscicki
SPI - Software Process & Infrastructure
14
LCG Policies
•
LCG Policies
•
•
•
•
•
•
They are a needed by the LCG
•
•
They are defined by the LCG
projects, collected by SPI
If everything is different is too
difficult to use and to automate
•
•
•
CVS and Build Directory Policy
Software Testing Policies
Version Numbers, Tagging and
Release Procedure
Installation Directory Structure
Platform string, binary names,
debug flags and more
compromising on our habits, for
project needs
tell when they are not followed
First time that this exists at this
extent, and that is checked
J.Moscicki
SPI - Software Process & Infrastructure
15
SPI Software Distribution Service
• Simple solution to use
• local installations (external sites,
laptops,...)
• using simplest approach
• python downloader + tar format
• replicate the central AFS tree (in
a optimized way)
• package dependency from
SCRAM
• ...until a complete, long-term
solution available
• Looking into pacman as a
suitable solution
• SPI will adopt what LCG Grid
Deployment decides to
provide
J.Moscicki
• Simple tool to install
• successful for users:
- POOL @ Karlsruhe
- BNL nightly builds, CMS
- developers at home, etc
• very easy to use and
reliable
• Different use-cases should
have different solutions
• Our tool is adequate as a
temporary solution for LCG
Application Area Distribution
• but long-term solutions must be
investigated:
- pacman, LCFGng ....
• GRID WN installations should
be supported differently
SPI - Software Process & Infrastructure
16
Software Distribution – http://spi.cern.ch/lcgsoft
J.Moscicki
SPI - Software Process & Infrastructure
17
Summary
• The set of services is working and fully available
•
•
•
•
•
•
Savannah Project Portal
Software Testing
External Software Service
Quality Assurance and Policies
Software Distribution
…and many more
• We have followed the strategy defined
•
•
•
•
Work with the users
Ask their help
Develop as little as possible in order to have little maintenance
Provide simple and modular solutions
• We have commitment to provide a sustainable
service
• Most people moved on to new projects, as it was planned
• The services are used by LCG projects, and many outside LCG
• Unlike in the past, we try to match the environment
and habits of the way people work in HEP
J.Moscicki
SPI - Software Process & Infrastructure
18