UNITED NATIONS ECONOMIC COMMISSION FOR EUROPE CONFERENCE OF EUROPEAN STATISTICIANS Working paper 2013/19 7 June 2013 High-level group for the modernisation of statistical production and services (HLG) Meeting, Geneva, 14 June 2013 Software Collaboration and Sharing at Statistics Canada Executive Summary In keeping with its international strategy, IT strategy and Corporate Business Architecture principles, Statistics Canada would like to further collaborate with its NSO partners on sharing and developing software. This paper describes Statistics Canada’s experience and practices, highlighting key success factors and challenges. The key models for collaboration are: Model Scope of collaboration Share software code Share software executable Collaborative development The organization that owns the software provides a ‘licence to inspect and modify the code’, and possibly to distribute the modified code. The organization that developed the software provides a ‘licence to use’, directly or via a reseller. Typically, maintenance and support is also provided. Organizations develop the software together. Partners can contribute in kind or financially, and third parties may also contribute, such as contractors or the Internet community. An appropriate open source software licence would best meet the needs for this model. While there is no formal guidance to the Government of Canada departments on software collaboration, Statistics Canada can provide a licence to others to use, modify and distribute software where it is owner of the copyright for all components. release its software under an Open Source licence of its choice, if this will meet its needs and with an understanding of the risks involved. participate in existing Open Source initiatives and hire contractors to contribute to Open Source software. enter into other forms of collaborative development arrangements, with some limitations. StatCan should select with its partner NSOs an appropriate first collaboration initiative of mutual value. To maximize success, the recommended approach is to adopt best practices, processes, culture, tools and methodologies from the Open Source, to adopt agile and lean methodologies, and to use a suitable recognized Open Source licence. Some key success factors for collaborative software development are clearly articulated purpose, outcomes and roles and responsibilities strong project management, with a focus on participant co-operation, communication and empowerment commitment to compromise to achieve desired common outcomes, supported by a solid planning and change management framework to control diversity, complexity and scope strong management support engaging all participants who have the right mindset, with mechanisms in place for continuous interaction a core of participants with experience with open source software development or co-development. Key challenges are adjusting to the culture of the open source model fear of loss of control compliance with legislated standards and directives of each country. Software collaboration at StatCan Page 1 of 23 StatCan’s significant experience building generalized systems and adoption of service oriented architecture and shared models such as GSIM and GSBPM makes it now possible to pursue a software collaboration initiative with partner NSOs. In addition, StatCan is looking into making some of its existing generalized systems available to other NSOs as Open Source, provided it can ensure that the software can continue to meet its own requirements. 1. Introduction This document presents the Statistics Canada (StatCan) and Government of Canada (GC) context for collaborating on software development and sharing software with other organizations such as other national statistical offices (NSOs). It also presents some case studies, with associated lessons learned, and proposes a set of requirements for successful collaboration, including licensing considerations. StatCan context — collaboration as an enabler to CBA In 2009, StatCan embarked on a redesign of its corporate business architecture (CBA). This initiative has three key objectives: enhance the quality of its products and services improve responsiveness in delivery of new statistical programs deliver operational efficiencies. To achieve these objectives, StatCan is streamlining core business processes and implementing a rationalized set of robust and flexible systems aligned with its business architecture. Key enablers include a suite of generalized systems that automate statistical methods, standard collection architecture, tools and processes, and shared IT platforms and services. The GC is also reviewing its business architecture for the delivery of common services including finance, HR, and IT. Its Chief Information Officer is directing all departments to rationalize, standardize, reuse, cluster and collaborate. StatCan, in keeping with its international strategy, IT strategy and CBA principles, would like to further collaborate with its NSO partners, to initiate new joint software development initiatives, enabled by the Generic Statistical Business Process Model (GSBPM), the Generic Statistical Information Model (GSIM) and the Common Statistical Production Architecture (CSPA), colloquially known as the “Plug and Play initiative” leverage the efficiencies and potential for innovation of the open source collaborative process itself improve how it shares software with partners in the GC and counterpart NSOs. Why should we do this? The benefits that StatCan seeks through this increased collaboration are improved quality of its software, which in turn will improve the quality of its programs software that is more agile to respond to changes in business requirements reduced total cost of ownership of software, where common processes and methods are adopted improved alignment to the GSBPM and the GSIM, which in turn will enable further collaboration improved interaction with stakeholders by using common models and standards as well as greater interoperability opportunities for staff to learn new statistical and IT methodologies, approaches and skills. 2. Legal Framework in the GC Before presenting how software collaboration is affected by the GC legal framework, it is important to provide some key definitions. Software collaboration at StatCan Page 2 of 23 What is software? Software essentially comprises “computer programs,” which are considered literary work and therefore subject to Canada’s Copyright Act. Software also comprises elements that are also subject to the Copyright Act, such as executables, or object code source code — for the core software and any supporting tools documentation data that might drive functionality (e.g., configuration or code tables). The owner of a copyright title and moral right title to software provides others specified rights to it via a licence, which can provide the licensee the rights to do one or more of the following: install and use distribute instances of the original software inspect and learn from the source code change the source code, install and use it distribute the modified software. The licence also provides any other rights, restrictions, conditions or obligations required by the owner of the software. It typically includes terms regarding intellectual property (IP), liability, indemnification, confidentiality, termination and appropriate law, among others. A key consideration is that determining entitlement for software requires knowing the source of all its elements. If GC employees produced all the elements, then the GC owns all rights to the software; if open source or propriety software components are included, or if contractors developed components, then determining entitlement becomes more complex. For federal government organizations, the Financial Administration Act makes software a capital asset that can be capitalized, expensed, and declared surplus. Federal managers therefore track the effort and costs associated with designing, developing, implementing, enhancing and supporting in-house software, as these costs are capitalized or expensed in departmental financial statements. What is FOSS1? ‘Free and open source software’ (FOSS), or simply ‘open source software’, refers to computer software applications for which the authors or copyright holders release the source code under a licence. That licence authorizes anyone to use the software, modify it and redistribute it. While FOSS terms and obligations vary greatly, several categories of clauses are commonly used: general disclaimers of warranty and/or liability notice obligations, which generally require the licencee to notify downstream users of the particular FOSS licence that applies to the work source code obligations, which generally require the licencee to provide the source code of the software when they distribute it hereditary licensing obligations, which generally require the licencee to only release any modifications or improvements as FOSS under the same licence. The first two — disclaimers and notice obligations — are present in nearly every FOSS licence. FOSS licences are usually classified as hereditary, where the licence includes an obligation to ‘inherit’ the licence for modifications made to the software permissive — where the licence includes no such obligations. This categorization is very simplified: hereditary and permissive licences include widely varied obligations and circumstances where they take effect. For more details, please refer to Annex A. 1 Source: Free and Open Source Software Licensing Primer for Natural Resources Canada – Hickling, Arthurs and Low (2012) Software collaboration at StatCan Page 3 of 23 Guidance to Canadian federal departments and agencies There is presently no up-to-date formal guidance to Canadian departments and agencies on software collaboration and participation to FOSS development. Instead, departments and agencies are asked, before embarking on such initiatives, to consult their departmental legal advisors: each situation is unique and can present different risks depending on the participants, the nature of the software, existing licensing applicable, liability implications and other considerations. That said, we can confirm the following advice. Departments and agencies are encouraged to collaborate on software development and to share software across all levels of the Canadian public sector. The Financial Administration Act must be given due consideration when determining capitalization of software assets and coding employee time and other monies spent on FOSS or other collaborative software development, enhancement and support. Compliance with the Contracting Policy is essential: co-development arrangements cannot be used or be even perceived as an alternative to procurement of IT services. StatCan is subject to the Communications Policy of the Government of Canada and must manage the administration and licensing of Crown copyright in co-ordination with the Department of Public Works and Government Services. Within StatCan, Information Management Division is responsible for all licensing except software licensing, which is the responsibility of Informatics Branch. Consult Legal Services before finalizing any co-development agreement with an organization outside the GC, or before formally selecting a licence for the resulting software. Consult Legal Services before issuing any new licences for software StatCan owns. Ensure that any existing licensing rights, conditions and obligations of components or supporting tools are respected by employees, contractors and partners using their software. Ensure that IP rights of all other parties are respected, including those of contractors who have contributed code and may have retained IP rights as per clauses in their contracts. The resulting software is still subject to all GC policies, directives and standards such as Common Look and Feel (CLF) for Web applications and bilingualism. Exemptions can be sought when necessary for the success of the initiative, but should be approved before proceeding. The table below provides some examples of what we can and cannot do in the GC: Action Can we? Notes Give away IP to another organization to our existing code Provide a licence to our software or contribute our existing software to a software collaboration initiative or open source initiative No, even if we have all rights for all its components. It depends. The type of licence we can offer depends on the rights we have to all the components of this software. Automated tools are available to support this analysis. No, but we may still have a licence that allows us to modify the code, use it, and perhaps distribute it, depending on the licence granted to us. Yes, GC employees can, and do, contribute to FOSS development projects, as long as they have been approved by their senior managers and are of benefit to the GC. Departments can also hire consultants to enhance FOSS that is used to support their programs, and to provide maintenance and support services. Yes, but subject to proper procurement rules (granting authority) if the organization is outside the GC — essentially the same rules as for procuring a licence. Yes, but an MOU should be put in place to ensure clear understanding of all parties, and licence agreement should also be negotiated beforehand if outside of GC. Yes, referred to as ‘forking’: but all restrictions specified in the open licence Instead, we can provide a licence with the appropriate terms and conditions. The decision must be fully informed and must consider all implications and risks. StatCan already provides various types of software licences to NSOs and others. Take over IP to code we don’t own after we have modified it Start a new open source software development project, or contribute to an existing project Contribute financially to another organization’s software development in return for a licence Co-develop software with other organizations (with or without a FOSS licence) Continue on our own with software developed initially as open source Software collaboration at StatCan In the end, what matters is that we can do what we need to. We should ensure that our needs can be met before we embark on a co-development agreement. If we start a new project, it gives us the flexibility to choose the appropriate licence model, unless reusing existing components with more restrictive licences or preexisting IP. Resources spent on FOSS development should be coded as per Financial Administration Act rules. An MOU or agreement should be put in place if contributing before development is completed, and include the licence to be issued. If development is completed, only the licence is required. There are many forms of collaboration: contributions can be ‘in kind’ or financial, a third party can be contracted to do the work, for example. Forking should be a last resort option to maximize benefits of collaboration. Page 4 of 23 Action Can we? Enter into a co-development arrangement to enhance our software need to be respected. Yes, but must clearly not even appear to bypass procurement policies. Notes Licence agreement should be negotiated beforehand. 3. Collaboration Models Collaboration on software can range from sharing best practices, models and/or methodologies to working together from initiation to implementation and even sharing infrastructure as well as support and maintenance services. All collaboration models can achieve benefits, though at different levels and with different advantages and disadvantages. It is also possible to go back and forth between models at different stages, with some limitations. Annex B presents four StatCan examples of software collaboration using different models. Sharing model — “To allow someone to use or enjoy something that one possesses” In this model, partners share their software under different terms, which may or may not allow the recipients to ‘make it their own’. This model offers great opportunities for cost-saving, and can bring us closer to the true collaboration model — the more we already share, the easier collaboration will be. Sharing is the model most used by NSOs and is further enhanced by a shared inventory of software and the creation of user communities. The table below presents some shared and associated considerations: Deliverables All non-code deliverables Code (executable) and documentation — production version Source code and documentation Implementation, maintenance and support services Infrastructure Software as a service (Saas) Considerations The Copyright Act is still applicable for user requirements, specifications, design documents, models and standards, etc. In practice however, these are shared freely with the understanding that they remain the IP of the author, and that citations will be included where distributed to third parties. Code is shared by its owner by issuing a licence to the software to those who wish to use it. This licence includes the rights to install and execute the software, but may or may not include the right to view the code, modify it and distribute it, depending on the circumstances. For most applications built by StatCan and shared with other NSOs, the licence does not provide access to the code. The licence issued may be free or for a fee: the code may be provided directly by the organization that developed it, or may be issued and licenced via a reseller. The organization acquiring the software will of course need to separately acquire any required database management system (DBMS) and other commercial off-the-shelf (COTS) software required to run the code, such as SAS. However, the software may also be provided in the form of ‘software as a service’ (SaaS), hosted on infrastructure provided by the organization that developed it, or by a third party. The organisation acquiring the licence under this model would conduct a fit-gap assessment of the functionality offered by the software against its own requirements, just as it would for any COTS. A key consideration when acquiring the software under this licence model is whether the provider organization will consider its requirements when maintaining and enhancing the software, especially if it plans to invest significantly in the software implementation. As well, the provider organization must also be very careful in committing to integrating client organizations`future requirements or ensuring backward compatibility. ‘Best effort’ is generally the principle applied. As for the above, code is shared by its owner by issuing a licence (possibly FOSS), but in this case it provides the rights to modify the source code. The acquiring organization may then create its own version of the software, and can contribute back their updates (which may be required by the licence issued). What started as code sharing can then later turn into a collaboration initiative. Sharing in this context means that one organization provides the services to another. Given the costs incurred by the provider organization, financial arrangements are required. This service can also be provided by one organization (or a third party) after an application is developed collaboratively and can even be provided when an organization has modified the source code to meets its unique requirements. For generalized systems, StatCan provides maintenance and support services to NSOs who acquire a production version and pay the annual maintenance fee. Sharing in this context is one organization providing the infrastructure (development, testing and/or production) to another. This could include the required platforms in a ‘platform as a service’ model, such as application integration software or DBMS. However, given the current StatCan network topology context, and the legal implications of having SSI data residing on another country’s infrastructure, this would be extremely challenging for StatCan. Sharing in this context is one organization hosting the entire application to another — software as a service. The same considerations and challenges as for sharing infrastructure apply to SaaS. Software collaboration at StatCan Page 5 of 23 Collaborative development model — “Working together to achieve a goal” While collaboration potentially offers more benefits than sharing, it also offers more challenges. The resulting application needs to meet the needs of all participating organizations, including compliance with their respective standards and the ability to run on all partners’ platforms (unless using Saas): the application must also be robust, flexible and high-performance. The table below provides a simplified overview of which deliverables can be developed collaboratively, and the associated attributes. More details on what is needed for successful collaboration are covered in Section 4. Deliverables Scope of collaboration Attributes Planning artefacts, methodologies and practices User requirements and specifications Develop together project management plans, governance structures, methodologies and other documents. Develop user requirements and specifications that meet the needs of all partners — with plans to develop jointly — or have one partner develop and later share code or executable under a licensing model, or each partner still develops their own. Collaboration could be partial — initial requirements and/or specifications done together and then bifurcate for the following phases. Similar to above, but could be done with or without collaborating on the code. User requirements need not be identical. Joint development of models and standards can be done independently of collaboration on any other deliverable. Unless subsequent deliverables are also developed collaboratively, value-added may be limited. Collaborating on user requirements and specifications would be of limited value if the resulting application is not developed collaboratively or developed by a partner and shared by others. Agreeing to a set of prioritized requirements for the shared software that meet the needs of multiple NSOs would be challenging but necessary if code would be jointly developed. Would be of value independent of other deliverables being developed collaboratively. Architecture and design Models and standards Code and documentation Support services Maintenance services Infrastructure Software as a service (Saas) The partners would use a ‘divide and conquer’ approach — break down the software into smaller parts or modules and give them to different developers or teams of developers to work on separately. A development platform and agreedupon licence would be required. System and user documentation for the shared/core software should also be written together and then translated as needed by each organization. Partners can develop a fully collaborative service model, in which support services are provided by all partners. This can be done even if previous deliverables were not developed collaboratively (share model). However, the fact that all previous deliverables were developed collaboratively does not automatically require that support services be provided collaboratively. Unless a partner has decided to go their own way, they would continue to collaborate for the ongoing maintenance of the software. As for support services, they may also wish to collaborate on Maintenance even if the software was not developed collaboratively (share model) Partners collaboratively provide and support a shared infrastructure, or have it provided by a third-party. They can also only collaborate on the Development and/or Testing infrastructure. Partners collaborate to provide the hosted application as a service, on a shared infrastructure. Any required software licenses (for example SAS and DBMS) would need to permit such an arrangement. There are many benefits from collaborating on shared models and standards, even if everything else is done individually. GSBPM and GSIM are examples of successful collaboration on models. All above deliverables need to have been developed collaboratively for the shared functionality/core software. Some organizationspecific components and modifications to existing applications would also likely need to be developed by each partner. Each would also need to configure the core software to meet its needs, including any reference tables. Benefits include high code quality and reduced costs. Unless the application is hosted and provided as SaaS, each partner may want to provide its own first-level support services or use a third-party provider due to language and time zone barriers. However a best practice would be for secondlevel support to be shared. As for all previous phases, collaboration can take many forms, including financial contribution. Collaboration should continue for Maintenance to reduce costs. The same considerations apply here as for the ‘share infrastructure’ model. This may use an “infrastructure as a Service (IaaS) and/or “Platform as a Service (Paas) model. Same considerations apply here as for the share infrastructure model. The above table offers very simplified options, and many other variants are possible. Also, partners can use a divide and conquer approach for any deliverable above, though it is most useful for the construction phase. Of course, solid project management and governance will be essential to ensure cohesion and efficient delivery, and that each organization is realizing the desired outcomes of the project. Software collaboration at StatCan Page 6 of 23 4. What is needed to collaborate on software development? During the planning phase 2 Negotiate a written agreement. At minimum, this agreement, or MOU, should identify the vision and objectives for the initiative, the benefits sought and roles and responsibilities of all parties, the high-level project plan and associated deliverables, IP rights assignment for the resulting software, financial arrangements and contributions in kind, termination and notices clauses, liability, and other pertinent information. It should specify the licences to be used for the resulting software. Identify any existing code that will need to be used for the project including any FOSS, ‘shared source’2 or ‘closed source’: this will inform the decision on the software licence. Select a software licence. While an MOU is sufficient to collaborate with other federal departments, as the GC is considered one legal entity, collaboration with other organizations requires a software licence for the resulting software. A suggested approach is to select an existing commonly used open source licence: however, it is not recommended that StatCan select one licence model and impose it on all collaborative projects. Instead, StatCan and its partners should select the most suitable licence model for each project, considering potential security risks, IP protection, all partners’ constraints (including their respective legal frameworks), and the outcomes sought. Some considerations: o There will not always be a choice as to which licence to apply. Where a hereditary licence obligation is already in force, the code must remain under the same licence. o When distributing a project comprising entirely one’s own code, and/or code that does not engage hereditary obligations, any FOSS licence will do. It is also possible to dual-licence code, for example to participate in two different communities that use different licensing norms. o Overall, licensing decisions usually involve one consequential decision: whether to apply a hereditary or permissive licence. Permissive licences maximize the scope of downstream users, with broad appeal to the entire private sector; hereditary licences are appropriate in cases where it is important to receive back downstream changes, or where it is important to ensure that work built on an initial investment remains open and free. o In some circumstances, an open source licence may not be the right choice: instead, a customized licence may need to be developed in order to meet the needs of all parties. Agree on the set of prioritized user requirements that the new shared ‘core’ application must meet, and on what will be met by new country-specific components or existing country-specific applications. Coming to a consensus on user requirements, and associated methodologies, will be challenging but essential. We face this challenge continually in developing generalized systems that must meet the needs of disparate clients within StatCan. Every effort must be made to streamline and prioritize the collective set of requirements, yielding a system that is not unduly complex: needless complexity could compromise the performance and maintainability of the final system. Of course, multilingual support and CLF compliance for web applications are two key StatCan requirements; each country will have their own set of mandatory requirements to address. The key will be to find ways to address them while keeping the shared components as simple as possible. Standardization and alignment of statistical methods and processes across survey programs within StatCan has proven effective when a solid planning and change management framework is in place. Solid executive support is also key for consolidating and streamlining requirements. Agree on IT methodology, tools, platforms, models and standards. Participating organizations must select together a development methodology (Agile and Lean are most appropriate) and a set of development tools (including any database and application integration) to be used. The participants must also agree on a set of coding and documentation standards and code review processes. If the tools to be used involve COTS to run the resulting application (for example SAS), then all current and future participants will have to acquire this COTS and possibly harmonize future upgrades, assuming that the resulting application will be deployed independently in each organization (i.e., not a SaaS). Participants must also agree on the models and standards to be used. Select, acquire and install an application lifecycle management tool or ‘social coding platform’. At minimum, tools are needed to support versioning and issues tracking. Better yet is a complete platform For example, Microsoft Shared Source CLI, C#, and Jscript License, which are not FOSS Software collaboration at StatCan Page 7 of 23 offering all of these elements for collaboration in a single, integrated hub (code and document repositories, wikis, tracking tools, personal profiles, RSS feeds and discussion threads). These capabilities lower the barriers of collaboration, and help contributors write better code faster. Examples include GitHub, SourceForge,and TeamForge. Agree on a shared project management plan, project governance structure and project manager. Each participating organization also requires its own project plan and governance structure. The role of the project manager is key: but in the open source model, the focus is less on managing tasks and more on managing and facilitating interactions and collaboration in a distributed, virtual community of developers. One key consideration is breaking down the software project into smaller, simpler parts, or modules, with minimal interdependencies. This enables individual developers or small teams to work separately. A rigorous planning and priority framework will be required across participating NSOs to ensure that collaborative investments deliver common tools and shared solutions. Identify the project team members with the right skills and mindset from each participating organization, and identify their respective roles. If sub-contractors are used, their contracts must support the need to contribute to code under a specified licence. Employees on the team must be briefed on all aspects of the project, including the licensing model to be used. Ideally, they should be familiar with collaboration on open source software, or ‘inner sourcing’. The roles of developers must be assigned. Anyone can be a contributor, but only those identified as ‘approvers’ can commit submitted code, ensuring quality of the final product — for example verifying that secure coding practices and other standards were followed. Agree about logistics for communications, financial arrangements, resource allocation, etc. During design and development Ensuring that established frameworks and plans are still followed and that participation and engagement remains high is important during this phase, as is a commitment to co-operatively agree on changes to user requirements, specifications, architecture and design. The challenge associated with this cannot be understated: it is likely the largest single risk to the success of any software collaboration initiative. Some additional considerations: Licensing requirements associated with any new components, FOSS or otherwise, should be continually monitored. If an essential tool or code with an incompatible licence is selected, then the licence for the resulting software must be revisited or another tool selected. Developers will need to maintain an internal record of what code is in the software and what licence applies to each element. Some build automation tools support this task. For more complex projects, automated code scanning can be used to conduct an audit of the code. Using code from software libraries that have already audited their code is another way to minimize risks. If additional partners are added along the way, the MOU/agreement should be updated to reflect them. As well, their requirements and contribution into the overarching project plan should be incorporated. New contractors hired to contribute code under the FOSS licence (i.e., part of the shared core) need to have contract clauses compatible with the license selected. If there is an impasse on a key decision, or timelines no longer meet the needs of a partner, that partner can go it alone from that point on, assuming the license issued permits it. The partner might rejoin the project later — this is facilitated by social coding platforms, which is one more reason to use one. Collaboration can still continue, but in a different form. Organization-specific components, interfaces, or modifications to in-house systems, as well as development of new ‘services’ to in-house systems to support the use of the new software, are the responsibility of each organization. Each partner must also ensure that these will work with the core software, and that project timelines are aligned. Of course, any organization-specific configuration including reference files (for example, to drive multilingual support) are also the responsibility of each organization. After development is completed Each partner must conduct its own integrated testing on its infrastructure, then analyze track the issues that are its own to fix. Training is also best provided by each organisation. Software collaboration at StatCan Page 8 of 23 Before distributing the software, a last due diligence assessment of licence obligations and licence interoperability should be conducted. Once the shared software and all in-house components are completed, each organization must implement on its own infrastructure (unless infrastructure is shared or using a SaaS model) The support model for the shared software should be defined, taking into consideration the requirements of all parties. Maintenance of the software itself is best done collaboratively as per the open source model, but arrangements can be made for some partners to contribute financially rather than in kind. First-level enduser support is best provided by each partner; second-level support can be provided collaboratively. Changes or enhancements to the initial requirements must be agreed to collaboratively with every attempt to address unique functionality via add-ons. The shared software’s roadmap will need to be managed collaboratively. Forking is an option at any point, if all else fails. Partners should take stock of the initiative and evaluate its success, and analyze lessons learned to inform any subsequent collaboration initiative. 5. Conclusions and Recommendations StatCan’s vision is to design and construct all statistical production systems using a component-based approach built on robust and flexible common systems and processes. Software collaboration with NSOs and other partners supports this vision, and offers many opportunities. Collaboration can be achieved by sharing software and other deliverables or services, or by working together towards a common goal, for some or all stages of software development. Hybrid models are also feasible: it is also possible to go from one model to another, with some limitations. Key models possible include: Model Scope of collaboration Attributes Share code The organization that developed the software provides a ‘licence to inspect and modify the code’, and possibly to distribute the modified code. Typically the code is provided ‘as is’, with no support or maintenance. The organization that developed the software provides a ‘licence to use’, directly or via a reseller. Typically, maintenance and support is also provided. Easiest model for the providing organization. Most flexible for the acquiring organization. Offers the least financial benefits. Has been used at StatCan with various partners. Similar to using a COTS. For the acquiring organization, this model is best if the provider provides support and maintenance and incorporates its requirements into future releases. Can result in significant savings for the acquiring organization when compared to the cost of building their own solution. Can be burdensome for the provider organization. Has been used at StatCan with various partners. Offers the most challenges and the most benefits. Has been used at StatCan in context of FOSS. Key enablers for using with NSOs include common standards and models service-oriented architecture business architecture alignment. Share executable Collaborative development Organizations develop the software together. Partners can contribute in kind or financially, and third parties may also contribute, such as contractors or the Internet community. An appropriate open source software licence would best meet the needs for this model. StatCan should continue maximizing the use of the first two models, and select with its partner NSOs an appropriate first collaborative software development initiative. The functionality selected should offer mutual value to all partners and leverage the shared models and common statistical architecture, and the requirements should already be aligned. The resulting shared software should be compliant and secure, robust, scalable, interoperable, flexible and high-performance. To maximize success, the recommended approach is to adopt best practices, processes, culture and tools and methodologies from the open source, to adopt agile and lean methodologies, and to use a recognized open source licence. The licence selected for a given initiative should be the one that best supports the desired outcomes, meets all partners` current and future needs and addresses constraints imposed by any existing code that will be part of the final product. Some key success factors for collaborative software development are clearly articulated purpose, outcomes and roles and responsibilities strong project management, with a focus on participant co-operation, communication and empowerment Software collaboration at StatCan Page 9 of 23 commitment to compromise to achieve desired common outcomes, supported by a solid planning and change management framework to control diversity, complexity and scope strong management support engaging all participants who have the right mindset mechanisms in place for continuous interaction of participants a core of participants with experience with open source software development or co-development. Key challenges are adjusting to the culture of the open source model fear of loss of control compliance with legislated standards and directives of each country. StatCan has seen success in many software collaboration and sharing initiatives, and has experienced challenges in others, from which important lessons were learned. In addition, StatCan has significant experience building generalized systems and has adopted service oriented architecture and shared models such as GSIM and GSBPM. StatCan`s commitment to building component-based solutions makes it now possible to pursue a software collaboration initiative with partner NSOs. Such an initiative, even partial in scope and resulting in separate systems, would be worthwhile. In addition, StatCan is looking into making some of its existing generalized systems available to other NSOs under a FOSS licence, provided it can ensure that the software can continue to meet its own requirements. Software collaboration at StatCan Page 10 of 23 Annex A — FOSS3 FSF: The Free Software Definition A program is free software if the program's users have the following four essential freedoms: The freedom to run the program, for any purpose (freedom 0). The freedom to study how the program works and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this. The freedom to redistribute copies so you can help your neighbor (freedom 2). The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this. The Free Software Foundation (FSF) and the Open Source Initiative (OSI) publish the two most widely accepted definitions. The FSF adopts a principles-based approach that promotes the idea that software should be “free as in freedom.” The FSF's ‘Free Software Definition’ sets out four essential freedoms for free software (see text box at left).4 The FSF also maintains a list of approved free software licences that meet its criteria under the definition.5 The OSI adopts a more business-focused approach to FOSS, promoting the ways that the private sector can benefit from using and releasing open source software. The OSI’s ‘Open Source Definition’ has 10 criteria for open source software (see table below).6 Where an open source licence meets these criteria, it becomes ‘OSI approved’.7 OSI: OSS licences must permit and include the following: 1. Free redistribution 6. No discrimination against fields of endeavor 2. Source code 7. Distribution of licence 3. Derived works 8. Licence must not be specific to a product 4. Integrity of author's source code 9. Licence must not restrict other software 5. No discrimination against persons or groups 10. Licence must be technology-neutral One distinctive and commonly-encountered feature of FOSS divides licences into two categories: with a hereditary obligation (also called ‘copyleft’, ‘share-alike’, or ‘reciprocal’), which stipulates that modifications to the software must also come under the same licence. If users makes changes to hereditary software or include some hereditary-licenced code in their own software, they cannot distribute the new work as restricted-source software: they must only redistribute the new code under the same FOSS licence; without such an obligation.8 FOSS licences without this stipulation are generally referred to as permissive. In these cases, authors of any modifications are under no obligation to place their changes under the same licence, or even under a FOSS licence at all. In general, authors of modifications can even use and redistribute the modified software as a restricted-source software application. The two different types of FOSS licences are illustrated in the figure below. A. Distribution under a hereditary licence B. Distribution under a permissive licence Source: Free and Open Source Software Licensing Primer for Natural Resources Canada – Hickling, Arthurs and Low (2012) 4. Free Software Foundation, “Free Software Definition.” 2012. <http://www.gnu.org/>. 5. Free Software Foundation, “Various Licenses and Comments about Them.” 2012. <http://www.gnu.org/>. 6. Open Source Initiative, “The Open Source Definition” (accessed 2012). <http://opensource.org/> [OS Definition]. 7. Open Source Initiative, “Open Source Licenses” (accessed 2012). <http://opensource.org/>. 8. See, for example, Heather Meeker, The Open Source Alternative (John Wiley & Sons, 2008, Hoboken NJ), 22. 3 Software collaboration at StatCan Page 11 of 23 However, the distinction between these types of licences can be blurred: different hereditary licences stipulate to varying degrees when this hereditary obligation to distribute under the same licence engages. This is perhaps the most confusing aspect of FOSS licences and deserves careful attention. To determine whether a certain activity invokes a hereditary obligation, one must consider the type of distribution and the extent of integration with the original code. With respect to the type of distribution, hereditary obligations only arise with certain ‘distribution-like’ activities. Where there is no such distribution, the licences almost always allow a person to use and modify hereditary-licenced software without ever releasing their code. For example, StatCan can change and customize software under a hereditary licence for its own use or any GC use, and is not obligated to share this software or the software source code. However, a ‘distribution’ does trigger a hereditary obligation to distribute under the original licence. Two common types of triggers are found in FOSS licences: distribution of source or object code: Distribution of the software, either over the Internet or on a physical medium such as a CD, whether as source code or object code, is the most common trigger for a hereditary obligation. For example, this is the trigger set out in the popular GPL licence. Access over a computer network: This is a much broader trigger that imposes a hereditary obligation whenever others access the licenced software over a computer network. Note that even if the software is distributed, the hereditary obligations still only extend to certain modifications, depending on the licence. The subsistence and extent of the copyleft feature is generally the FOSS issue most important to FOSS users and contributors. The table below presents the key differences in benefits for both types of licences. Permissive Hereditary Beneficiaries of the FOSS Everyone: commercial software vendors, support services, etc. release Everyone, but only those willing to release their software as FOSS, under the same licensing terms as were granted to them (some restricted-source software vendors absolutely prohibit hereditary licences such as the GPL). Beneficiaries of downstream code changes The whole community, but only if the business, or other developer, chooses to contribute modifications back under the permissive licence. The whole community in every case where a business, organization, or individual distributes the modifications, as the licence then mandates releasing the changes under the same FOSS licence. Licence complexity Often very simple and understandable (e.g., the popular ‘Two-clause BSD’). Relatively complex, requiring careful legal analysis (and some risk of misinterpretation). Interoperability Permissively licenced code can be included in projects under hereditary licences, other permissive licences, or restricted-source licences. Hereditary-licenced code cannot generally be included in a project under any other single licence. Benefits of FOSS Cost. The total cost of ownership for using any software involves three major categories of expenses: upfront licensing costs, ongoing maintenance and support costs, and software upgrade costs. When using FOSS, the up-front licensing cost is zero (aside from internal costs of assessing the software and the licence). As well, FOSS software often incurs lower ongoing maintenance costs, as it permits anyone to install, fix, and otherwise support the software. This can enable a more competitive tendering of support services amongst firms. Software collaboration at StatCan Page 12 of 23 Flexibility. As the code is available, it also allows an organization to adapt it to meets its own unique needs, within the constraints of the FOSS licence, which might require for example to contribute back changes or limit the choices of licences when distributing the modified software. Interoperability. FOSS fosters the use of open formats and standards, which promotes interoperability. Security. When using FOSS, all of the source code is openly published. This makes it difficult for anyone to surreptitiously insert malicious code. It also enables security auditors and security researchers to inspect the code for flaws. For this reason, defense and national security agencies very often use FOSS.9 Where FOSS is well-developed and well-inspected by third parties, security is generally improved. There are some security risks as well, discussed below. Avoiding vendor lock-in. When a restricted-source software application becomes entrenched in an organization’s or business’s processes or products, the organization becomes ‘locked in’ to that software vendor for any new features, bug fixes and, in many cases, product support. Where a vendor is unwilling to implement a new feature, the organization may need to switch to an alternative — often at considerable cost. When a particular vendor is slow to provide bug fixes or support, timelines and security may be compromised. FOSS offers the advantage of creating an open marketplace for providers of all types of support. Any support business with sufficient software development competencies can add new features and fix bugs in the software; FOSS users can also switch to a different support provider whenever an existing company no longer meets their needs or timelines. Support for the local economy. Depending on the dynamics of a software industry in a particular locality, FOSS use may better support local businesses in the area.10 Because FOSS is openly available, distributable, and modifiable, a greater breadth of smaller support service businesses may provide commercial services. Rather than a single vendor providing the warranty and technical support, any competent local IT business can provide these services. Rather than a single vendor being the only one in a position to customize software, a local business can provide specialized versions of the software. Risks and Drawbacks Lower user familiarity. User familiarity with FOSS software is often much lower than with restrictedsource software packages. Restricted-source software has a strong foothold in schools (at least in Canada) and many people learn to use computers with restricted-source software, as it is pre-installed by hardware suppliers. For this reason, FOSS can require more training and support.11 Security risks. Because all of the source code of FOSS is openly published, anyone — including ‘black hat hackers’ — can examine the code for security holes.12 In particular, malicious attackers can observe where FOSS software shares the same design, code base, or architecture as software that may have known security vulnerabilities.13 Of course, this drawback must be weighed against the security benefits mentioned earlier. In many cases, malicious hackers can still gain access to the source code of restricted-source software where non-disclosure agreements, company ethics procedures, or vendor security mechanisms fail. While restricted-source software security depends on an attempt to maintain an information imbalance between the 9. Fadi Deek and James McHugh, “Open Source Technology and Policy” (New York: Cambridge University Press, 2008), 310–311 (extensive use of FOSS in the U.S. Department of Defense and the National Security Agency) [Open Source Technology and Policy]. 10. Ibid. 11. See Migration of Public Administrations, supra note 12. Kirk St. Amant and Brian Still, Handbook of Research on Open Source Software (Hershey: Information Science Reference, 2007) [Handbook on OSS], 210. 13. Ibid., 192. Software collaboration at StatCan Page 13 of 23 ‘white hats’ (computer security analysts) and the black hats, FOSS software security depends on an open competitive process of finding and closing security holes. Loss of control over software management. The easy, no-cost option of downloading FOSS from the Internet, and instantly installing it, offers an attractive way for employees to sidestep procurement processes. This creates two problems: employees may contractually commit the organization to the terms of the software licence without management’s knowledge; and losing track of what software is running on employee workstations complicates systems security management. However, these systems management issues are present whether or not an organization chooses to make use of FOSS. A licence-specific obligations map is presented below, but this general mapping is only a guideline. Nuances in particular licences often alter these sets of obligations. The following decision flowchart sets out more fully these core obligations for the most popular FOSS licences.14 14. Shown are core obligations only for BSD licenses (two-clause and three-clause), MIT license, GPL license suite (versions 2.0 to 3.0), Apache License 2.0, Eclipse Public License 1.0 (EPL), and the Mozilla Public License (MPL, versions 1.1 and 2.0). Software collaboration at StatCan Page 14 of 23 Annex B 6. Software OpenM++ and Modgen Both OpenM++ and Modgen are microsimulation development frameworks. They are used to develop continuous-time event-based microsimulation models. StatCan develops these models for its own analytical needs, but also builds models for clients on a cost-recovery basis. Modgen is the software developed and currently used by StatCan and others. It is now at ‘end of life’ and needs to be replaced. OpenM++ is a proposed set of open source tools that will be portable and scalable, provide 64-bit support, will support parallelism and may replace Modgen. OpenM++ is also targeted to provide software as a service (SaaS) models for the research community. Function Partners Funding Collaboration model How it is used Key contacts For OpenM++ (new) o University of Ottawa (U of O) — current lead o Statistics Canada — expected future lead o External Web Communities (future) For Modgen (StatCan’s current tool) o Statistics Canada (owner) o Various NSOs and other organizations (users) For OpenM++ o OpenM++ initial development is funded by a grant received by U of O o If StatCan adopts it as a replacement for Modgen, it will fund its maintenance and support, with in-kind contributions from partners Modgen development was primarily funded by StatCan; its maintenance and support is fully funded by StatCan Modgen’s model is ‘share object code via licence’ — free of charge, with some collaborative development. Client organizations such as the World Health Organization have provided funds to implement enhancements; others have made in-kind contributions — for example, the Canadian Partnership Against Cancer developed a web interface. Initially, U of O’s proposal was to co-develop a new version of StatCan’s Modgen software. However this approach failed (see lessons learned): instead, U of O proposed to begin development of a new FOSS, tentatively called OpenM++. OpenM++’s model is ‘collaborative development using FOSS licence’. Initial design and development of OpenM++ is being done by U of O, with participation by StatCan and others (later). The framework, a key component of which is a pre-compiler, is used by subject-domain experts to build complex microsimulation models using statistical data. Researchers then use the models to support analysis — conduct what-if scenarios, etc. They also do this analysis with tools that are part of the framework. Role Name IT Team Lead responsible for the Michael Goit project at StatCan (613-951-5921) System Engineering Division Informatics Branch Project Manager: U of O University of Ottawa Steve Gribble Program-area Director responsible Chantal Hicks at StatCan (613-951-5311) Licensing Organization in Modelling Division in Analytical Studies Branch The FOSS license for OpenM++ has not yet been selected but StatCan favors a hereditary license Software collaboration at StatCan Page 15 of 23 OpenM++ and Modgen Software 6. model to maximize the receipt of downstream changes, and to ensure that work built remains open and free. For Modgen, a simple licence to use the software ‘at your own risk’ is used. StatCan status and outlook Methodology, OpenM++ is at the design and prototyping stage. Modgen has been in production for 15 years. StatCan is taking a keen interest in the development of OpenM ++. If OpenM++ can ultimately replace Modgen, StatCan may be able to avoid a multi-year redesign project on its own. StatCan hopes to have enough information to make a decision on OpenM ++ by the end of 2013. platform and tools Lessons learned Potential for NSOs For OpenM++ o Subversion repository and wiki (for now) o Linux platform for compiling and testing o Agile methodology is used o Primary development environment are Windows using Microsoft Visual Studio 2012, and C++ 11 as the primary language for coding o Parallel hardware/software technologies should be used, as well as FOSS tools for parallel computation, lexical analysis and grammar parsing, as well as database No lessons have been learned yet on the co-development process for OpenM++ as it is still in the early stages. However, originally U of O, which needed a 64-bit version of Modgen and received a grant to fund this effort, proposed to co-develop this new version. Ultimately, this option was not pursued because of clauses in the co-development agreement requested by U of O, to minimize their risks, plus constraints and issues associated with these requirements for StatCan. Some lessons learned from this effort: Before embarking on a co-development initiative with an organization outside the GC, ensure that it is not being done as an alternative to procurement. One key consideration: if the primary objective of the co-development arrangement is to meet StatCan objectives, this could be seen as an alternative to procurement. IP cannot be promised to be relinquished ahead of time (even after it was modified by the other party) upon pre-determined events such as StatCan “stopping support” of the software or “not accepting the work delivered by the other party as the official version.” StatCan should not promise to provide a licence to freely redistribute the modified code ahead of time, unless for a limited time frame (e.g., 5 years). Instead, StatCan should only offer to “consider in good faith to grant further rights” at a future date, once it has the facts to make an informed decision. Otherwise, the GC would be facing an endless commitment to notify the other party of the software ‘support status’. Essentially, the GC would be making a decision ahead of time without knowing the entire context it may be facing at this future date., U of O, for its part was concerned about making a financial investment in modifying the code without assurance of being able to not only use the modified software, but also to distribute it freely if StatCan stopped support or did not accept the modified version. A consideration for StatCan was the impact on its program and its clients of having potentially multiple production versions of Modgen in use and possible liabilities faced by StatCan for the relinquished modified version. Modgen is already used by other NSOs and others. OpenM++ is expected to be similarly used by other NSOs. Software collaboration at StatCan Page 16 of 23 7. Software Web Experience Toolkit (WET) The WET is a set of web accessibility and presentation tools. It consists of reusable components for building and maintaining websites that are accessible, usable, and interoperable. WET eases compliance with the Government of Canada Standard on Web Accessibility, the Standard on Web Usability and the Standard on Web Interoperability. The WET solutions were applied to the presentation framework of Drupal to develop a publically accessible distribution platform to meet Government of Canada requirements for accessibility, usability and interoperability. The framework has been tailored for organizations that need to comply with standards for accessibility and bilingualism or that simply need a distribution that gets them up and running quickly using a set of modules that can support common enterprise business requirements. Function Within StatCan, the WET tools and Drupal will be used for all StatCan external sites by 2016 (approximately) and for most intranet sites. It is also used for the new open data portal developed by StatCan, data.gc.ca. Key projects recently implemented using this new framework include the Canadian Economic Action Plan, the Statistics Canada blog, consultation module and Innovation Channel, the Public Works buyandsell portal, Science.gc.ca, Ottawa.ca, and the University of Ottawa (in September 2013). Partners Funding Collaboration model How it is used Core resources funded by TBS. Contributions in kind provided by all partners. The ‘collaborative development using FOSS licence’ model was used. The WET software/tools were co-developed from inception using a FOSS license, although the number of contributors has changed over time. It is based on Drupal, a pre-existing renowned FOSS web content management system (WCMS) with a much broader community. Key contacts Treasury Board Secretariat (TBS) is the lead. Canadian Public Service organizations (federal, provincial and municipal), and nongovernment organizations. Key participants include Statistics Canada, the Canadian Transportation Agency, the City of Ottawa, and the University of Ottawa. External web communities. More than 93 members representing 50 organizations now contribute to the Drupal Web Content Management Framework project. The WET tools themselves and the Drupal software are used and configured by IT developers to build WCMSs). The end users of the resulting WCMS application are the content owners of the websites that use them to manage the content of their websites. Ultimately, the general public are the end users of the external websites created using Drupal and the WET, and StatCan employees are the end users of intranet sites created using these tools. Role Name Organization IT Chief responsible for the project Andrew Sinkinson at StatCan (613-951-6882) Administration and Dissemination Systems Division (ADSD) in Informatics Branch (IB) Project Manager, GC level Treasury Board Secretariat Software collaboration at StatCan Paul Jackson (613 960-1032) Page 17 of 23 7. Software Web Experience Toolkit (WET) Program area Chief responsible Dissemination Division MIT license is used for WET. Drupal is licensed using the GNU General Public License, version 2 or later. Licensing model StatCan status Lucie Gauthier (613 951-5374) and outlook StatCan has been using WET solutions (widgets) since February 2012. WET templates were adopted in January 2013. Drupal has been in production at StatCan also since January 2013. Initially, it was used only for static websites. The current WET is here to stay at StatCan and in other GC departments. The recent launch of the Web Renewal Initiative: its goal is to modernize the Government of Canada's online communications capabilities, including websites and use of social media. This will enable the government to engage online with its constituents using Web 2.0 technologies, and enable more and better access by citizens and businesses to government information and services on mobile platforms. The first release is the consolidation of Internet sites to one single Internet portal www.canada.gc.ca for information on programs, services, initiatives and Government of Canada products. The GC will soon select and procure the WCMS for the consolidated Internet portal. tools GitHub used as the ‘social coding’ platform. IRCan was used previously Development tools are various IDEs, xdebug, GIT and Drush. Development methodology is Agile (iterative). Lessons While there were growing pains at first related to the development process, this initiative has matured. It is now a highly successful collaboration, resulting in high quality websites and significant cost savings. IRCan was initially used as the platform, but GitHub was adopted because o significant resources were needed to manage WET on IRCan o the non-intuitive interface discourages participation from collaborators and external stakeholders o TBS would need to invest more resources to ensure IRCan complies with the standards on web accessibility and web usability and official languages policy. o No application programming interface enabled automation of time-consuming tasks. Methodology, platform and learned Potential for NSOs Drupal is already used by other NSOs. WET is not currently used but would definitely benefit other NSOs. Other NSOs can leverage the capabilities of the Government of Canada web community. Canada has been putting its web experience toolkit onto GitHub, an open source platform that enables web professionals outside the Government of Canada to use our artifacts and contribute back. Software collaboration at StatCan Page 18 of 23 8. Software BLAISE BLAISE is a powerful and flexible system used for computer-assisted survey processing. It can perform computer-assisted telephone interviewing, computer-assisted personal interviewing, computer-assisted self-interviewing, computer audio recorded interviewing, web-interviewing, interactive editing, high-speed data entry, and data manipulation. Blaise also has full survey management capabilities. It is used worldwide by National Statistical Offices (NSOs) and related scientific research organizations for producing official statistics. These organizations use BLAISE on a wide variety of surveys — labour force surveys, consumer price surveys, multilevel rostering household surveys, panel surveys, business and economic surveys, institutional surveys, health surveys, energy surveys, environment surveys, agricultural surveys and other related surveys. Function Partners Funding Statistics Netherlands is the lead and owner of the software. About 30 organizations worldwide use BLAISE. Westat is the reseller. Initial software development was funded by Statistics Netherlands. Organizations using BLAISE contribute to its maintenance and enhancements by paying annual licensing fees to the reseller. Ongoing support is also covered by the annual licensing fees. Collaboration model How it is used The ‘share object code via licence’ model is in effect, with use of a reseller. Statistics Netherlands developed the software, which is now used by others via a licence to use the software issued to each organization, distributed by the reseller, Westat. Ongoing maintenance and enhancements are driven by a user group representing organizations that use the software. This group meets annually at a users conference and at an annual meeting of the BLAISE Corporate Licence Users Board. The user group communicates throughout the year, discussing issues, improvements, needs and direction of the software. Tier 1 support is provided by the reseller; Tier 2 by Statistics Netherlands. Statistics Netherlands makes all modifications to the software. IT staff use the BLAISE software platform to develop and generate collection applications to support the operations of collection partners. Program areas (business and social) determine the specifications for the surveys and use the end product of surveys — data — generated using BLAISE survey applications. The end product generated by BLAISE (called Survey Instrument) is used by Collection Program Management Division staff in the regional offices and head office to perform interviewing and other data-collection activities for the social, business, institutional and agricultural program areas. Role Key contacts IT Chief StatCan Name at Robert Meester (613-951-7099) Collection Systems Division (CoSD) in Informatics Branch (IB) IT Team Leader responsible Éric Joyal at StatCan (613-951-5550) Collection Systems Division (CoSD) in Informatics Branch (IB) Project Manager for BLAISE Lon Hofman Statistics Netherlands Primary Contact for Reseller (Vice President) Jane Shepherd, Ph.D. (301-294-3967) Westat Program responsible Organization area Assistant Sylvie Bonhomme Software collaboration at StatCan Collection Planning and Management Page 19 of 23 8. Software BLAISE Director responsible Division in Collection Services Branch and Regional The BLAISE license is typical of those used for proprietary software. Selected features: StatCan pays an annual fee for the rights to use the software for a specified number of ‘usage units’. The licensee is the Government of Canada — not just StatCan, as all departments are part of the same legal entity. The licence does not allow StatCan to ‘rent’ the software, host it or time-share it. The obligation to support the software is in effect for one year from signature of the licence, renewable yearly for additional years. There are no clauses that provide the source code to any user organization if the owner or the reseller no longer supports it. Each organization would need to find an alternative solution and migrate its surveys to the new solution. The reseller is liable up to $1 million for direct damages if the contract is terminated for default before it elapses (one year plus four option years). Licensing model StatCan status (613-951-1164) and outlook Methodology, In use at StatCan since 1998 and for a broad range of surveys, BLAISE is expected to be phased out by 2019. In its place, the Integrated Collection and Operations Systems (ICOS) platform and a web-only model will be implemented, featuring electronic questionnaires and other applications built in-house by StatCan. N/A – StatCan does not have the code, so BLAISE is essentially like any other COTS product. platform and tools Lessons learned Potential for This ‘share object code’ model works very well, especially for ‘big players’, who have more leverage to get enhancements they need into the product. This is very much in line with the situation for COTS. As a large-scale user of BLAISE, this has served StatCan well. Challenges for StatCan include the ability for the current software version to meet Common Look and Feel requirements and to be the single tool for collection, irrespective of model, and scalability. Common Look and Feel requirements are a challenge for many GC departments when using COTS and in collaboration initiatives. The model fostered innovation in finding solutions given the broad base of users, which led to the collaborative development of supporting tools, such as CAPI monitoring tools. The model led to other collaboration initiatives with participants. For example, the United States Department of Agriculture shared code built using BLAISE, and StatCan then customized this software. StatCan also established a partnership with the University of Michigan: the two organizations collaborated on the research and development of computer assisted interviewing methodologies, processes and technologies. The collaboration also included sharing of research and development experiences, findings and best practices. BLAISE is already in use in other NSOs. NSOs Software collaboration at StatCan Page 20 of 23 9. Software Generalized Systems — ‘G-Suite’ StatCan’s comprehensive suite of generalized systems spans the spectrum of the survey processing cycle. The suite includes sampling, estimation, edit and imputation, tabulation, disclosure avoidance, export to standardized formats (for dissemination) and microsimulation, to name a few generalized functions. Many of the systems are available in both batch mode and interactive mode, and can be embedded within other systems. They are also very modular, configurable and interoperable; some are now available as web services. Generalized systems do not require the intervention of IT staff to be used within a given application. Instead, subject matter and methodological expertise is needed to ensure that processing, specified by end users’ parametization, is done properly to deliver the correct outcomes. At StatCan, generalized systems are centrally funded. Their use by all program areas is also prescribed for all functions they support. Function Partners Statistics Canada (owner). Various NSOs and other organizations (users). Funding All development, maintenance and enhancements is funded centrally by StatCan. Software support is mostly funded by StatCan: as well, organizations that have acquired production versions of the software and pay annual maintenance and support fees. StatCan has issued close to 200 licences for its generalized systems, not including Modgen, which is provided as a free download. The collaboration model with those outside StatCan is ‘share object code via licence’ – for a fee, or at no cost, depending on the software. Client organizations contribute to development by reporting bugs and proposing enhancements. However, StatCan makes all decisions about the software roadmap, functionality and tool set used. Upgrades are driven by StatCan business requirements and IT drivers. Within StatCan, elements of the collaborative software development process also apply: generalized systems are developed to be generic, and to meet the needs of a large group of users. Collaboration model How it is used Key contacts IT specialists responsible for generalized systems work with ‘business owners’ (typically in Methodology Branch) and with program areas to continually improve the functionality, reliability and performance of generalized systems, and to ensure they are maintained and supported. For most generalized systems, methodologists develop the methodology and write specifications (and often develop prototypes). Then, IT staff design, develop and implement the final system. Final acceptance testing is then performed by methodologists and subject matter areas. IT staff responsible for a program area system that requires automation of a generalized function work with IT specialists responsible for the generalized system to integrate it to meet client requirements. Program areas use the resulting systems/solutions to deliver their programs. Role IT Chief responsible generalized systems Name for SAS Yves Deguire 613-951-1282 Organization System Engineering Division (SED) in Informatics Branch (IB) IT Chief responsible for other Benoît Fontaine generalized systems 613-951-1163 System Engineering Division (SED) in Informatics Branch (IB) Program area Directors responsible Various at StatCan (‘business owners‘) Methodology Branch, Analytical Studies Branch, Dissemination Division, Standards Division. Software collaboration at StatCan Page 21 of 23 9. Software Generalized Systems — ‘G-Suite’ Licensing model StatCan status and outlook Methodology, platform and tools learned The first generalized systems were developed over 30 years ago: the suite has grown, and systems have become more comprehensive, flexible, performant and robust. All systems are ever greened as per an established roadmap agreed on with all stakeholders. Additional systems are still being added — more recently G-Tab (tabulation) and G-Export (export for dissemination). StatCan will continue to invest in its suite of generalized systems for the foreseeable future. It also plans to make all generalized systems into web services, to maximize interoperability and flexibility and simplify integration into other systems. Lessons For production licences provided for a fee, a customized licence is used that gives the licensee perpetual permission to use the software (with restrictions). The small fixed fee includes the costs of issuing the licence, on-premise installation and consultation. StatCan also offers optional maintenance and support of its production licenses for an annual fee. It offers additional services on a cost-recovery basis. For production software provided at no cost (such as Modgen), a customized licence providing the licensee perpetual permission to use the software is used. The software is offered as is, with minimal support. Evaluation software is offered using a customized licence providing, for a limited time, permission to use, at no cost. Unsupported and beta software are also offered using a customized licence, providing perpetual permission to use at no cost. on collaboration /sharing Rational Unified Process is the development methodology. Various tools are used, most commonly SAS, C/C++ and the .NET framework, sometimes in combination. Team Foundation Server is used for code management, version control, collaboration and document management. Developing a suite of generalized systems was very worthwhile for StatCan. It yielded cost savings, standardized methodologies, systems of higher quality that are better supported and maintained, quicker turnaround to develop new applications, and greater developer productivity as they get familiar with the suite of generalized systems and how to integrate them. However challenges include clearly delineating the functionality that belongs in the generalized system versus what is ‘pre-processing’ and ‘post-processing’. Adopting the GSIM may reduce the need for information formatting that makes up most of the pre- and post-processing. Other challenges are having all stakeholders agree on requirements for the core system functionality and managing competing priorities and timelines of different clients and projects. Finding resources with the right skills and ‘mindset’, who can code complex applications that are very generic or metadata driven is another challenge. Statcan would have to face all these challenges, though, when collaborating on software development with other NSOs. For NSOs that acquire StatCan’s generalized systems, the financial benefits are significant given the small acquisition cost and maintenance and support costs. If they also adopt these systems broadly across their organization, they should also accrue the same benefits as StatCan. The only disadvantage is that, while they may influence development, they have no control over what enhancements or fixes are done and when. If they require enhancements that StatCan does not, their needs may not be met. This is essentially the situation when using COTS, but NSOs typically have more influence on the development of StatCan generalised systems does than a typical COTS user. The broad base of users of generalized systems, which includes clients outside of StatCan, contributes significantly to the high quality of the systems. Software collaboration at StatCan Page 22 of 23 Software 9. Generalized Systems — ‘G-Suite’ Potential for Central funding of generalized systems was critical to their success. Early adopters do not have to bear the cost of the shared functionality, and this ensures that systems are continually updated and supported to the benefit of the whole organization. In collaborations with NSOs, central funding would also be of benefit. Governance and directives stating that the generalized systems are to be used whenever a function needs to be automated are also key to realizing the desired outcomes. Providing access remotely in different time zones provides some challenges, especially given our limited capacity. Most generalized systems are already used by other NSOs and others. NSOs Software collaboration at StatCan Page 23 of 23