Front cover The Business Value of the Jazz Architecture: How the Right Architectural Choices Unlock the Value of Lifecycle Tools Redguides for Business Leaders Carl Zetie John Wiegand Understand how teamwork, communication, and collaboration lead to successful software projects Learn the key role that lifecycle tools play in fostering communication and collaboration Apply the Jazz architecture to eliminate long-standing barriers to teamwork Executive overview Almost as soon as software development entered the business mainstream, it evolved from being the work of a small handful of programmers to the product of a team operating within a recognizable lifecycle, incorporating identifiable activities of requirements definition, design, coding and test. Just as rapidly, practitioners and project managers alike recognized an essential truth: coordination between team members and across the lifecycle is both essential to delivering business value but extremely hard to achieve1. Today, software delivery is both more complex and more business critical than ever. In the first part of this IBM® Redguide™ publication, we talk about the business value of lifecycle integration in software development. We then discuss the major architectural approaches that organizations have tried in the past to achieve that business value. We explain why these approaches, which might seem attractive on paper, fall short in the real world of business. In the second part of this IBM Redguide publication, we introduce the innovative IBM Jazz™ approach. We explain both how it is a new technical architecture that draws heavily on the extraordinarily successful architecture of the web, overcoming the shortcomings of previous architectures, and also a new approach to the relationships between competing vendors and between vendors and customers, emphasizing the creation of a value network. Introduction Increasingly, development organizations turn to lifecycle tools not only in the hope of improving individual productivity, but more significantly, team productivity. Such organizations need to integrate those tools to address the lifecycle challenges of collaboration, integration and information across the whole team, including its business sponsors: Collaboration enables team members from other roles to work together effectively, enabling them to share resources and communicate in context to resolve problems or ambiguities, such as failing test cases, more efficiently. Unfortunately, all too often stand-alone tools for other members of the team build up rather than break down barriers to communication. 1 As early as 1975, Fred Brooks documented his experience managing the development of the seminal OS/360 operating system in the 1960s (The Mythical Man Month, ISBN 0-201-00650-2). He coined the maxim now known as “Brooks’ Law,” which says that adding more people to a late software project only makes it later, a testament to the exponential growth of the complexity of communication in a software team. © Copyright IBM Corp. 2011. All rights reserved. 1 Integration enables the team to manage repetitive tasks such as builds, reliably, rapidly, and with minimal manual intervention. Integration also plays a role in guiding team members through the development process and supporting business-critical governance such as checkpoints, handoffs and audits. However, when lifecycle tools have limited integrations, slow and error-prone manual intervention becomes necessary. Information gathered automatically by lifecycle tools and presented to team members for easy consumption is perhaps the most important business benefit of lifecycle integration, but also the capability most frequently lacking. Timely information provides every stakeholder with insight into project progress and even the ability to forecast future progress through appropriate metrics, such as the rate of feature delivery or the story backlog reduction. In many organizations, though, essential information lies behind the opaque walls of individual tools, making it extremely hard to discover meaningful insights across the lifecycle. The unfulfilled promise of lifecycle tools Companies that adopt lifecycle tools generally do so not merely to solve problems of technical integration, but also with the expectation of a real return on investment (ROI). They hope to reap a number of business benefits: Higher quality Increased customer satisfaction Alignment of IT priorities and resources with business goals Faster time to market The development organization also expects to improve its own internal performance through lower costs, higher productivity and more predictable delivery cycles. These all seem like reasonable expectations, but in reality, the business outcome is often disappointing. Typically, as few as one third of adopters of lifecycle tools report that they achieved the value that they expected, leaving two thirds disappointed with their ROI. Many things can cause this shortfall, but two stand out above all others: The pressure of day-to-day work and deadlines distracts teams from the work required to implement and exploit their tools most effectively. Lifecycle tools prove much harder to integrate than anticipated. As a result, teams fail to reap the expected benefits of tools, both for the individual and, because of the lack of integration, the team. Many teams thus find themselves in this unfortunate dilemma: to reduce the pressure of day-to-day deadlines, they need to better integrate their lifecycle tools. However, better integration requires taking time away from day-to-day work, which is impossible because of the deadline pressures. As an old adage puts it, “When you’re up to your neck in alligators, it is hard to remember that you were trying to drain the swamp.” Sadly, this is hardly a new problem. Vendors have struggled with the need for, and difficulty of, lifecycle integration for many years. It is instructive to briefly discuss the approaches that have been tried in the past to better understand how the Jazz architecture represents an innovative approach that avoids the traps into which previous efforts have fallen. 2 The Business Value of the Jazz Architecture These earlier efforts can be broadly divided into four architectural families: A single repository: Usually controlled by a single vendor, this approach promises a coherent, integrated meta-model and a range of associated tools tightly bound to a proprietary repository technology. In practice, adopters are disappointed to find that they are limited to few choices of tools from the vendor and its closest partners; furthermore they have to give up their existing tools, and migrate their data to a new environment. Universal metadata: Often under the supervision of a formal standards body, a coalition of interested parties attempts to define all the metadata of all the lifecycle assets in the hope that tools will be able to import and export each others’ data. In reality, such over-ambitious coalitions invariably move far too slowly to keep pace with the market, and the desire to create a universal standard proves impractical as each vendor tries to entrench its own advantages into the common standard. Common implementations: If lifecycle tools were all written on a single implementation framework, surely integrating them would become much easier. Unfortunately, this aspiration is even more ambitious than that of universal metadata. It is unlikely that a critical mass of vendors would choose to abandon their existing implementations to start over, or even if they did, that they would all adopt at the same pace as required for compatibility. This approach has been known to work for a focused subset of the tools stack, but not for the broader lifecycle.2 Tool-to-Tool integrations: In the absence of a workable implementation of any of the other approaches, most tool users resort to the most direct approach, namely point-to-point integrations between pairs of tools. While this approach can solve the tactical problem for a few critical integrations, it quickly becomes unmanageably complex as a strategy as the number of tools grows. This approach depends on continuing cooperation between the vendors of every pair of tools that the user wants to integrate, but worst of all, it is extremely brittle. Integrations are prone to break unpredictably every time a tool is upgraded, and users might be held back from upgrading for want of one missing link in their tool chain. First of all, these approaches all suffer from certain combinations of being slow to emerge, disruptive to adopt, and restrictive of choice. Secondly, they do little to address the broader challenges of integration, such as workflow, which crosses tool boundaries, consistent governance regardless of where data resides or the ability to aggregate information about status and progress from multiple sources. Thirdly, they fail to acknowledge the practical difficulties of integrating existing tools and the important data that they contain or processes that they embody. Even when IT organizations do manage to achieve a degree of resource integration, they often find that the multiple tools required for other roles introduce a new problem of their own: each tool brings its own approach to managing coordination across the lifecycle. Each tool may have its own concept of user accounts and permissions, projects and baselines, events and workflow, and all the other elements. This means that tools do not merely connect to each other, but actively coordinate with each other. Given these shortcomings and difficulties, perhaps it does not seem so surprising that so few development organizations achieve the integrations they hoped for nor that so much potential value is left unrealized. Clearly, solving this problem demands new thinking. This realization led to the Jazz architecture. 2 Eclipse and the various integrated development environments (IDEs) and related plug-ins provide an example of successfully applying this approach to one part of the lifecycle. 3 Introducing the IBM Jazz initiative IBM Jazz is an initiative for improving collaboration across the software and systems lifecycle. This initiative consists of three elements: platform; community; and products. Jazz offers the following benefits: An open platform for lifecycle tool integration: The platform comprises an architecture for common lifeycle integration patterns, along with a set of optional frameworks, toolkits and helpers that simplify the work of tool creators. A transparent community working together to integrate and develop lifecycle tools: All stakeholders, whether inside or outside IBM, can collaborate at the Jazz.net website. This collaboration permits visibility into and influence over the evolution of the Jazz platform and products to any interested party. A set of products that take advantage of the Jazz platform: One of the essential features of the Jazz initiative is that it doesn’t apply only to newly-developed tools; existing tools can evolve to take advantage of Jazz integration patterns and can do so at their own pace. The fact that the platform and the products are being developed in tandem ensures that the platform is robust and useful, validated in real customer scenarios. This paper primarily addresses the Jazz platform and, in particular, the architecture of Jazz. Linked Lifecycle Data At the core of the Jazz architecture is an implementation of the concept of Linked Data. This is a design approach, advocated most notably by Tim Berners-Lee, that aims to make the inherent connectivity of the web as valuable and as flexible as possible. As the Linked Data community website succinctly puts it, “Linked Data is about using the web to connect related data that wasn't previously linked, or using the web to lower the barriers to linking data currently linked using other methods”.3 Linked data takes a few simple rules about how one document or asset can refer to another and applies them universally. Those rules are intended to ensure that the names of assets are uniform and unique; that those names can be readily discovered; that the assets are available in common representations that can be processed or formatted automatically; and that assets are richly populated with links to other assets, promoting discovery of rich connections that are not immediately obvious. Linked Lifecycle Data is quite simply an application of the Linked Data design approach and technical implementation to the particular domain of assets in the software lifecycle. We treat each asset, whether it be a Requirement or a Test Case or a source code file, as a resource, with a unique resource identifier, or URI (this is the same as the URL that identifies a web page). Every asset, regardless of which tool created it or owns it, is named and accessed by its URI. For one asset to refer to another it simply stores that URI as the value of a property, and can then ask the tool that holds that asset to perform simple operations (such as retrieving it or updating it) using the URI to identify it. Linked Data requires a shift in thinking from integrating tools to integrating data. The immediate consequence of this shift is that tools are no longer dependent on each others’ proprietary application programming interfaces (APIs) or, worse, internal data structures to find assets and follow relationships. Nor is there generally any need for data to be consolidated into a single repository or exported and imported between different tools4. Assets can remain with the tool that created them, in any format: to provide access to those assets, the tool need merely expose a single standard interface. 3 4 More information about Linked Data and the many companies and individuals supporting it can be found at http://linkeddata.org/. The Business Value of the Jazz Architecture For example, the link from a Requirement to a Test Case is simply a Uniform Resource Identifier (URI) stored in the requirement management tool’s store. The Test Case in turn might have links to other assets, such as Work Items or test logs. When the requirements management tool needs access to the test case in question, it does not need to concern itself with what specific quality management tool is serving its requests, and conversely, the quality management does not care where the request came from. All tool-specific dependencies are eliminated in favor of a uniform, simple interface. Contrast this with older approaches that depended on replicating data and keeping copies in sync. The Jazz architectural approach delivers immediate advantages: A loosely-coupled solution: Tools are not tightly dependent on each others’ APIs or structures that change with every upgrade. Consequently, customers can upgrade their tools independently of each other without waiting for vendors to synchronize their integrations and without extensive and expensive regression testing of every tool upgrade. Technology-neutrality: Jazz does not depend on any particular programming language or technical implementation. Just as the content on the web is generated by applications written a variety of programming languages and technical implementations (many of which did not even exist when the web was first conceived) delivered by a variety of web servers to a growing variety of clients, both desktop and mobile, so Linked Lifecycle Data allows existing and new tools to participate on an equal footing. Thus by imitating the approach pioneered by the web, we gain the freedom for tools implementers to use the language that makes sense to them, allowing in-house developed tools, open source projects, and new tools such as IBM Rational® Team Concert™ all to participate in the web of Linked Lifecycle Data. The concept of Linked Lifecycle Data avoids the feasibility problems of two of the older integration approaches, the single repository or pairwise tool-to-tool integrations. Linked Lifecycle Data also eliminates any need to force convergence on a single technical implementation. However, we also need to beware of the trap of trying to define an all-encompassing meta-model. Another advantage of the Linked Lifecycle Data approach is that we need to define only as much of the meta-model as is required to support specific integration scenarios, to immediately reap the benefit of those integrations. As additional scenarios and integrations are needed and understood, they can be added without invalidating existing integrations. Again, the web itself has proven that this approach is realistic and attainable even on a global scale. 4 In practice, there might be scenarios where for practical reasons such as performance, it makes sense to consolidate certain data in a single place. For example, it might be useful to aggregate status and progress information into a data warehouse to support business intelligence and reporting activities for managers of IT projects. Just as in the equivalent situation for any other production data, the data in the tools remains the “information of record,” while the familiar extract, transform and load steps for populating a data warehouse provide a view that is more efficient for querying and navigation. 5 Linked Lifecycle Data and OSLC Of course, even the best approach to data integration is only relevant if it is widely adopted. For that reason, IBM has attempted to eliminate any hindrance that might discourage adoption of the Linked Lifecycle Data approach. We have established an open, transparent community called Open Services for Lifecycle Collaboration (OSLC) to develop and publish the specifications for lifecycle resources5. The work at OSLC is divided into domain-specific workgroups, including requirements management, architecture management, change management and quality management, with others being explored all the time. Any motivated individual, whether an employee of a vendor or a user company, a researcher at an academic institution, a contributor or committer on an open source project, or simply an individual in his or her own right, can observe, participate in and contribute to the effort. The specifications created by OSLC are as open as the process. Published under a Creative Commons license, they are freely available to any implementer who wants to consume the interfaces or provide the interfaces for others to consume. Furthermore, the Creative Commons license is irrevocable: after it is granted, the right to use the published specification can never be withdrawn nor a licensing fee imposed. Linked Lifecycle Data and richer integrations After tools have adopted this linked data approach to creating relationships, other interesting integrations become much more feasible. For example, it becomes possible to perform cross-cutting analyses, such as asking what the coverage of test cases to requirements is, without needing to know anything about the test management tool. All that is necessary is to verify that each requirement includes a valid link to a test case. Tools can also use Linked Lifecycle Data to provide a simple, low-cost form of user interface integration. In the past, separate tools have had separate user interfaces, and viewing the resources of another tool required the user to learn and run an entire additional tool. Linked Lifecycle Data enables a much lighter-weight approach. One innovation is the idea of user interface previews: a suitably enabled tool can present the user with a preview of the resource at the other end of a link. (If you have used an internet search engine such as Bing or Google that allows you to preview a search result without navigating to the page in question, you will be familiar with this concept). This requires no coupling between the tools or dependency on the APIs of the other tool: it is enabled simply by the data-linking mechanism. The Jazz architecture While OSLC provides a powerful and versatile base for integrating the assets used in the lifecycle, it is only the beginning of full integration. Extending integration across the lifecycle is the role of the complete Jazz architecture, of which Linked Lifecycle Data forms one part. Jazz architecture adds many additional services that promote a consistent user experience, shared processes and governance, and higher value and lower cost of ownership. 5 6 The online home of OSLC is http://open-services.net The Business Value of the Jazz Architecture Unlike older frameworks that required tools to conform to a specific technical implementation, the Jazz architecture frees tools from dependencies on the implementation of the common services. Tool access service interfaces through integration protocols that are technology-neutral.The Jazz architecture enables several capabilities: Users can manage project progress with reports, dashboards, and process checkpoints regardless of which tool owns individual assets. Users can trace changes and perform impact analysis across tools, following links from asset to asset. Users will be able to perform configuration management across all their lifecycle assets without having to migrate all their assets to a single store. Users will be able to define processes that govern the use and status of assets across various tools, even though individual tools might have separate process models6. In order to support these scenarios, IBM Jazz tools conform to a number of conventions that we call “integration patterns.” Each tool in a particular integration scenario that conforms to a given pattern increases the value of all the participating tools and decreases their cost of ownership. Several examples of patterns for information integration are listed in Table 1. Table 1 Common Jazz integration patterns Pattern User view Lifecycle data access I want my tools to be able to expose and share lifecycle artifacts with one another to accomplish several possible day-to-day tasks: Create artifacts in one tool based on tasks performed in or by another (for example, create a defect as a outcome of a security scan or test execution). Identify relationships between one artifact and another (for example, relate a change request to the requirement it satisfies). Access properties of artifacts managed by one tool to support user or automated tasks in another tool (for example, run a test suite based on changes to last night’s build content or view the summary of a work item from my project dashboard). Project dashboards I want to create dashboards to coalesce information from several lifecycle tools. I want to be able to include information from other parts of my business (for example, finance, social business, and so forth) in the same dashboard. Cross-lifecycle reporting and analysis I want operational reports that rely on data from multiple tools to help me understand overall project health, which can give me insight into how a change in one part of the lifecycle might impact other parts. Secure and seamless tool navigation Even though each of my tools have their own user management and access control capabilities, I want them to be able to access data from one another and to support tool-to-tool navigation without disruptive user authentication challenges, while at the same time ensuring data remains secure. Other patterns address additional areas of value such as process integration and common administration. Furthermore, these patterns provide a design template for Rational and its partners to design future software products that can be expected to work well together. 6 Several of these capabilities are already available. Others will be implemented over time. 7 Business value of the Jazz architecture With the Jazz architecture in place, the business value of lifecycle tools finally begins to approach the business expectations that customers have in mind. Customers will be able to reap the valuable benefits of lifecycle integration without having to sacrifice timely evolution, incremental adoption, and open choice of tools. Conclusion This Redguide began with a pessimistic view of the disappointing ROI of typical ALM projects and an analysis of the shortcomings and limitations of previous attempts to deliver lifecycle integrations. We have now seen how the Jazz architecture brings new thinking to the challenge and overcomes those limitations. With the Jazz architecture implemented in Rational tools, IBM delivers the first lifecycle integration architecture with several benefits: Realistic: The Jazz initiative recognizes that customers will not replace their current investments wholesale Pragmatic: Jazz architecture allows tools and services to be upgraded independently, without sacrificing rich integration Open: Jazz integrations support the requirement to have a variety of tools from other sources Built for the 21st century: Designed using web architectural principles implemented with web technologies, Jazz architecture delivers the distributed, scalable, and flexible lifecycle integration that modern businesses demand. This offering is only the beginning. Like the web itself, the Jazz architecture encourages new, innovative, often unanticipated integrations, offering greater connectivity and fostering collaboration. You can join the continuing discovery as part of the Jazz community at this address: https://jazz.net/ The team who wrote this guide This guide was produced by a team of specialists from around the world working at the International Technical Support Organization (ITSO). Carl Zetie is a Senior Strategy Manager in the Rational Business Strategy team and is responsible for numerous aspects of business strategy, including integration and open standards. Before joining IBM five years ago, Carl worked as an industry analyst covering the software development market for several years and authoring numerous industry reports, as well as being a frequent contributor to the trade press. He is also known for his contributions to the software usability community. John Wiegand is an IBM Distinguished Engineer, working in Oregon, US. He has 25 years of experience in software development environments. He holds an MS degree from the University of Illinois. John is the Rational Chief Architect and is responsible for open tooling architecture for the Jazz platform. Before this, John was the principal architect for the Eclipse Platform infrastructure and is a former member of the Eclipse Foundation Board, playing a key leadership role in establishing Eclipse as a successful open source project. 8 The Business Value of the Jazz Architecture Now you can become a published author, too! Here’s an opportunity to spotlight your skills, grow your career, and become a published author—all at the same time! Join an ITSO residency project and help write a book in your area of expertise while honing your experience using leading-edge technologies. Your efforts will help to increase product acceptance and customer satisfaction as you expand your network of technical contacts and relationships. Residencies run from two to six weeks in length, and you can participate either in person or as a remote resident working from your home base. Get more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html Stay connected to IBM Redbooks Find us on Facebook: http://www.facebook.com/IBMRedbooks Follow us on Twitter: http://twitter.com/ibmredbooks Look for us on LinkedIn: http://www.linkedin.com/groups?home=&gid=2130806 Explore new IBM Redbooks® publications, residencies, and workshops with the IBM Redbooks weekly newsletter: https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm Stay current on recent Redbooks publications with RSS Feeds: http://www.redbooks.ibm.com/rss.html 9 10 The Business Value of the Jazz Architecture Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. © Copyright IBM Corp. 2011. All rights reserved. 11 This document, REDP-4781-00, was created or updated on September 13, 2011. ® Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol (® or ™), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml Redbooks® The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: The following terms are trademarks of other companies: Redbooks (logo) IBM® Jazz™ ® Rational Team Concert™ Rational® Redbooks® Redguide™ Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. 12 The Business Value of the Jazz Architecture