A Seven-Step Guide to API-First Integration Key Takeaways Adoption of APIs by enterprises have increased in leaps and bounds over the past decade. Be it RESTful APIs or not, APIs act as the gateway to your digital enterprise. Therefore it is important to properly manage and maintain APIs to provide a better digital experience for the enterprise stakeholders. APIs need to be considered similar to business contracts. Design and implementation of APIs (business contracts) can make or break the enterprise's digital business strategy. Integration-first and API-first are the two widely used strategies in most digital transformation projects; the latter proves to be the most effective and future proof method of the two. APIs in an enterprise can be categorized as edge, utility, and domain APIs. Edge APIs act as the gateway to the enterprise systems while domain APIs expose the internal systems to the integration layer. Utility APIs contain the integration logic acting as the glue between edge APIs and the internal systems. In API-first approach, we go through a seven step execution plan. Firstly, we identify the project objectives, then we get to know the enterprise ecosystem. After that, we identify probable integration points of the existing systems, required integration capabilities and use cases. Then for each identified use case, we design the APIs beginning from edge APIs and finally implement the integration. From the time you wake up to the time you go back to bed, how many digital services do you think you interact with within a given day? All digital media -- e.g., mobile applications, web sites, social media applications, everyday banking applications, and ATMs -- use APIs, and, directly or indirectly, we interact with APIs throughout the day. According to Akamai, the world’s leading content delivery network (CDN) provider, 83% of its traffic came through APIs, in contrast to HTML traffic in 2019. For comparison, the company reported API traffic of around 47% in 2014. Thus, as integration specialists, we can’t ignore the presence of APIs. We have to embrace it and develop our integration use cases with an API-first integration mindset. In this article, we discuss how we can run an integration project in an API-driven manner with a seven-step execution plan. RELATED SPONSORED CONTENT SQL to NoSQL: Architecture Differences and Considerations for Migration Couchbase vs RavenDB: Performance at Rakuten Kobo Single Page Application Security 101 Architecting for Scale [Two Free Chapters] - Download Now (By O'Reilly) Key Concepts: Serverless Database Architecture Before we delve into the details of "API-first integration," let’s review the benefits of using APIs. APIs as a contract We know that API is an acronym for Application Programming Interface. In an API-first integration strategy, we have to think of APIs similar to a business contract. When we get into any serious business with another party, we initially make a contract or an agreement between the two parties. We don’t start doing business until we get the contracts right. Similar to that, in integration use cases, we need to get the APIs done first. With that in mind, we design the APIs to expose a predefined set of capabilities of the backend systems to the frontend applications such as websites, mobile applications, and IoT devices. Figure 1: The role of APIs The benefits Properly designed APIs are a key enabler for any enterprise’s digital presence. The following are a few other examples. Acts as gateways for an enterprise’s digital assets Allows the enterprise to quickly build new digital consumer experiences Opens up new revenue channels and expands existing revenue channels Enables the enterprise to cater to future expansions All the above benefits depend on how we design and implement our APIs and the rest of the integration use cases. Let’s look at how we can achieve the above benefits in our integration projects. Integration-first vs. API-first Nowadays, we predominantly see two approaches taken by enterprises when developing their APIs, be it RESTful or some other. These can be described as integration-first or API-first. An integration-first approach is a traditional bottom-up approach where we build the integration logic first on top of backend services and then expose the integrated services (also known as composite services) as managed APIs to users. In an API-first approach, we start with the API design and then move into the implementation of APIs as well as the integration logic. Since we begin with API design, we call this the API designfirst or, for brevity, API-first approach. Throughout this article, we will use these two terms interchangeably to denote the same approach. Let’s look at those two approaches within the context of a slightly abstracted-out example of an integration project. Project Requirement: Provide a digital experience to customers using a mobile and web application by exposing the company’s current capabilities through an online portal. There are backend and frontend development teams involved in this project. An integration-first approach 1. Based on the project requirements, the backend development team develops the integration layer exposing the existing systems. 2. The backend teams expose the backend services through REST APIs. 3. Then, the frontend team starts connecting their web and mobile applications to these APIs. 4. The frontend team finds it difficult to implement some features with existing APIs and creates a change request (CR) for the backend team. 5. The backend team updates the backend to cater to the new capability and improve the API. 6. This will go for several iterations until the backend and frontend teams agree on the proper API for each functionality. This approach adversely affects the project timelines and costs. There is less communication between the teams and the design flaws are identified at a much later phase of the project. Figure 2: An integration-first approach An API design-first approach 1. The frontend and backend teams get together and design the APIs. The idea behind the involvement of the frontend team in the design is to get a better user experience for the API developer. This in turn leads to shorter development cycles for feature implementations since developers don’t have to work around API design lapses. 2. Build a set of sample services that will respond with mostly static responses called mock backends. 3. The backend and frontend teams start implementing the system in parallel. Frontend teams use the mock backends in their development work. 4. When backends are completed, they are connected to the APIs, and integration testing is done. This approach drastically reduces project delays and cost overruns due to miscommunication between frontend and backend teams leading to changes in APIs and backend systems. After designing the APIs, it can take some time to get the live backend systems up and running for the frontend teams to make API calls and test the system. To overcome this issue, frontend teams can set up dummy services, called mock backends, that mimic the designed APIs and return dummy data. You can read more about it in this API mocking guide. There can be instances where the requirements are vague or the development teams aren’t sure about the right approach to design the APIs upfront. In that case, we can design the API for a reduced scope and then implement it. We can do this for several iterations, using multiple sprints until the required scope is implemented. This way, we can identify a design flaw at an earlier stage and minimize the impact on project timelines. Figure 3: An API-first approach The API façade In software engineering, the façade design pattern is used to provide a more user-friendly interface for its users, hiding the complexity of a system. The idea behind the API façade is also the same; it provides a simplified API of its complex backend systems to the application programmers. As mentioned earlier, what we design using the API design-first approach is the Edge APIs that fronts this API façade. The following diagram shows how it all glues together in an enterprise system. Figure 4: An overview of an enterprise deployment According to the above diagram, we have different backend systems of the organization and the user base (mobile, web, and direct API consumers) on either side. And, we use a combination of Edge APIs and API façades to connect those two entities. An API façade contains the simplified RESTful APIs to expose the functionalities of the backend systems. An API gateway consisting of edge APIs provides full lifecycle API management capabilities, including, rate limiting, authentication, authorization, etc. If we further analyze the API façade, we can see it consists of utility APIs as well as domain APIs. Figure 5: An API façade Utility API If your backend logic is complex or communicates in a different protocol, you can design a utility API to expose that backend to the rest of the integration logic. It isn’t a must to expose all your backend services through utility APIs. If you have a RESTful SaaS application like Salesforce, you can directly connect to it if there are no complex interactions involved. Otherwise, you might design a simplified utility API to front the backend service. Domain API Once you have your backends exposed directly or through your utility APIs, you can use a domain API that contains the message mediation logic or the business logic. This message mediation can contain things like message transformation, routing, service chaining, etc. Identifying digital assets Another aspect of designing an API-first integration solution is to know your existing digital assets. In fact, this is one of the initial tasks of an integration project. There are different types of digital assets, and the way we integrate with them varies for each type. Application silos that hold an enterprise’s key business capabilities Application silos are mostly the systems designed to be used by a specific department or a unit. These were never intended to be used organization-wide or exposed to the outside world. Mostly these systems are tightly-coupled to a specific department’s operating procedures and practices. Exposing these kinds of systems through the mediation layer needs special care and attention. Enterprise SaaS applications These are already developed as services to be used by the organization, hence need minimal effort. These are typically exposed through RESTful APIs. Enterprise data stored in various storage mechanisms, including RDBMS, files, spreadsheets, and CSV files When it comes to data-at-rest, organizations use different formats to store them. We need to get an idea of the data storage mechanisms, formats, and any sensitive data that we need to transfer from one entity to another through our integration layer. Applications that execute the process flows in the enterprise We need to identify any workflow systems in the existing system. These may have different approval processes or human interventions. Systems based on proprietary protocols and data formats There can be existing systems that work in proprietary data formats. For instance, old database management systems, CRMs, etc. Key integration capabilities Understanding the existing capabilities of the system as well as the new requirements is key to a successful integration project. For instance, understanding the required capabilities will be a decisive factor in either using an existing integration product or developing one in-house to cater to these capabilities. Let’s look at some of the key integration capabilities we see in most integration projects. API and service hosting -- There should be a way to develop and deploy services. This can be either REST or SOAP services. Orchestration of services and APIs -- Ability to compose end-to-end services by aggregating multiple services. Routing -- Capabilities such as content-based and header-based routing for incoming requests Transformations -- Data mapping from incoming messages to outgoing messages. This can be of the same data format but different schema, or different formats altogether. Protocol switching and the ability to process different data formats -- Transformation of messages from one format to another. For instance, when exposing a SOAP service through a RESTful API with JSON payloads. Parallel processing -- Scenarios like sending requests to multiple backends in parallel and aggregating backend responses and responding to the client. If you lack the proper experience or knowledge in designing integrations with the capabilities identified, it would be best to get help from an integration vendor. These integration vendors have vast experience working in many integration projects across the globe in different business domains. They will be able to assist you throughout the integration project from API design and implementation to even maintaining the system once it is up and running. When selecting the most suitable integration vendor for your project, you need to look at whether the capabilities you require are provided by that vendor. For instance, you might be looking at a hybrid integration solution to gradually move your existing IT infrastructure to the cloud that would lead to a reduction in IT infrastructure costs. Execution plan We discussed several aspects to be aware of when we undertake an enterprise digital transformation project. Now we’ll describe a seven-step execution plan that follows the API-first strategy, to bring order to the chaos and deliver a successful enterprise integration project. 1. Identify the project objectives By working with stakeholders and gathering requirements, we have to formulate the project objectives. Based on that, we decide the project scope. 2. Understand the enterprise ecosystems a. Identify any application silos b. Understand the data Once we identify the project objectives, we can go through the existing system to identify application silos, different backend systems, data handling requirements, etc., as we discussed earlier under Identifying digital assets. 3. Identify the possible integration points for each system With the objectives of the project, we might not need to touch some of the existing systems in the enterprise, or we might need to think about future expansions and select an integration vendor based on that information as well. Therefore, it is important to identify the systems we need to interact with and the specific functionalities we need to expose from those systems. 4. Identify the integration capabilities required With the understanding of the system’s digital assets and the overall objectives, we can decide on the integration capabilities required as discussed under Key integration capabilities. 5. Design the edge APIs These are the APIs that frontend teams will interact with. By first designing the edge APIs, which mostly cover a particular system functionality, we avoid leaking complexities of backend systems into frontend applications. This leads to a much cleaner API design for application programmers. 6. Design utility APIs and domain APIs Once the edge APIs are designed, the backend team can design the utility APIs and the domain APIs to suit the requirements of the edge APIs. If unforeseen issues are encountered, they can go back and update the edge APIs as required. 7. Implement the frontend and integration logic Once the edge, domain, and utility APIs are designed, both the frontend and backend teams can start implementing the system. These steps can be followed for several sprints by breaking down the business requirements into a set of smaller problems, rather than trying to implement the whole business requirement in one go with a set of edge APIs. Conclusion In this article, we discussed an API-first approach to integration projects that reduce the project costs and timelines, increase collaborative work between teams, and improve the overall quality as well as room for future expansions. Also, we discussed how we can formulate this approach as a strategy with a seven-step execution plan. Gartner says, through 2020, integration work will account for 50% of the time and cost of building a digital platform. So it is of utmost importance to build your digital platform adhering to proper development best practices.
