Topics in Computer Science: COMS E6998-07-2015-03
Micro-service Application and API Development
Lecture 3: Design Specification, Composition, Persistence
Dr. Donald F. Ferguson
Donald.Ferguson@software.dell.com
(Admin: Melissa.Peoples@software.dell.com)
© Donald F. Ferguson, 2015. All rights reserved.
Contents
2
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Top
Level
Design
Specification
3
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Top-Level Design Specification Template
4
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
4+1 architectural view model
http://en.wikipedia.org/wiki/4%2B1_architectural_view_model
5
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
The Architecture Views
http://en.wikipedia.org/wiki/4%2B1_architectural_view_model
•
Logical view : The logical view is concerned with the functionality that the system provides to end-users. UML Diagrams used to represent the logical view include Class diagram,
Communication diagram, Sequence diagram.[2]
•
Development view : The development view illustrates a system from a programmer's perspective and is concerned with software management. This view is also known as the
implementation view. It uses the UML Component diagram to describe system components. UML Diagrams used to represent the development view include the Package diagram.[2]
•
Process view : The process view deals with the dynamic aspects of the system, explains the system processes and how they communicate, and focuses on the runtime behavior of the system.
The process view addresses concurrency, distribution, integrators, performance, and scalability, etc. UML Diagrams to represent process view include the Activity diagram.[2]
•
Physical view : The physical view depicts the system from a system engineer's point of view. It is concerned with the topology of software components on the physical layer, as well as the
physical connections between these components. This view is also known as the deployment view. UML Diagrams used to represent physical view include the Deployment diagram.[2]
•
Scenarios : The description of an architecture is illustrated using a small set of use cases, or scenarios which become a fifth view. The scenarios describe sequences of interactions between
objects, and between processes. They are used to identify architectural elements and to illustrate and validate the architecture design. They also serve as a starting point for tests of an
architecture prototype. This view is also known as use case view
6
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Some Diagrams
http://www.agilemodeling.com/essays/umlDiagrams.htm
7
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Some Diagrams
Class Diagram
Component Diagram
8
Communication Diagram
Sequence Diagram
Component Diagram
Activity Diagram
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Application to 1st Project
Class Diagram
Component Diagram
9
Communication Diagram
Sequence Diagram
Component Diagram
Activity Diagram
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Agile Development and User Stories
10
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Documenting API Example
URI
https://mysite.com:3911/api/members/{id}
HTTP verb
PUT
Parameters
id : Card number of the member.
Body
name : Name of the member.
email : Email adress of the member.
langage : Langage used by member (Fr_CA ou En_US)
Sample body
{
"name":"Mario Cardinal",
"email":“mcardinal@mariocardinal.com",
"language":"fr_CA"
}
Success
Response
Status Code: 204 No Content
Error Response
Error Code
11
Status Code: 400 Bad Request, Body: {"Error Code":"..."}
Status Code: 401 Unauthenticated, see WWW-Authenticate value in header
Status Code: 403 Forbidden
Status Code: 404 Not Found
Status Code: 429 Too Many Requests, see Retry-After value in header
Status Code: 500 Internal Server Error
Status Code: 503 Service Unavailable
10: Inactive member
20: Denied access member
110: Database issues, Retry later
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Observations
Weinberg's Second Law: If builders built buildings the way programmers wrote programs, then the first woodpecker that
came along would destroy civilization.
Sr. Celestina: “If you cannot write it down, you do not understand it.”
“it is a custom
More honor'd in the breach than the observance”
Hamlet Act 1, scene 4, 7–16
Comments
I have gone through a very systematic version of TLDS.
This model applies more to infrastructure SW than application SW.
All development projects do some flavor of this approach.
Just want to give you a feel for the types of things you can include in your TLDS, but you only need a really simple version and I am
flexible.
– Use common sense
–
–
–
–
12
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Application to 1st Project
• You do not need to use UML, but should use the concepts.
• Produce the following diagrams
– Component diagram
– Class diagram documenting the logical datamodels for components
– Some sequence diagrams showing end-to-end flows
• Document (and implement) three components
– Student info management microservice, including datamodel changes
– Course info management microservice, including datamodel changes
– Request router microservice fronting Student info microservice
• Document (implement) use cases with sequence diagrams
– Basic scenarios
– Referential integrity between StudenInfo.student and CourseInfo.course
– Data partitioning and scaling, including configuration, for RequestRouter and StudentInfo
13
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Composition
14
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Introduction
15
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Composition Diagrams
Control Flow
•
•
Data Flow
The diagrams are always “nodes” and “arcs.”
There are three types of composition
–
Structure:
–
–
–
–
Node A <x> node B, where x is Requires/depends/calls/synchronizes with/…
Can be major subsystem composition, or internal module composition
Control:
–
–
16
Structure
Execute A and then execute B, where A and B are tasks.
There is a shared set of “logical data objects” the tasks manipulate.
Data flow: Do A to the message/document/… and then do B to A’s output
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Structure
17
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Monolithic to Micro
StoreInfoService
•
•
Java
SQLite
XXX XXX
•
Node.js
•
Redis
AccountingService
•
PDP
•
MongoDB
XXX XXX
•
MMM
•
NNN
Tradition (J2EE) web application connects “services” using
•
Classpath
•
J2EE resource names and JNDI lookup
But, this gets really hard with microservices
• Documenting dependencies?
ShippingService
•
•
• Resolving dependencies?
Ruby
Amazon S3
• Unanticipated connections/integrations?
• ……
18
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
SOA Component Architecture (Example)
Five aspects of a microservice/component
1.
2.
3.
4.
5.
19
Packaging
Exported interfaces
Required interfaces
Configuration interface
Implementation artifact
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
SOA Component Architecture  Microservices
In microservices
• No explicit “wiring” together of components.
• Dynamically add/modify/move microservices.
• Individual microservices dynamicall bind to dependencies.
20
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Inject application
implementation into
reusable SW containers
Reusable
infrastructure
containers
Reusable SW
containers but with
core technology
and frameworks
21
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Internal Composition
22
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Microservice Internal Composition
• There are typically two elements
1.
A reusable “container” the provides reusable, configurable libraries and functions to general application/solution logic.
– Examples
– REST endpoint and call routing.
– Database connectivity and CRUD.
– Security and Session
– Often come as integrated frameworks, e.g. Node Express
2.
Pure application libraries added to a container.
• Development
– Focuses on application functionality and implementation.
– Select and add to an appropriate container (or build one).
– Deploy instance.
23
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
24
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
External Composition
How do you
•
•
•
25
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
Document?
Resolve/bind?
Add unanticipated?
© Donald F. Ferguson, 2015. All rights reserved.
Answers
• Fortunately,
– There are many good approaches.
• Unfortunately
– There is not standard, agreed, technology independent approach
• Some examples
– Readable documentation and config files
– Conventions on URL namespace
– Service Registry aware/enabled microservices, e.g.
– ZooKeeper
– Consul
– Eureka
– Read code or run it and see what calls fail.
– ……
26
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Some Patterns – Client Side
Service A
Service A
Service B
Service B
Service C
Service C
27
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Server Side Pattern
Router implements
all requested
interfaces.
Router queries registry
and performs mapping.
Service A
Services register
implemented
interfaces
Service B
Service C
Admin task
configures
mapping.
28
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Summary
• No substitute for good API and dependency documentation
– Default “web app” for exploring and testing implemented APIs, perhaps using
tools and frameworks (e.g. Swagger).
– Dependency/called APIs
– User property files to map symbolic names to actual names.
– Some form of router pattern
– Server side
– Client side
– API gateway
• You will eventually wind up using ZooKeeper or something similar
– Dynamic reconfiguration of environment, e.g. add service, move service.
– Avoid failed instances of a replicated service.
– etc.
29
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Event
Driven
30
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Think About 1st Project
Student
Course
• The microservice developers
– Did not know about the other service in advance.
– And could not have added dependencies/calls to ensure integrity.
– There is literally no code to make the calls to dependents.
• Microservices should be composable.
– One aspect is enabling unanticipated dependencies/integrations.
– By emitting events for each CUD on a resource.
31
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
SOA vs Microservices
32
http://www.pwc.com/us/en/technology-forecast/2014/cloudcomputing/features/microservices.jhtml
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Event Driven Integration/Composition
…/student/…
…/course/…
Student
Student
…/feed/student/…
…/feed/course/…
If the microservice emits a feed/log
33
If you have a bunch of microservices
that do not emit events/feeds.
•
Front with an API GW that
POST
PUT
DELETE
GET
•
•
•
Add a new microservice
•
Reads feeds/logs
•
Calls appropriate REST API
•
Has an API for configuring the feeds/logs
to read and mapping to REST APIs
Delegates onto actual service
•
Looks at response
•
Writes a feed
…/student/…
API GW
Student
…/feed/student/…
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
•
© Donald F. Ferguson, 2015. All rights reserved.
Amazon Web Services
34
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Google Cloud Pub/Sub
https://cloud.google.com/pubsub/overview
35
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
RabbitMQ
https://www.rabbitmq.com/getstarted.html
36
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
RabbitMQ
https://www.rabbitmq.com/tutorials/tutorial-five-python.html
37
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Topic Trees
• Subscribe to /sales
• Will see e1 and e2
• If they match the filter
e1
e2
38
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Backup
39
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Microservice Patterns
New Microservice Patterns
http://microservices.io/patterns/
40
Classic Patterns, New Realization
“Enterprise Integration Patterns,” Fowler et al.
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
ACID Transactions
41
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
Distributed Transactions
42
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
ACID Transactions
• ACID transactions
– Atomic: A set of writes all occur or none occur.
– Durable: The data “does not disappear,” e.g. write to disk.
– Consistent: My applications move the database between consistent states, e.g. the transfer function
works correctly.
• Isolation
– Determines what happens when two or more threads are manipulating the data at the same time.
– And is defined relative to where cursors are and what they have touched.
– Because the cursor movement determines what you are reading or have read.
43
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.
CAP Theorem
The CAP theorem, also known as Brewer's theorem, states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees:
• Consistency (all nodes see the same data at the same time)
• Availability (a guarantee that every request receives a response about whether it succeeded or failed)
• Partition tolerance (the system continues to operate despite arbitrary partitioning due to network failures)
44
Micro-service Application and API Development (E6998-07-2015-03) –
Lecture 3: Design Specification, Composition, Persistence
© Donald F. Ferguson, 2015. All rights reserved.