How to Deploy Rational License Key Server in a
High Availability Environment
PDD-2387
Robert W Olsen Boeing
robert.w.olsen@boeing.com
Karthik Narayanan State of Michigan
NarayananK@michigan.gov
Kim Søderhamn Technical Services Consultant IBM
kims@dk.ibm.com
Tokens ... What Are They?
•
What is a Token?
•
A generic ”single license key” unit
•
”Tool neutral”
•
Each tool consumes a pre-defined quantity of tokens
•
•
•
ClearCase uses 5 tokens, Rational Team Concert uses 8
Reusable
Provide flexibility
• Tokens are the most flexible method of licensing
• It is a floating license key where each license has a predefined token value replacing the static license
quantity.
• Tokens allow you to maintain a “token pool” of licenses usable across users, using the right Rational
product at the right time throughout your development cycle.
• When someone stops using a product, the tokens return to the token pool.
1
How a Token Works
Rational License Key
Server(s) (RLKS)
+ART Agent
Pull tokens for license
use
Administration & Reporting
Tool (ART)
Start/Stop pull report
data
Jazz Team Server
(JTS)
+ART Agent (optional)
2
Why Tokens? ... Boeing
Why did your company decide to move from floating to tokens?
• Access to large number of IBM products and capabilities (new tools)
• Tokens allowed us access to larger portfolio of tools without having to purchase licenses for each and
every new tool that Boeing decided to include in Boeing’s portfolio of tools
• Reduction in license deployment from 45 days to 2 days
• This was due to having to go thru a “service request” process if we needed more ClearCase
licenses. We could have had an abundance of ClearQuest requests but no ClearQuest licenses so
we would have to order them. With the tokens it doesn’t matter what the product is.
• End-User/Program license tracking and chargeback improvement
• Programs could now only be charged for “token hour” usage of a tool versus having to pay for a full
year of a license and have that license “sit on the shelf” for a portion of time when not being used
• Supported POC for new products without having to get temporary licenses
• IBM was coming out with new products, especially with the advent of the Jazz Environment, tokens
allowed us to do POC against new tools without having to deal with temporary licenses
3
Why Tokens? ... Boeing
Did everyone move over, or just parts of your company.
•
Everyone that could move over, was requested to move over to tokens
•
Reasons for not moving over:
•
Rational application not tokenized and ...
•
Contractual obligations would not allow to upgrade to tokenized version
•
There are still tools that are not tokenized, so floating licenses are still needed
How did you calculate the number for the number of tokens you needed.
•
IBM's recommendation, based on their experience with other customers who have
implemented tokens, is 7 tokens per developer for Rational products.
•
When calculating, keep in mind “How many products?” and “How many user’s of each
product?”
•
Using a single Rational Tool does not benefit from switching to tokens.
4
Why Tokens? ... State of Michigan
Why did your company decide to move from floating to tokens?
•
Simplified license procurement for a variety of rational products with
•
varying usage demand in the long term
•
Ability to evualate new rational tools and quickly add capacity with less lead time
•
Leveraging the tokens at enterprise level provides a way to standardize the tool sets
across the enterprise
Did everyone move over, or just parts of your company.
•
Multiple SOM IBM software customer accounts of various departments were consolidated
to a single SOM account that included the token licenses
How did you calculate the number for the number of tokens you needed.
•
IBM specialists reviewed the current perpetual license capacity/usage and factored in the
growth/demand forecast to calculate total license pool
5
The 3 Solution Options for HA
1) Redundant (Triad) setup on 3 sites (Boeing)
Requires 3 sites
Cost: 3 servers
Strength - automatic failover handled by RLKS
Weakness: If sites are "far" apart heartbeats between servers can become a problem.
2) Redundant (Triad) setup on a single site with secondary site backup (State of
Michigan)
Requires 2 sites
Cost: 3 servers + 3 server capacity on backup site
Strengh - vm-ware failover
Weakness: No new licenses can be obtained until vm-ware failover is completed
3) Single server with backup on secondary site (Danske Bank)
Requires 2 sites
Cost: 2 servers
Strengh - upgrade can be tested on a single site
Weakness: failover by outage detection (in rare cases server can seem available but
process still failing)
6
Redundant (Triad) Setup on 3 Sites (Boeing)
Site A: Primary Server
Site B: Secondary Server
Heartbeats
Site C: Tertiary Server
•Triad Basics (Rational Tools, non-Jazz)
•Primary Server initial MASTER
•Primary goes down, Secondary takes over becomes MASTER
•Secondary remains MASTER until manual switchover to Primary or
Secondary goes down and Primary takes over
•Tertiary never serves up licenses
•If both Primary and Secondary down, system is down
•Each application would then identify the redundant servers
7
Redundant (Triad) setup on 3 sites (Boeing)
•Triad Basics (Rational Tools – Jazz Based)
•Jazz Tools point to single Jazz Team Server
•Could not apply to three Jazz Team Servers
•With IBM’s support we figured it out
•Under the Server Administration section, in “Advanced Properties”,
change the value of “IBM Rational License Key Server” to the triad
license string
•In “License Key Management”, change “Floating License Server” to point
to itself, save updates
•Scroll to the bottom and verify “IBM Rational Common Licensing Service”
status is “OK”
•The last steps involve installing the license keys and move all of the
users from floating to token licensing
8
Redundant (Triad) Setup on a Single Site with
Secondary Site Backup Capacity (State of Michigan)
TCP/HTTPS/4743
Users
TCP 27000/27001
LDAP server
TCP 27000/27001
F5
ASM
TCP/HTTPS/4743
TCP 389/636
Rational License Server
Application Server
9
State of Michigan License Server Configuration
• Port 27000 (lmgrd daemon) and 27001 (ibmratl daemon) inbound access to the
license servers are required if the servers are behind a firewall
(bi-directional firewall access not necessary)
• F5 load balancer with a public URI in front of the ART application server for SSL
offloading and ASM
(See details for installing ART with a specific DNS name later)
• License server access from remote sites under implementation
10
State of Michigan the Transition to Tokens –
How Did We Do It?
• Engaged the IBM token transition team early on for site review and pre-deployment
planning
http://www-01.ibm.com/support/docview.wss?uid=swg27042135#2.2
• Install/configure the triad and validate using a variety of rational products
• Automate the transition to tokens for ClearCase/ClearQuest clients by updating the
windows registry keys thru SCCM automated scripts
(Including the transition from ClearCase Atria licenses to Rational Common Licensing)
• Document and provide end user instructions to switch a variety of rational products (RPE
(system environment variable), RFT, RSA, RAD (Installation Manager) ,
ClearCase/ClearQuest/ReqPro (RLKAD) from perpetual to token licenses
• Monitor the license logs of the old license server to validate the transition is complete and
ready for decommission
11
State of Michigan Token Implementation –
Lessons Learned
• Configuring a F5 load balancer in front of the Triad servers for license service
did not work and was not a IBM supported configuration
• Urbancode Deploy and Release were token enabled products whereas
Urbancode ubuild was not
• Token activation kits for JTS servers version 5 and below needs version specific
activation kits. They are NOT available by default in the license key center and
need to be requested through the license support team
12
Single Server with Backup on Secondary Site
(Danske Bank)
SITE A
RLKS
+RLKSMonitor
JTS
SITE B
Load Balancer
RLKS
+RLKSMonitor
Need to understand how to get licenses for a dual license server setup- special agreement with
local IBM needed!
13
Understanding Reports and the Log File
• What reporting tools you are using?
• Current vendor vs Rational License Key Server Reporting Tool
• Currently using configured version of OpenIT (https://openit.com)
• Updated to handle the new token outputs in the log files
• Rational License Key Server Reporting Tool (RLKS Reporting Tool)
• Started experimenting with RLKS Reporting Tool
• Still fairly new
• Have been working with IBM Support through testing
14
Interesting Notification in the Log File
Looking at the log file
Understanding “pseudo” denials
10:13:54 (ibmratl) DENIED: "ClearQuest" system@TEST10 [system_cq_win_24591] (Licensed number of users already reached. (-4,342))
10:13:54 (ibmratl) OUT: "TLSTOK" User001@TestSystem01 [ClearCase] (5 licenses)
10:13:54 (ibmratl) DENIED: "ClearCase" User002@TestSystem2 [User002] (Licensed number of users already reached. (-4,342))
10:13:54 (ibmratl) IN: "ClearQuest" User234@TestSystemAB [User234_cq_win_0]
10:13:55 (ibmratl) DENIED: "MultiSite" User543@TestSystemFF [User543] (Licensed number of users already reached. (-4,342))
10:13:55 (ibmratl) OUT: "TLSTOK" User234@TestSystemAB [ClearQuest] (5 licenses)
10:13:55 (ibmratl) DENIED: "ClearCase" User100@TestSystemPP [User100] (Licensed number of users already reached. (-4,342))
10:13:56 (ibmratl) OUT: "TLSTOK" ww730e@A5429583 [ClearCase] (5 licenses)
The “DENIED” you see above are not “true” license denied messages. The denial to be
aware of is when you see “DENIED: TLSTOK”, this would mean that you are out of
tokens!
15
Installing ART on a Specific DNS Name
Will make it easier to remember the URL
Will make it possible to move or replace the machine
Can only be done during installation
After the install completes you must “kill” the automatically starting Tomcat
process.
Then follow the steps enclosed in the Appendix A of this presentation
16
Switching Application Over to Tokens
How you get your license for a triad?
Rational
License
Key
Center
LicenseFile.dat
Triad License
Servers
JazzToken.zip
JTS
Server(s)
17
Switching Application Over to Tokens
Sample License.dat file
SERVER license_server1.boeing.com 55555AAAAAAA 27000
SERVER license_server2.boeing.com 55555BBBBBBB 27000
• Listed as Primary, Secondary, Tertiary
• Should be listed same in applications
SERVER license_server3.boeing.com 55555CCCCCCC 27000
VENDOR ibmratl PORT=27001
INCREMENT TLSTOK ibmratl 2.0 31-mar-20xx 5000 vendor_info="0|IBM Rational Main Token Feature|0" ISSUED=02-Jul-2014 \
NOTICE="Sales Order Number:0054562824_Master_Token_Order“ AUTH={ rational=( LK=D00CAFC9B197) ibmratl=( SIGN="00F8 7BB4 \
C8AB 31AF 82A3 B08A") }
INCREMENT DOORS telelogic 2019.03312 31-mar-20xx 1 \
C0E020D0DB6A25CA719A \
Number of tokens required
for this tool
VENDOR_STRING=IBM:t,TLSTOK,1.0,DOORS,10:DOORS sort=150 ISSUED=02-Jul-2014 SIGN2="169D C852 F3E2 37D3 4A27 F2E7 08C0 \
BC81 EC24 08EB"
INCREMENT ClearCase rational 1.000002 31-mar-20xx 1 4FFFFFFFFFFF \
VENDOR_STRING=0|Floating|0|0:t,TLSTOK,2.0,ClearCase,5:CLEARCASE OVERDRAFT=0 sort=150 DUP_GROUP=U ISSUED=02-Jul-20xx
INCREMENT ClearQuest rational 1.100002 31-mar-20xx 1 899CCCCCCCCC VENDOR_STRING=0|Floating|0|0:t,TLSTOK,2.0,ClearQuest,5:CLEARQUEST \
OVERDRAFT=0 sort=150 DUP_GROUP=U ISSUED=02-Jul-20xx
Note: merge non-tokenized licenses here
18
Switching Application Over to Tokens
• Non-Jazz Tools … ClearCase, ClearQuest, etc
• Linux
• List the triad in the primary, secondary, tertiary order
• Separate servers with “,” not “;” or “:”
• Windows
• Using Rational License Key Server (RLKS)
• Why this order and why a “,”?
• License search path should be same order as listed in license file
• “,” designates “redundant” servers, “:” or “;” designates “separate” servers
19
A Word About Reading in the New License File
• Protecting against inadvertent shutdowns
• When reading in a new license file, need to reset “Start Server Options”
20
How is it working today ... Boeing
• Looking at savings first
• Reduction in license servers
• From license server per program area, now only using 3
• Reduction in server administration
• Less servers, less administrators
• Cost per program for license usage
• This is still being evaluated since we started in end of 2014
• Couple of “gotchas”
• Underestimated token need, was hitting 100% usage and getting token denial
requiring an additional purchase of more tokens
• Getting everyone to use the right syntax and order in their setup
21
How is it working today ... State of Michigan
• Build forge token transition had to be delayed due to not meeting minimum version requirements
Check for rational product and its minimum version requirements that supports tokens.
http://www-01.ibm.com/support/docview.wss?uid=swg21673529
• Token keys have to be created with “measured usage” license quantities for it to work for Rational
System Architect clients.
• Removing the floating license server pointer (while switching to tokens) for a JTS that points to another
remote JTS for floating licenses had its user licenses automatically unassigned if the local JTS did not
have the equivalent activation kits installed
22
How is it working today ... State of Michigan
• Easier license administration/reporting with centralized management of licenses when compared with
AU licenses/activation kits
• ART has several tomcat security vulnerabilities that is under review by IBM for remediation
• ART login id is case sensitive and has been identified as a defect, a PMR is opened and is under
investigation
• Setup of a test environment in triad is in the works for testing patches/upgrades etc.
• Reporting of non-LDAP users license consumption may be a challenge and will have to be investigated
for reporting accuracy
23
Where are we going next … State of Michigan
• ART charge back capability and reporting will be critical for cost sharing of the
token pool by various departments / programs
• Providing the CLM/Jazz/Urbancode as one of the standard solution for
exercising DevOps across the enterprise - Tokens provide the capacity and
Jazz/Urbancode provide the capability.
24
Questions?
Appendix A
Complete the installation of RLKS ART 814.
Do not start RLKS ART at the end of installation. (It starts by it self so you need to kill the process)
Edit following files to change the host-name info alone to new host name.
Edit the value of following key's in properties file <install location>/server/rcladminsetup.resp
register.applications.page\:lqe.discovery.url=https\://<new host name>\:4743/lqe/scr
configure.public.uri.page\:server.webapp.url=https\://<new host name>\:4743/jts
1\:register.applications.page\:application.discovery.url=http\://<new host name>\:4380/scr.rdf
Edit the value of following key's in properties file <install location>\server\conf\rcladmin\RCLServerAdminConfiguration.properties
discovery.url=https://<new host name>:4743/jts/discovery
application.root.url=https\://<new host name>\:4743/rcladmin/Main.jsp
LQE_Endpoint=http\://<new host name>\:4380/lqe/sparql
jts.host=<new host name>
Edit following file to replace existing host-name with new host-name
<install location>\server\tomcat\webapps\ROOT\application-about.rdf
<oslc:Publisher rdf:about="https://<new host name>:4743/application-about.rdf">
Edit the following file and replace existing host-name with new host-name
<install location>\server\tomcat\webapps\ROOT\scr.rdf
<jd:registration rdf:resource="https://<new host name>:4743/rcladmin/jts-registration"/>
<jd:contextRoot>https://<new host name>:4743/rcladmin</jd:contextRoot>
<oslc:publisher rdf:resource="https://<new host name>:4743/application-about.rdf"/>
Start ART.
Access the ART using URL : https://<new host name>:4743/rcladmin/Main.jsp
26
Thank You
Your Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.