Deep Dive into Keystone Tokens and Lessons Learned Priti Desai & Brad Pokorny Who are we? Priti Desai Brad Pokorny Advisory Software Engineer, IBM Principal Software Engineer, Symantec Deep Dive into Keystone Tokens and Lessons Learned What token format should we configure in our OpenStack Deployment? Token Formats UUID PKI PKIZ Fernet Deep Dive into Keystone Tokens and Lessons Learned UUID Deep Dive into Keystone Tokens and Lessons Learned UUID • Simplest and Most Light Weight • Version 4 UUID • Configuration in keystone.conf : [token] provider = keystone.token.providers.uuid.Provider Deep Dive into Keystone Tokens and Lessons Learned Token Generation Workflow Request Token with: • User Name • Password • Project Name Keystone Identity User Validation Retrieves User ID Resources Project Validation Retrieves Project ID and Domain ID Assignment Retrieves Roles for this User on the Project or Domain Returns Failure if the User does not have any Role Retrieves Services and Endpoints for all the services Catalog Bundles Identity, Resource, Assignment, and Catalog information into Token Payload Creates Token ID : uuid.uuid4().hex KVS UUID – Token Generation Workflow Token Store them in SQL/KVS: • Token ID • Expiration • Valid • User ID • Extra Sample UUID Token in SQL Backend id: f10700e71ff045cbb850072a0bd6a4e6 expires: 2015-10-08 21:18:43 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-1008T21:18:43.995255Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["bI1EMzqUQM2sqFimOtIPpQ"], "issued_at": "2015-10-08T20:18:43.995284Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "f10700e71ff045cbb850072a0bd6a4e6", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580 Deep Dive into Keystone Tokens and Lessons Learned Token Validation Workflow Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Token KVS Retrieves Token payload from token backend KVS/SQL Check if a token is expired, current time is calculated in UTC No Token Not Found Valid? Yes Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Parse Token and Retrieve Metadata No Current Time < Expiry Time Token Not Found Yes Check if a token matches any revocation events Is Revoked? Yes Token Not Found No HTTP/1.1 200 OK UUID – Keystone Token Validation Workflow Token Revocation Workflow Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Validate X-Subject-Token Before revoking a token, validate it based on Token Validation Workflow Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Filter existing revocation events based on Revoke At No Audit ID? Revoke by Token Expiry Yes Revoke by Audit ID Create Revoke Event with: Audit ID Revoke At Issued Before Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Prune Expired Events Token KVS Set valid to False UUID – Keystone Token Revocation Workflow UUID - Multiple Data Centers VM Instance Token Not Found Nova Nova Keystone Middleware Keystone Middleware Token Validation Keystone Token Found Tokens KVS UUID Tokens US-WEST Users Groups Domains Projects Roles Catalog Assignments Token Validation Request Token Keystone UUID Token LDAP Replication (Directory Tree is always in sync) MySQL Replication (Database is always in sync) UUID Across Multiple Data Centers Token Not Found Users Groups Domains Projects Roles Catalog Assignments Tokens KVS UUID Tokens US-EAST Pros and cons • Pros – Simplest and Smallest Token Format – Recommended for Simple OpenStack Deployment • Cons – Persistent Token Format – Token validation can only be done by Identity service – Not feasible for multiple OpenStack deployments Deep Dive into Keystone Tokens and Lessons Learned PKI/PKIZ Deep Dive into Keystone Tokens and Lessons Learned PKI PKIZ • Cryptographically Encrypted Signed Document using • Compressed PKI X509 Standards • CMS • Prefixed with “PKIZ” • Converted to custom URLSafe format Deep Dive into Keystone Tokens and Lessons Learned PKI/PKIZ Configuration - Certificates • Signing Key (signing_key.pem) : • Generate private key in PEM format • Signing Certificate (signing_cert.pem) : • Generate CSR using Signing Key • Submit CSR to CA • Receive Certificate from CA • Certificate Authority Certificate (ca.pem) Deep Dive into Keystone Tokens and Lessons Learned PKI/PKIZ Configuration • Configuration in keystone.conf : [token] provider = keystone.token.providers.[pki|pkiz].Provider [signing] certfile = /etc/keystone/ssl/certs/signing_cert.pem keyfile = /etc/keystone/ssl/private/signing_key.pem ca_certs = /etc/keystone/ssl/certs/ca.pem Deep Dive into Keystone Tokens and Lessons Learned Token Generation Workflow Request Token with: • User Name • Password • Project Name Validate Identity, Resource, and Assignment Create JSON Token Payload Sign JSON Payload with Signing Key and Signing Certificate openssl cms –sign –outform PEM PKI PKIZ Convert it to UTF-8 Compress using zlib Base64 URL Safe Convert CMS Signed Token in PEM format to custom URL Safe format: • “/” replaced with “-” • Deleted: “\n”, “----BEGIN CMS----”,“----END CMS---” Convert it to UTF-8 Append Prefix PKIZ Store Token into SQL/KVS PKI/PKIZ – Token Generation Workflow Sample PKI Token in SQL Backend id: b460fec2efcd0d803e2baf48d3bcd72b expires: 2015-10-09 20:07:36 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-09T20:07:36.656431Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["8dh07HudSh6rHoU1G9bs-Q"], "issued_at": "2015-10-09T19:07:36.656460Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "MIIDiwYJKoZIhvcNAQcCoIIDfDCCA3gCAQExDTALBglghkgBZQMEAgEwggHZBgkqhkiG9w0BBwGgggHKBIIBxnsidG9rZW4iOnsib WV0aG9kcyI6WyJwYXNzd29yZCJdLCJyb2xlcyI6W3siaWQiOiIxNjg4NDQ5Y2YxZGY0NDgzOWIxMGE0MWUzZDliMDlkZCIsIm5hb WUiOiJhZG1pbiJ9XSwiZXhwaXJlc19hdCI6IjIwMTUtMTAtMDlUMjA6MDc6MzYuNjU2NDMxWiIsInByb2plY3QiOnsiZG9tYWluIjp7I mlkIjoiZGVmYXVsdCIsIm5hbWUiOiJEZWZhdWx0In0sImlkIjoiNDIzZDQ1Y2RkZWM4NDE3MGJlMzY1ZTBiMzFhMWIxNWYiLCJuY W1lIjo…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580 Deep Dive into Keystone Tokens and Lessons Learned Sample PKIZ Token in SQL Backend id: c48321ac51a903b07c264ac3e80809c6 expires: 2015-10-12 18:45:23 extra: {"token_data": {"token": {"methods": ["password"], "roles": [{"id": "1688449cf1df44839b10a41e3d9b09dd", "name": "admin"}], "expires_at": "2015-10-12T18:45:23.806229Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "audit_ids": ["kKmQzTuxSnCN9vo3bzxErw"], "issued_at": "2015-10-12T17:45:23.806257Z"}}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": "admin"}, "key": "PKIZ_eJxtlMtyqzgQhvc8xexTqcPFdsLiLCQEWCSCgAGBdgZscbVxDOHy9CMnc6mpGlWpSmqpW39_Uuv5WTRo2tj9wy CHxiN35dqjqybi9eb6DuE7ZLd7_WxtAd6MtR1wP7PT5PxJE2F7U53WYH5D5qZbc53OSkeWPoo3hdrU7VQwhe5JBReo 71GWv72WT2vLPRk62_XuDmt_T9sZku-veT-xPfUaEk…", "token_version": "v3.0", "tenant": {"domain": {"id": "default", "name": "Default"}, "id": "423d45cddec84170be365e0b31a1b15f", "name": "admin"}, "metadata": {"roles": ["1688449cf1df44839b10a41e3d9b09dd"]}} valid: 1 trust_id: NULL user_id: 1334f3ed7eb2483b91b8192ba043b580 Deep Dive into Keystone Tokens and Lessons Learned Token Validation Workflow Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Unique ID of X-Subject-Token Hash PKI Token with the preconfigured hashing algorithm No Token KVS Retrieves Token reference from token backend KVS/SQL Token Not Found Valid? Yes Read cached token reference and parse: • User ID • Project ID • Audit ID • Token Expiry Parse Token and Retrieve Metadata Check if a token is expired, current time is calculated in UTC No Current Time < Expiry Time Token Not Found Yes Check if a token matches any revocation events Is Revoked? Yes Token Not Found No HTTP/1.1 200 OK PKI/PKIZ – Token Validation Workflow Token Revocation Workflow Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Validate X-Subject-Token Before revoking a token, validate it based on Token Validation Workflow Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Filter existing revocation events based on Revoke At No Audit ID? Revoke by Token Expiry Yes Revoke by Audit ID Create Revoke Event with: Audit ID Revoke At Issued Before Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Prune Expired Events Token KVS Set valid to False PKI/PKIZ – Keystone Token Revocation Workflow PKI/PKIZ - Multiple Data Centers VM Instance Nova Nova Keystone Middleware Keystone Middleware Token Validation Keystone Tokens KVS Users Groups PKI/PKIZ Tokens Domains Projects Roles Catalog Assignments US-WEST VM Instance Request Token PKI/PKIZ Token LDAP Replication (Directory Tree is always in sync) MySQL Replication (Database is always in sync) PKI/PKIZ - Across Multiple Data Centers Token Validation Keystone Users Groups Tokens KVS Domains Projects Roles Catalog Assignments PKI/PKIZ Tokens US-EAST Pros and Cons PKI • PKIZ • Pros Pros – Token validation without a request to Keystone • to Keystone Cons – Larger than standard HTTP Header Size – Complex configuration – base64 –d <pki_token – Not truly feasible for multiple OpenStack Deployments – Token validation without a request • Cons – Still Larger than standard HTTP Header Size – Similar to PKI Deep Dive into Keystone Tokens and Lessons Learned FERNET Deep Dive into Keystone Tokens and Lessons Learned Fernet • Cryptographic Authentication Method – Fernet • Symmetric Key Encryption • Fernet Keys stored in /etc/keystone/fernet-keys/ – Encrypted with Primary Fernet Key – Decrypted with a list of Fernet Keys Deep Dive into Keystone Tokens and Lessons Learned Fernet Configuration • Configuration in keystone.conf : [token] provider = keystone.token.providers.fernet.Provider [fernet_tokens] key_repository = /etc/keystone/fernet-keys/ max_active_keys = <number of keys> # default is 3 Deep Dive into Keystone Tokens and Lessons Learned Fernet Keys • Fernet Key File - 256 bits 83b4sCF0Q4pb3aNWJYtSdtdaH8PMA_5dlN7OswXKbvE= \xf3v\xf8\xb0!tC\x8a[\xdd\xa3V%\x8bRv\xd7Z\x1f\xc3\xcc\x03\xfe]\x94\xde\xce\xb3\x05\xcan\xf1 SHA256 HMAC Signing Key (128 bits) AES Encrypting Key (128 bits) Deep Dive into Keystone Tokens and Lessons Learned Fernet Keys • Fernet Key File Name - Integers starting from 0 • ls /etc/keystone/fernet-keys => 0 1 2 3 4 • Type 1: Primary Key • • – Encrypt and Decrypt – Key file named with the highest index Type 2: Secondary Key – Only Decrypt – Lowest Index < Secondary Key File Name < Highest Index Type 3: Staged Key – Decrypt and Next In Line to become Primary Key – Key file named with lowest index (of 0) Deep Dive into Keystone Tokens and Lessons Learned Fernet Key Rotation Staged Key 0 No Secondary Key 1 Primary Key Staged Key 0 1 2 Primary Key 3 Primary Key Rotate Secondary Key Rotate Staged Key 0 1 Secondary Key 2 Secondary Key Token Generation Workflow Signed using Signing Key Fernet Token Version Current Timestamp Token Payload: Version User ID Methods Project ID Expiry Time Audit ID IV Cipher Text HMAC Padding Encrypted using Encrypting Key Fernet – Token Generation Workflow Sample Fernet Token in SQL Backend gAAAAABWLUzy0dxSNo2--K3trDutnX7LpUpv3us0crQIl8BDHLLd3lR3F243VwnYpNJHIaUiPEE2roYJJNASwBe1swDcr6MYaFR1t9ZYcYF4GRqDm3N9_1EGgXgICbzE_GuUVidG4gky0Cv8 f1nwD7XM26NRh59VEnt2iVTAxlnvAICJDeK5k Deep Dive into Keystone Tokens and Lessons Learned Token Validation Workflow Validate Token with: GET v3/auth/tokens • X-Subject-Token • X-Auth-Token Restore Padding Re-inflate token with “=” and return token with correct padding Decrypt using Fernet KeysNo to retrieve Token Payload Version: Fixed Versioning by Keystone: • Unscoped Payload : 0 • Domain Scoped Payload : 1 • Project Scoped Payload : 2 Determine the Version from the Token Payload Check if a token is expired, current time is calculated in UTC For Project Scoped Token: • User ID Project ID • Methods Token Expiry • Audit ID Disassemble payload to determine validation fields Current Time < Expiry Time Token Not Found No Yes Check if a token matches any revocation events Is Revoked? Yes Token Not Found No HTTP/1.1 200 OK Fernet – Keystone Token Validation Workflow Token Revocation Workflow Revoke Token with: DELETE v3/auth/tokens • X-Subject-Token • X-Auth-Token Validate X-Subject-Token Before revoking a token, validate it based on Token Validation Workflow Sample Revocation Event: { "audit_id": "HVvI0d-cTD21yatAfQc4IQ", "issued_before”: "2015-10-24T21:20:45.000000Z" }, Filter existing revocation events based on Revoke At No Audit ID? Revoke by Token Expiry Yes Revoke by Audit ID Create Revoke Event with: Audit ID Revoke At Issued Before Create Revoke Event with: User ID Project ID Revoke At Issued Before Token Expiry Prune Expired Events Token KVS Set valid to False Fernet – Keystone Token Revocation Workflow Fernet - Multiple Data Centers VM Instance Nova Nova Keystone Middleware Keystone Middleware Token Validation Keystone Validate Fernet Token Users Groups Domains Projects Roles Catalog Assignments US-WEST VM Instance Token Validation Request Token Fernet Token LDAP Replication (Directory Tree is always in sync) MySQL Replication (Database is always in sync) Fernet - Across Multiple Data Centers Keystone Validate Fernet Token Users Groups Domains Projects Roles Catalog Assignments US-EAST Pros and cons • Pros – No persistence – Reasonable Token Size – Multiple Data Center • Cons – Token validation impacted by the number of revocation events Deep Dive into Keystone Tokens and Lessons Learned Fernet Token Validation Revocation Events Impact on Validation Time 600 510.058 400 376.604 300 234.398 200 Response Time (ms) 163.529 124.974 100 83.654 46.406 0 Revocation Events Deep Dive into Keystone Tokens and Lessons Learned 4000 3000 2000 1500 1400 1300 1200 1100 1000 900 800 700 600 500 400 300 200 100 11.17 0 Time (ms) 500 Fernet Token Validation 100 90 80 70 60 50 40 30 20 10 0 89.46 Token Validation Requests 21.55 11.95 8 5.77 1.96 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 2000 3000 4000 Validations Per Second Revocation Events Impact on Validation Requests Revocation Events Deep Dive into Keystone Tokens and Lessons Learned What token format should we configure in our OpenStack Deployment? Fernet for Multiple OpenStack Deployments with minimal Revocation Events Deep Dive into Keystone Tokens and Lessons Learned HORIZON AND TOKENS Deep Dive into Keystone Tokens and Lessons Learned How horizon uses tokens • Tokens for each logged in user • Unscoped token and project scoped token • Token reuse • Reduced transaction load on Keystone • Stored in the session • Configurable token storage methods • Local memory cache • Cookie backend • Memcache • Database • Cached Database Deep Dive into Keystone Tokens and Lessons Learned Cookie backend • Currently the devstack default • Token stored in browser cookie • Secure cookies in production, use https • CSRF_COOKIE_SECURE = True • SESSION_COOKIE_SECURE = True • http://docs.openstack.org/developer/horizon/topics/settings.html • http://docs.openstack.org/security-guide/dashboard/cookies.html • Highly scalable • The dreaded boot back to login Deep Dive into Keystone Tokens and Lessons Learned Cookie backend • The dreaded boot back to login • Now sign in… Deep Dive into Keystone Tokens and Lessons Learned Cookie backend • And you see… • Cookie overflow! Deep Dive into Keystone Tokens and Lessons Learned memcache backend • • • • Allows storage of larger token sizes Tokens stored on server side Requires memcached Can be used with backing DB • http://docs.openstack.org/developer/horizon/topics/deployment.html Deep Dive into Keystone Tokens and Lessons Learned Token hashing • • • • Hashed in Django OpenStack Auth (DOA) Keeps stored token data small Currently not working for PKI tokens New config in Liberty to disable • OPENSTACK_TOKEN_HASH_ENABLED • PKI - Will increase memcache storage requirement Deep Dive into Keystone Tokens and Lessons Learned Multiregion and tokens • Service regions vs. Authentication regions • Service regions in Keystone catalog • Auth regions specified in AVAILABLE_REGIONS • UUID, PKI, and PKIZ Tokens don’t work across auth regions • Token replication is infeasible • But Fernet tokens work between Authentication regions! Service Region Deep Dive into Keystone Tokens and Lessons Learned Authentication Region Horizon and Fernet • Yes, Fernet tokens work with Horizon • Liberty and beyond – No patches necessary • Kilo – Needs a patch for DOA • https://review.openstack.org/#/c/169994/ Deep Dive into Keystone Tokens and Lessons Learned V3 domains • Extra token for Horizon Domain Scoped Token Project Scoped Token "auth": { "identity": { }, “scope”: { ”domain": { “name”: “Default” } } } "auth": { "identity": { }, “scope”: { “project”: { ”domain": { “name”: “Default” }, “name”: “ProjectA” } } } Deep Dive into Keystone Tokens and Lessons Learned V3 domains • Requires changes in Django OpenStack Auth and Horizon • Planned for Mitaka • Info on usage (a bit out of date): • http://www.symantec.com/connect/blogs/how-use-horizon-keystone-v3 • Domains patches: • https://review.openstack.org/#/c/148082/ • https://review.openstack.org/#/c/141153/ • https://review.openstack.org/#/c/196328/ Deep Dive into Keystone Tokens and Lessons Learned Will fernet tokens solve all our problems? • • • • Smaller token size No persistence for tokens Seamless authentication across regions Performance issues with token revocation Deep Dive into Keystone Tokens and Lessons Learned Thank You !!! Questions ? References • Token: https://clubpenguincheatscitya4.files.wordpress.com/2011/08/1_token.jpg • Key to Cloud: https://www.hc1.com/wpcontent/uploads/2013/10/14916002_cloud_computing_and_storage_security_concept_blue_glossy_cloud_icon_with_with_golden_key_in_keyhole _.jpg • User Icon: http://findicons.com/files/icons/1075/scrap/300/user_2.png • Password: http://icons.iconarchive.com/icons/sbstnblnd/plateau/512/Apps-password-icon.png • UUID: https://plugins.qgis.org/static/cache/21/c0/21c0d3fedb5bf42ff8a6a11712595124.png • PKI: http://www.zaheerspeaks.com/wp-content/uploads/2009/10/PKI-Certificate.gif • PKIZ: http://i571.photobucket.com/albums/ss153/rijal_abror/pun170-winzip-file-compress-icon59.gif • Identity: https://www.innopay.com/assets/Uploads/icon-digitalidentity-232x232.png Deep Dive into Keystone Tokens and Lessons Learned EXTRA Deep Dive into Keystone Tokens and Lessons Learned What is an Openstack Token? Key to OpenStack Cloud Deep Dive into Keystone Tokens and Lessons Learned How can I generate a token? Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”MyDomain.com” }, ”name": ”PritiDesai", "password": "secretsecret” } Deep Dive into Keystone Tokens and Lessons Learned How can I generate a token? Keystone curl -s POST https://keystone.com/v3/auth/tokens "auth": { "identity": { "methods": [ ”token" ], ”token": { ”id": ”e8079ab…” } } Deep Dive into Keystone Tokens and Lessons Learned Token generated using password Header: X-Subject-Token: a740dcd6f3fc404aaaf556b9cbd2f994 "auth": { "identity": { "methods": [ "password" ], "password": { "user": { ”domain": { ”name": ”Default” }, ”name": ”Smith", "password": "secretsecret” } Body: { "token": { "methods": [ "password" ], "expires_at": "2015-10-05T20:25:03.180741Z", "extras": {}, "user": { "domain": { "id": "default", "name": "Default" }, "id": "1334f3ed7eb2483b91b8192ba043b580", "name": ”smith" }, Deep Dive into Keystone Tokens and Lessons Learned Token generated using token Header: X-Subject-Token: 3fb7b3b0a0a8489882f07fdb9cd2a990 "auth": { "identity": { "methods": [ ”token" ], ”token": { “id”: “a7409b” } } } Body: { "token": { "issued_at": "2015-10-05T19:40:38.943250Z", "audit_ids": [ "4vNgmP5cQk6sMpPiw7EnCg", "HFwMKdDrSCOq-MAtkXKTlw" ], "user": { "name": ”smith", "id": "1334f3ed7eb2483b91b8192ba043b580", "domain": { "name": "Default", "id": "default" } }, "extras": {}, Learned Deep Dive into Keystone Tokens and Lessons Identity Token Path User Keystone OpenStack Service Step 1: Obtain unscoped token with credentials POST v3/auth/tokens Step 2: Discover projects you have access to GET v3/users/<user_id>/projects Step 3: Obtain project scoped token either with your credentials or unscoped token from step 1. Step 4: Invoke the target service by sending requests to endpoints in token from step 3 Step 5: Validate roles and access metadata in token with Keystone service or Keystone Middleware Step 6: Serve API request Step 7: Return response Token Creation Time Per Request 90 80 70 60 50 40 30 20 10 0 Requests Per Sec 20 15 Time Per Request 10 Requests Per Sec 5 0 UUID PKI PKIZ Fernet UUID Deep Dive into Keystone Tokens and Lessons Learned PKI PKIZ Fernet Token Validation Time Per Request Requests Per Sec 14 100 12 80 10 60 8 Time Per Request 6 Requests Per Sec 40 4 20 2 0 0 UUID PKI PKIZ Fernet UUID Deep Dive into Keystone Tokens and Lessons Learned PKI PKIZ Fernet Configurable token hashing https://github.com/openstack/django_openstack_auth/blob/master/openstack_auth/user.py Deep Dive into Keystone Tokens and Lessons Learned