Building a Robust Multithreaded
Server in Java
Alexander Day Chaffee
jGuru Training by the MageLang
Institute
alex@jguru.com
Copyright © 1998 Alex Chaffee
Abstract
• Mark Andreesen says, ”Client-side Java is dead."
Fortunately, server-side Java is alive and kicking!
This session will quickly introduce you to the
concepts of threading and networking in Java. We
then build a robust, multithreaded server (using a
simple chat as our application). We examine several
different threading models, specifically comparing the
one-thread-per-connection model with the queuing
model.
Copyright © 1998 Alex Chaffee
Introduction
• jGuru Training by the Magelang
Institute
– http://www.jguru.com
– Java Training and Consulting
• Alex Chaffee
– Creator of Gamelan
– Cool Java Dude
Copyright © 1998 Alex Chaffee
Background
• Java language basics
• Java networking
• Java threading
Copyright © 1998 Alex Chaffee
Overview
•
•
•
•
•
Start slow, end up really really fast
Part I: Networking and I/O in Java
Part II: Threads and concurrency
Part III: Multithreaded server design
Part IV: Implementations
– Design decisions
– Object models
– Problems and solutions
Copyright © 1998 Alex Chaffee
Part I: Networking and I/O in Java
Copyright © 1998 Alex Chaffee
Java and Networking
•
•
•
•
•
Built into language
One of the 11 buzzwords
Network classloader
Java.Net API
Based on TCP/IP, the internet protocol
Copyright © 1998 Alex Chaffee
Client-server
• Difference between client and server is
semantic
• It’s all just peers talking to each other
• Protocol - roles, vocabulary, rules for
communication
Copyright © 1998 Alex Chaffee
TCP/IP: The Internet Protocol
Application Layer (HTTP, FTP, SMTP)
Transport Layer (TCP, UDP)
Internet Layer (IP)
Physical Network
Copyright © 1998 Alex Chaffee
Sockets and Ports
• Port: a meeting place on a host
– One service per port
– 1-1023 = well-known services
– 1024+ = experimental services, temporary
• Socket: a two-way connection
Copyright © 1998 Alex Chaffee
Sockets and Ports (Diagram)
Client
Socket
port 13
Time Service
port 80
Web Service
Socket
Server
Copyright © 1998 Alex Chaffee
The Socket Class
•
•
•
•
Socket(String host, int port)
InputStream getInputStream()
OutputStream getOutputStream()
void close()
Copyright © 1997 Alex Chaffee
Socket Code
Socket s = new
Socket(“www.sigs.com”, 80);
• Simple, huh?
• Creates the connection
• We still need to get the data there and
back again
Copyright © 1997 Alex Chaffee
Streams
• A stream is a sequence of bytes
• I/O from disk, network, memory, etc.
is handled in exactly the same way
Copyright © 1997 Alex Chaffee
InputStream and OutputStream
• InputStream:
abstract int read()
• OutputStream:
abstract void write(int b)
• Common:
abstract void close()
Copyright © 1997 Alex Chaffee
Filter Streams
• Like photographic filters
• Change the input (or output) on the
way through
• E.g.: BufferedInputStream,
PrintStream, LineNumberInputStream
Copyright © 1997 Alex Chaffee
Filter Streams (Diagram)
Abc
Abc
FileInputStream
AllCapsInputStream
ABC
BC
DataInputStream
NoVowelInputStream
readLine()
“BC”
DataInputStream in = new DataInputStream(
new NoVowelInputStream(
new AllCapsInputStream(
new FileInputStream(“input.txt”))));
String line = in.readLine();
Copyright © 1997 Alex Chaffee
PrintStream
• print()
– outputs an object, primitive, or String
• println()
– same, but adds a newline
• System.out is a PrintStream
PrintStream ps =
new PrintStream(outputstream)
Copyright © 1997 Alex Chaffee
DataInputStream
• int readInt()
• type readType()
• String readLine()
DataInputStream in =
new DataInputStream(inputstream)
Copyright © 1997 Alex Chaffee
Browser.java (source)
import java.net.*;
import java.io.*;
public class Browser {
public static void main(String[] args)
{
String host = "www.stinky.com";
int port = 80;
String file = "/index.html";
Copyright © 1997 Alex Chaffee
Browser.java (source)
Socket s = new Socket(host, port);
InputStream strIn = s.getInputStream();
OutputStream strOut = s.getOutputStream();
PrintStream out =
new PrintStream(strOut);
DataInputStream in =
new DataInputStream(strIn);
Copyright © 1997 Alex Chaffee
Browser.java (source)
out.println("GET " + file + " HTTP/1.0");
out.println();
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
in.close();
out.close();
Copyright © 1997 Alex Chaffee
Java Servers
• A server listens on a port and accepts
connections
• Must be applications (or signed
applets)
• Only one service per port for the
entire host machine
– Java throws an exception if you try to open
more than one
Copyright © 1998 Alex Chaffee
ServerSocket
ServerSocket ss = new
ServerSocket(1234);
Socket socket = ss.accept();
• accept() returns only after a client
makes the connection
Copyright © 1997 Alex Chaffee
ServerSocket
• Now we have a socket, so we can
call...
InputStream in =
socket.getInputStream();
OutputStream out =
socket.getOutputStream();
Copyright © 1997 Alex Chaffee
EchoServer.java
ServerSocket ss =
new ServerSocket(1234);
Socket s = ss.accept();
InputStream in = s.getInputStream();
OutputStream out = s.getOutputStream();
Copyright © 1997 Alex Chaffee
EchoServer.java
int ch;
while ((ch = in.read()) != -1) {
out.write(ch);
}
out.close();
in.close();
Copyright © 1997 Alex Chaffee
EchoServer Demo
•
•
•
•
Connect
Type some stuff
Disconnect
Problem: Multiple simultaneous
connections
– Connections are made, but server doesn’t
respond
– Even if we loop, can still handle only one at
a time
Copyright © 1998 Alex Chaffee
Multiple Connections
• Solution:
– Java threads!
Copyright © 1998 Alex Chaffee
Part II: Threads
Copyright © 1998 Alex Chaffee
Java Threads
• A thread is not an object
• A thread is a flow of control
• A thread is a series of executed
statements
• A thread is a nested sequence of
method calls
Copyright © 1998 Alex Chaffee
The Thread Object
• A thread is not an object
• A Thread is an object
void start()
– Creates a new thread and makes it runnable
void run()
– The new thread begins its life inside this
method
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Thread Creation Diagram
Object A
Object BThread (extends
Thread)
BThread() {
}
Thread t = new BThread();
t.start();
doMoreStuff();
void start() {
// create thread
}
void run() {
doSomething();
}
Copyright © 1998 Alex Chaffee
Runnable Interface
• A helper to the thread object
• The Thread object’s run() method calls
the Runnable object’s run() method
• Allows threads to run inside any
object, regardless of inheritance
Copyright © 1998 Alex Chaffee
Runnable Example
Talker talker = new Talker();
Thread t = new Thread(talker);
t.start();
--class Talker implements Runnable {
public void run() {
while (true) {
System.out.println(“yakitty yak”);
}
}
}
Copyright © 1998 Alex Chaffee
Blocking Threads
• When reading from a stream, if input is not
available, the thread will block
• Thread is suspended (“blocked”) until I/O is
available
• Allows other threads to automatically
activate
• When I/O available, thread wakes back up
again
– Becomes “runnable”
– Not to be confused with the Runnable interface
Copyright © 1998 Alex Chaffee
Thread Scheduling
• In general, the runnable thread with
the highest priority is active (running)
• Java is priority-preemptive
– If a high-priority thread wakes up, and a
low-priority thread is running
– Then the high-priority thread gets to run
immediately
• Allows on-demand processing
– Efficient use of CPU
Copyright © 1998 Alex Chaffee
Thread Starvation
• If a high priority thread never blocks
• Then all other threads will starve
• Must be clever about thread priority
Copyright © 1998 Alex Chaffee
Thread Priorities: General Strategies
• Threads that have more to do should get
lower priority
• Counterintuitive
• Cut to head of line for short tasks
• Give your I/O-bound threads high
priority
– Wake up, immediately process data, go back
to waiting for I/O
Copyright © 1998 Alex Chaffee
Race Conditions
• Two threads are simultaneously
modifying a single object
• Both threads “race” to store their
value
• In the end, the last one there “wins
the race”
• (Actually, both lose)
Copyright © 1998 Alex Chaffee
Race Condition Example
class Account {
int balance;
public void deposit(int val)
{
int newBal;
newBal = balance + val;
balance = newBal;
}
}
Looks good,
right?
Copyright © 1998 Alex Chaffee
Race Condition Example
class Account {
int balance;
public void deposit(int val)
{
int newBal;
newBal = balance + val;
balance = newBal;
}
}
What if we are
swapped out
right here?
Copyright © 1998 Alex Chaffee
Thread Synchronization
• Language keyword: synchronized
• Takes out a monitor lock on an object
– Exclusive lock for that thread
• If lock is currently unavailable, thread
will block
Copyright © 1998 Alex Chaffee
Thread Synchronization
• Protects access to code, not to data
– Make data members private
– Synchronize accessor methods
• Puts a “force field” around the locked
object so no other threads can enter
• Actually, it only blocks access to other
synchronizing threads
• A rogue thread can "sneak in" and modify a
variable it has access to
Copyright © 1998 Alex Chaffee
Synchronization Example
class Account {
private int balance;
synchronized public void deposit(int val)
{
int newBal;
newBal = balance + val;
balance = newBal;
}
synchronized public void withdraw(int val)
{
int newBal;
newBal = balance - val;
balance = newBal;
}
}
Copyright © 1998 Alex Chaffee
Thread Deadlock
• If two threads are competing for more
than one lock
• Example: bank transfer between two
accounts
– Thread A has a lock on account 1 and wants
to lock account 2
– Thread B has a lock on account 2 and wants
to lock account 1
Copyright © 1998 Alex Chaffee
Avoiding Deadlock
• No universal solution
• Ordered lock acquisition
• Encapsulation (“forcing directionality”)
• Spawn new threads
• Check and back off
• Timeout
• Minimize or remove synchronization
Copyright © 1998 Alex Chaffee
Wait and Notify
• Allows two threads to cooperate
• Based on a single shared lock object
Copyright © 1998 Alex Chaffee
Wait and Notify: Code
• Consumer:
synchronized (lock) {
while (!resourceAvailable()) {
lock.wait();
}
consumeResource();
}
Copyright © 1998 Alex Chaffee
Wait and Notify: Code
• Producer:
produceResource();
synchronized (lock) {
lock.notifyAll();
}
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify Sequence
Lock Object
1. synchronized(lock){
2.
lock.wait();
9.
consumeResource();
10. }
3. produceResource()
4. synchronized(lock)
5.
lock.notify();
6.}
7. Reacquire lock
8. Return from wait()
Consumer
Thread
Producer
Thread
Copyright © 1998 Alex Chaffee
Wait/Notify: Details
• Often the lock object is the resource
itself
• Sometimes the lock object is the
producer thread itself
Copyright © 1998 Alex Chaffee
Wait/Notify: Details
• Must loop on wait(), in case another
thread grabs the resource...
– After you are notified
– Before you acquire the lock and return from
wait()
• Use lock.notifyAll() if there may be
more than one waiting thread
Copyright © 1998 Alex Chaffee
Wait/Notify Example: Blocking Queue
class BlockingQueue extends Queue {
public synchronized Object remove() {
while (isEmpty()) {
wait();
// really this.wait()
}
return super.remove();
}
public synchronized void add(Object o) {
super.add(o);
notifyAll(); // this.notifyAll()
}
}
Copyright © 1998 Alex Chaffee
Part III: Multithreaded Server Design
Copyright © 1998 Alex Chaffee
Different Models of Information Flow
•
•
•
•
Pull model
Push model: One thread per client
Buffer model: One thread per task
Single-thread model
Copyright © 1998 Alex Chaffee
Pull-based Flow
• Consumer asks producer for new
information
• Demand-driven problems
– Examples
• Not appropriate for chat/messagepassing server
Copyright © 1998 Alex Chaffee
Push-based Flow
• Producer triggers sequence of events
• One thread per client
• Receives message, delivers it, waits
Copyright © 1998 Alex Chaffee
Push-based: Pro
• Simple
• Efficient use of CPU
– Events occur only when message available no polling
– No time wasted passing information from
one stage to the next
Copyright © 1998 Alex Chaffee
Push-based: Con
• Much thread swapping
• Slow receivers can stall upcoming
messages from sender
Copyright © 1998 Alex Chaffee
Buffer Model
• Independent threads
• Communicate via buffers
• Producer puts message in buffer,
consumer takes message out
Copyright © 1998 Alex Chaffee
Buffer Model: Pro
• Can separate tasks into separate threads
• Optimize and prioritize per task
– e.g. Processing existing messages is more
important than accepting new connections
• Small number of threads means fewer
chances for deadlock
– Easier to model the system and avoid any
chance of deadlock
Copyright © 1998 Alex Chaffee
Buffer Model: Con
• More complex code
• Must deal with deadlock and
starvation
Copyright © 1998 Alex Chaffee
The One-Thread Model
• All work done by a single thread
• Finishes processing one request before
going to the next
Copyright © 1998 Alex Chaffee
The One-Thread Model (Cont.)
• Pro:
– Suitable for low-traffic scenarios
– Quick to implement
– Avoids any possibility of deadlock or race conditions
• Con
– Unsuitable for high-traffic server
– Hard to implement things like timers or callbacks
Copyright © 1998 Alex Chaffee
Part IV: Implementations
Copyright © 1998 Alex Chaffee
Implementation 1: Push Model Chat
Server
•
•
•
•
•
Listens on a port
Accepts connections
Spawns threads, one per connection
Receives a message from one client
Dispatches it to all clients
Copyright © 1998 Alex Chaffee
Chat Server Objects
• ChatServer
– Main
• ClientHandler
– One per connection
– Threaded
• Dispatcher
– Vector
Copyright © 1998 Alex Chaffee
Chat Server Objects (Diagram)
ChatServer
Dispatcher
ClientHandler
ClientHandler
Socket
Copyright © 1998 Alex Chaffee
Socket
ClientHandler
• Stores information for a single client
Socket incoming;
int id;
Dispatcher dispatcher;
Copyright © 1997 Alex Chaffee
ClientHandler.run()
•
•
•
•
generate filter streams
read a line
send it to the dispatcher
loop
Copyright © 1997 Alex Chaffee
ClientHandler.send()
• sends a single message to the client
• called by the dispatcher
• uses the socket
Copyright © 1997 Alex Chaffee
Dispatcher
• Keeps a list of all client handlers
• Uses java.util.Vector
– synchronized list structure
Copyright © 1997 Alex Chaffee
Dispatcher.dispatch()
• Sends a message to all clients
• Accepts a String and an identifying
integer
• Enumerates down all client handlers
and calls each one’s send() method
Copyright © 1997 Alex Chaffee
ChatServer
•
•
•
•
•
One method: main()
Creates a new ServerSocket
Creates a new Dispatcher
Listens for connections
When connection received, creates a
new ClientHandler and starts a new
thread running inside it
• Adds it to the Dispatcher
Copyright © 1997 Alex Chaffee
Push model: Pro and Con
• Pro
– Straightforward object, thread models
– Efficient CPU usage
• Threads unblock only when message available
• Con
– Must send message to all clients before
receive new one
– No upper limit on number of threads
• May thrash or starve
Copyright © 1998 Alex Chaffee
Implementation 2: Buffer Model
Message Server
Copyright © 1998 Alex Chaffee
Queuing Model
• Producer and consumer communicate
via a buffer
• Buffer is a queue
• First in, first out
Copyright © 1998 Alex Chaffee
Blocking Queue
• If a thread tries to remove an item
from an empty queue
• Then it blocks until an item is placed
inside
• Uses wait/notify technique
Copyright © 1998 Alex Chaffee
JDK 1.2 Collections
•
•
•
•
Set
List
Map
Queue
– Built on top of LinkedList
Copyright © 1998 Alex Chaffee
Basic Object Model
•
•
•
•
•
Acceptor (thread)
Clients (set)
Receiver (thread)
Incoming (queue)
Processor (thread)
Copyright © 1998 Alex Chaffee
Basic Object Model (Fig.)
Acceptor
create
write
Clients
Processor
read
Receiver
read
write
Incoming
Thread
Data
Copyright © 1998 Alex Chaffee
Acceptor Thread
• Listens for incoming connections
• Creates a client object
• Adds it to clients set
Copyright © 1998 Alex Chaffee
Clients Set
• Set of all active clients
• Synchronized methods
– Only one thread at a time can access
Copyright © 1998 Alex Chaffee
Receiver Thread
• Walks down clients set
• For each client, checks if message is
(fully) available
• Reads message and places it on
incoming queue
• Uses non-blocking input streams
Copyright © 1998 Alex Chaffee
Receiver Thread: Alternative
• Alternative: one receiver thread per
client
• Each receiver thread blocks until input
available
• Con:
– Reach number of threads bottleneck
– Scheduling not determined - could starve or
do in weird order
Copyright © 1998 Alex Chaffee
Incoming Queue
• Queue of messages
• Synchronized methods
– Only one thread at a time can access
• Blocking queue
– If a thread tries to get a message from an
empty queue, it blocks until another thread
adds a message
Copyright © 1998 Alex Chaffee
Processor Thread
• While incoming queue is not empty
• Pops message off incoming queue
• Delivers it
Copyright © 1998 Alex Chaffee
Full Object Model
• Checker (thread)
• Problems (list)
• Auditor (thread)
Copyright © 1998 Alex Chaffee
Full Object Model (Fig.)
Auditor
Problems
Checker
Processor
Clients
Acceptor
Incoming
Receiver
Thread
Data
Copyright © 1998 Alex Chaffee
Problems List
• List of clients who have problems
• Receiver can add a client to problems
list if it gets a socket read error
• Processor can add a client to problems
list if it gets a socket write error
• Checker thread periodically
disconnects them all
Copyright © 1998 Alex Chaffee
Checker Thread
• Wakes up every N seconds
• Walks list of clients
• Move to problems list
– If N seconds has passed (“timeout”)
– If socket has a problem
• Walk problems list and actively close
each client
Copyright © 1998 Alex Chaffee
Auditor Thread
• Every N seconds, wakes up
• Collects and prints statistics
Copyright © 1998 Alex Chaffee
Non-Blocking Input Stream
• Needed for receiver to do its job
• If a full message is not available, it
does NOT block
• Instead, it returns null
• Tricky to implement
– Must use available(), mark() and reset()
cleverly
Copyright © 1998 Alex Chaffee
Thread Safety Policies
• All data objects synchronized
• All data objects independent
• Therefore no possibility of deadlock in
the data objects themselves
Copyright © 1998 Alex Chaffee
Thread Safety Policies
• Thread objects must be very careful to avoid
deadlock
• Current model: no multi-object transactions
• Example
– Checker does clients.remove(c) then
problems.add(c)
– NOT
synchronized(clients) { synchronized(problems) {
clients.remove(c); problems.add(c);
}}
– It's OK if it's interrupted between remove and add
Copyright © 1998 Alex Chaffee
Priority: Why?
• In low-volume server, all threads are
normally blocked
• When something needs to happen, it
happens
• That's the beauty of threads!
• So priorities are irrelevant
– If you don’t know what you’re doing, don’t
mess with priorities
Copyright © 1998 Alex Chaffee
Priority: Why?
• In high-volume server, many threads
always have something to do
• Important to do things in the right
order
• Deal with starvation issue
Copyright © 1998 Alex Chaffee
Priorities: General Strategies
• Threads that have more to do get
lower priority
• Counterintuitive
• Cut to head of line for short tasks
Copyright © 1998 Alex Chaffee
Priorities: Queuing Strategies
• Most important: keep incoming queue
empty
• So we don't get bogged down
• Minimize latency for received
messages
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Don't want to add new messages if
we're already busy
– Therefore processor > receiver
• Don't want to accept new connections
if we're already busy
– Therefore processor > acceptor
• Implication:
– Receiver and acceptor may starve?
– No: queue will eventually get cleared
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Want to be able to accept new
connections, even if a currently
connected client has something to say
– Therefore acceptor > receiver
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Checker's job is brief but important
• Needs to wake up, check all clients,
and go back to sleep
• Therefore checker > processor
Copyright © 1998 Alex Chaffee
Priorities: Decisions
• Auditor's job is brief but important
• Need to get accurate statistics -timing is crucial
• Can't wait around for other threads to
finish
• Therefore auditor > checker
Copyright © 1998 Alex Chaffee
Priorities: Conclusion
•
•
•
•
•
•
Auditor - 10
Checker - 8
Processor - 6
Acceptor - 4
Receiver - 2
Goes by twos to accommodate
Windows threading model
Copyright © 1998 Alex Chaffee
Processor Thread Behavior
• Processor thread does:
1. Pop message off incoming queue
2. Deliver it
3. Loop
• Very independent
Copyright © 1998 Alex Chaffee
Processor Thread Blocking
•
•
•
•
Step 1 (pop message) may block
If queue is empty
This is good
Allows other threads to wake up and
fill queue
Copyright © 1998 Alex Chaffee
Processor Thread Blocking
• Step 2 (delivery) may block
– If network is busy
– If OS socket buffer is full
• Bad, because queue is still full
• Increases latency (delivery time) for
already-queued messages
– While new messages are arriving, old
messages should be delivered instead
Copyright © 1998 Alex Chaffee
Multiple Processor Threads
• Solution: have many processor
threads working simultaneously
• If one blocks, another picks up the
next message
Copyright © 1998 Alex Chaffee
Multiple Processors (Fig.)
Acceptor
create
write
Clients
Processor
Processor
Processor
Processor
read
Receiver
read
write
Incoming
Thread
Data
Copyright © 1998 Alex Chaffee
How Many Threads?
1. Constant, tuned per application
2. Dynamic a/k/a thread pooling
Copyright © 1998 Alex Chaffee
Thread Pooling
• Thread pool keeps a certain number of
threads alive
• Other threads ask the thread pool to
perform a task
• If an existing thread is available, that
thread performs the task
• Else a new thread is created, or the
task blocks until one is available
– (depending on policy)
Copyright © 1998 Alex Chaffee
Thread Pooling (Cont.)
• Removes overhead of creating threads
• Allows modular task strategy
Copyright © 1998 Alex Chaffee
Information Flow
• Decoupling threads with buffers is a
good idea
– Maximizes concurrency
– Smooths out bursty differences in rate
• Can cause problems with information
flow
– Producers outpace Consumers
– Too many threads
Copyright © 1998 Alex Chaffee
Limiting Flow
• Bounded Buffers
– Stalls producers if they outpace consumers
• Bounded Thread Pools
– Prevents thrashing
• Rate-adjustment / back-pressure
– Consumer asks Producer to slow down
Copyright © 1998 Alex Chaffee
Another Problem
• One message, multiple recipients, one
processor thread
• If one recipient is in Finland, it will
stall the remaining recipients for that
message
Copyright © 1998 Alex Chaffee
Another Problem: Solutions
• Solution 1: one message queue per
client; thread pool applied to these
delivery queues
• Solution 2: split multi-message into
several single-recipient messages
– Called a Splitter (or Fork or Multicaster)
– Also possible: Routers, Mergers, Collectors,
Combiners, Conduits -- see Lea’s
Concurrent Programming in Java
Copyright © 1998 Alex Chaffee
Debugging Techniques
• Make sure to trap all Throwables in
your Thread object’s run() method
– Otherwise a stray OutOfMemoryError or
NullPointerException will kill your thread
and it will look like deadlock
• Good logging procedure is vital
– I use a global log method that outputs the
name of the thread and the time in addition
to the message
– Name your threads at creation time
Copyright © 1998 Alex Chaffee
Thread Dump
• Control-backslash in Solaris
• Control-break in Windows
– Unfortunately, it scrolls
– There’s no way to redirect standard error
– Oops
Copyright © 1998 Alex Chaffee
Thread Watcher
• Data object
• All tasks (threads) have a pointer to the
master ThreadWatcher
• Put debug code in your tasks to inform
the ThreadWatcher when they change
state
• watcher.setIdle();
• Message m = incoming.remove();
• watcher.set(“Processing message “ + m);
Copyright © 1998 Alex Chaffee
Thread Watcher (cont.)
• ThreadWatcher has its own thread
– Periodically prints status of threads
– Prints an error message if one thread is
blocked for too long
• Alternative to fancy GUI thread
debugger
– Symantec Café
– Compuware DevPartner
– et al.
Copyright © 1998 Alex Chaffee
Bots
• PingBot
– periodically sends a message to itself,
measures the latency
• QuoteBot
– sends random Zippy quotes to mimic a chat
room
Copyright © 1998 Alex Chaffee
Measurements
• Number of bots before sharp latency
increase
• Average latency of single PingBot for N
ZippyBots
Copyright © 1998 Alex Chaffee
Test Architecture
• Need at least three machines
– Server
– Zippy Bot Host
– Ping Bot Host
• So they don’t interfere with one
another
• Should also have some clients on a far
away machine
– If your server will be running on the
Internet
Copyright © 1998 Alex Chaffee
Conclusion
Copyright © 1998 Alex Chaffee
Thanks to
•
•
•
•
•
Eric Malmstrom
Carl Muckenhoupt
Gerry Seidman
Greg Travis
For helping prepare this talk
Copyright © 1998 Alex Chaffee
Where to Get More Information:
Network Programming
• Cornell & Horstmann, Core Java (Sunsoft
press)
• Harold, Java Network Programming
(O’Reilly)
• Hughes et al., Java Network Programming,
(Manning, an imprint of Prentice-Hall)
Copyright © 1998 Alex Chaffee
Where to Get More Information:
Multithreading
– Cornell & Horstmann, Core Java (sunsoft press)
– Oaks and Wong, Java Threads (o’reilly)
– Lea, Concurrent Programming in Java (Addison
Wesley)
– Travis, Using thread pools to increase threading
efficiency, developer.com,
http://www.Developer.Com/journal/techworkshop/06
0498_thread.html
– Travis, Working with the blocking queue,
http://www.Developer.Com/journal/techworkshop/09
1098_blockq.html
Copyright © 1998 Alex Chaffee
Where to Get More Information:
• Web sites
– http://www.jGuru.com/ (Java Training)
– http://www.Developer.com/ (Gamelan)
– http://www.Javaworld.com/ (magazine)
– http://www.Purpletech.com/ (author’s site)
Copyright © 1998 Alex Chaffee