SWE 622- Distributed Systems
Project Phase III
Eric Barnes, David Chang, David Nelson
Fisayo Oluwadiya, Xiang Shen
Initial Design
Trader
Exchange
SwingWorker
start
update
JFrameExchangeGUI
JFrameTraderGUI
upd
query
start
start
ExchangeServer
Sqlite
DB
initDB
star
TraderClient
ne
Exchange
e
qu
,
ry
m
co
m
it
t
se
start
ate
SwingWorker
ll,
bu
y
w
one TCP connection
st
ar
Trader
t
business msg
output queue
tim
em
TraderOutput
TraderWorker
start
sg
t, m
ar ti
st n c
sy
start
Monitor
e
Processor
TraderInput
Listener
input queue
TraderInput
Exchange waits for new connections.
Spawn new threads TraderOutput/
Input for each new connection. It also
tim
e
...
...
business msg
processed queue
Listener
g
ms
start
creates all the queues.
TraderOutput
TraderWorker
output queue
start, sync time
business msg
some other exchange
some other trader
Final Design
Trader
Exchange
SwingWorker
start
update
JFrameExchangeGUI
JFrameTraderGUI
upd
query
start
start
ExchangeServer
Sqlite
DB
initDB
TraderClient
ne
Exchange
e
qu
ry
,
m
co
m
it
tim
ep
y
sc
he
Trader
du
le
start,
business msg
ExchangeConnect
ionManager
start
g
Processor
SyncClock
k
ms
oc
ve
cl
ali
nc
e/k
e
bu
sy
start
TraderOutput
SwingWorker
ll ,
w
one TCP connection
output queue
t
se
start
start
star
ate
TraderInput
Listener
input queue
TraderConnection
Manager
TraderInput
ti
Exchange waits for new connections.
Spawn a new thread
TraderConnectionManager, which starts
and manages threads TraderOutput/Input
and CheckTimeout for each new
connection.
/k
me
ee
pa
liv
s
em
...
...
business msg
processed queue
Listener
g
start
TraderOutput
output queue
ExchangeConnect
ionManager
start, sync time
business msg
some other exchange
some other trader
Initial Class Diagram
Final Class Diagram
Class Exchange init() method
public void init() throws Exception {
_serverSocket = null;
try {
_serverSocket = new ServerSocket(_exchangePort);
} catch (IOException e) {
log.error("Could not listen on port:" + _exchangePort);
System.exit(1);
}
while (true) {
Socket clientSocket = null;
try {
log.info("WAITING");
clientSocket = _serverSocket.accept();
log.info("ACCEPTING");
} catch (IOException e) {
log.error("Accept failed.");
System.exit(1);
}
TraderConnectionManager manager = new
TraderConnectionManager(clientSocket, this);
traderManagers.add(manager);
}
}
Class TraderConnectionManager constructor
public TraderConnectionManager(Socket clientSocket, Exchange _exchange)
{
this.exchange = _exchange;
this.outMsgQueue = new ArrayBlockingQueue<String>(50);
try
{
out = new PrintWriter(clientSocket.getOutputStream(), true);
in = new BufferedReader(new
InputStreamReader(clientSocket.getInputStream()));
}
catch (IOException e) { ...... }
Thread tin = new Thread(new TraderInput(clientSocket, outMsgQueue));
tin.start();
Thread tout = new Thread(new TraderOutput(clientSocket, outMsgQueue));
tout.start();
//set last keepalive so that we don't time out right away
lastKeepaliveReceived = System.currentTimeMillis();
//schedule keepalives
keepaliveTimer = new Timer("keepallive");
keepaliveTimer.scheduleAtFixedRate(new CheckTimeout(),
KEEPALIVE_INTERVAL, KEEPALIVE_INTERVAL);
}
Class ExchangeConnectionManager run() method
public void run() {
boolean connectedToExchange = initSocket(exchangAddr, port);
if (connectedToExchange) {
// set last keepalive so that we don't time out right away
lastKeepaliveReceived = System.currentTimeMillis();
// schedule keepalives
keepaliveTimer = new Timer("keepalive-" + _timeMsgQueue);
keepaliveTimer.scheduleAtFixedRate(new SendKeepalive(),
KEEPALIVE_INTERVAL, KEEPALIVE_INTERVAL);
// schedule clock syncs
clockSyncTimer = new Timer("clockSync-" + _timeMsgQueue);
clockSyncTimer.scheduleAtFixedRate(new SyncClock(), 0,
CLOCK_OFFSET_CHECK_INTERVAL);
}
}
Note:
KEEPALIVE_INTERVAL = 5000;
MAX_MISSED_KEEPALIVES = 2;
//check every 1 hour (1000 ms/sec * 60 sec/min * 60 min/hour
CLOCK_OFFSET_CHECK_INTERVAL = 1000 * 60 * 60;
Just in case the echo messages are out of order
TradeTransaction t = new TradeTransaction(fromServer);
boolean outOfOrder = t.getMessageID() < newestMessageProcessed;
//if the message was out of order, flag it, but don't set it as the last ID recieved to
// ensure that subsequent out of order messages are also flagged. e.g. if IDs arrive
// in the order 5 3 4, both 3 and 4 should be flagged
if(outOfOrder)
{
log.error("Out of order");
t.setOutOfOrder(true);
}
else
{
newestMessageProcessed = t.getMessageID();
log.info("Message sequence:" + newestMessageProcessed);
}
t.setStale(_clockDelta + System.currentTimeMillis());
processedTAQueue.put(t);
Just in case the echo messages are out of order
TradeTransaction t = new TradeTransaction(fromServer);
boolean outOfOrder = t.getMessageID() < newestMessageProcessed;
//if the message was out of order, flag it, but don't set it as the last ID recieved to
// ensure that subsequent out of order messages are also flagged. e.g. if IDs arrive
// in the order 5 3 4, both 3 and 4 should be flagged
if(outOfOrder)
{
log.error("Out of order");
t.setOutOfOrder(true);
}
else
{
newestMessageProcessed = t.getMessageID();
log.info("Message sequence:" + newestMessageProcessed);
}
t.setStale(_clockDelta + System.currentTimeMillis());
processedTAQueue.put(t);
How to figure how to the clock delta between exchange and trader
private class SyncClock extends TimerTask {
@Override
public void run() {
long requestTime;
long receiveTime;
long oneWay;
long exchangeTime;
long totalDelta = 0;
int tryTimes = 3;
String timeStr = "";
for (int i = 0; i < tryTimes; i++) {
requestTime = System.currentTimeMillis();
log.info("requestTime " + i + "=" + requestTime);
requestTime(requestTime);
try {
timeStr = _timeMsgQueue.take();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
exchangeTime = Long.parseLong(timeStr.replace("time:", ""));
log.info("exchangeTime " + i + "=" + exchangeTime);
receiveTime = System.currentTimeMillis();
log.info("receiveTime " + i + "=" + receiveTime);
oneWay = (receiveTime - requestTime) / 2;
totalDelta += exchangeTime + oneWay - receiveTime;
}
_clockDelta = totalDelta / tryTimes;
log.info("clockDelta=" + _clockDelta);
}
}
Source Lines Of Code
All SLOC:
SLOC
Directory
SLOC-by-Language (Sorted)
435
gui
java=435
375
exchange
java=375
368
trader
java=368
147
common
java=147
Totals grouped by language (dominant language first):
java:
1325 (100.00%)
Reused SLOC (GUI code provided by Prof. Sousa):
SLOC
Directory
SLOC-by-Language (Sorted)
311
gui
java=311
Totals grouped by language (dominant language first):
java:
311 (100.00%)
Total New SLOC:
SLOC
Directory
1014
src
SLOC-by-Language (Sorted)
java=1014
Totals grouped by language (dominant language first):
java:
1014 (100.00%)
Total Physical Source Lines of Code (SLOC)
=
Development Effort Estimate, Person-Years (Person-Months) =
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)
=
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Total Estimated Cost to Develop
=
(average salary = $56,286/year, overhead = 2.40).
SLOCCount, Copyright (C) 2001-2004 David A. Wheeler
1,014
0.20 (2.44)
0.29 (3.51)
$ 27,415
Effort spent for the project (man-hour)
• Effort spent in design:
120
• Effort spent in design coding:
100
• Effort spent in design integrating:
40
• Effort spent in design experiments:
60
• Total effort:
320
Note: All of them are estimates
Testing Environment
•
SUSE Linux 10 SP1 (i586)
–
–
–
–
•
Solaris 10
–
–
–
–
•
Kernel:
Memory:
CPU:
Java:
SunOS mason 5.10 Generic_138888-01 sun4u sparc SUNW,Sun-Fire-V890
32GB
8 x Sparc 1350 MHz
java version "1.6.0_11“
Microsoft Windows XP SP3
–
•
Kernel: 2.6.16.46-0.12-bigsmp #1 SMP Thu May 17 14:00:09 UTC 2007
Memory: 4GB
CPU:
2 x Intel(R) 3.40GHz
Java:
java version "1.6.0_12“
Java:
java version "1.6.0_01“
Microsoft Windows Vista SP1
–
Java:
java version "1.6.0_12“
Testing Network and network latency
LAN:
•
•
Ping < 1ms
Direct connected by a network switch
WAN:
•
10ms < Ping < 30ms
Pinging mason.gmu.edu [129.174.1.13] with 32 bytes of data:
Reply from 129.174.1.13: bytes=32 time=13ms TTL=246
Reply from 129.174.1.13: bytes=32 time=14ms TTL=246
Reply from 129.174.1.13: bytes=32 time=19ms TTL=246
Reply from 129.174.1.13: bytes=32 time=12ms TTL=246
Ping statistics for 129.174.1.13:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 12ms, Maximum = 19ms, Average = 14ms
•
Many network hops
Tracing route to mason.gmu.edu [129.174.1.13]
over a maximum of 30 hops:
1
2
3
1 ms
10 ms
12 ms
1 ms
9 ms
9 ms
1 ms
8 ms
9 ms
192.168.0.1
10.3.224.1
ip72-219-223-161.dc.dc.cox.net [72.219.223.161]
4
5
6
9 ms
13 ms
13 ms
10 ms
10 ms
64 ms
9 ms
10 ms
11 ms
mrfddsrj01-ge110.rd.dc.cox.net [68.100.0.161]
ashbbbrj01-ae0.0.r2.as.cox.net [68.1.0.220]
ae-11-69.car1.Washington1.Level3.net [4.68.17.3]
7
8
9
10
16 ms
*
*
13 ms
13 ms
*
*
15 ms
12 ms
*
*
12 ms
GEORGE-MASO.car1.Washington1.Level3.net [4.79.204.66]
Request timed out.
Request timed out.
mason.gmu.edu [129.174.1.13]
Latency between starting to send an order and proceeding with other activities
msec
180
160
140
Avg=71.55
120
100
80
60
40
Avg=51.72
20
0
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
Orders
Latency in a LAN
Lantency in a WAN
LAN: Ping statistics for 172.16.104.231: Approximate round trip times in milli-seconds
Minimum = 0ms, Maximum = 0ms, Average = 0ms
WAN: Ping statistics for 129.174.1.13 (mason.gmu.edu): Approximate round trip times in milli-seconds:
Minimum = 17ms, Maximum = 20ms, Average = 18ms
Latency between issuing an order and the last trader receiving it’s echo
msec
180
160
140
120
Avg=86.7
100
80
60
40
20
0
1
2
3
4
5
6
7
8
9
10
11
Orders
Latency
12
13
14
15
16
17
18
19
20
Latency between a trader failure and the last exchange stopping echoing orders
•
Between each Exchange and Trader there is a TCP connection
•
If the Trader fails, each Exchange will know immediately
•
–
Read failed error
–
Depends on the network latency
If the network also fails, each Exchange will disconnect the failed Trader after the time-out
–
MAX_MISSED_KEEPALIVES* KEEPALIVE_INTERVAL
In the log file:
ERROR
INFO
INFO
INFO
INFO
TraderConnectionManager:130 - Read failed. Client seems gone or the exchange is closing.
TraderConnectionManager:68 - Disconnecting
TraderConnectionManager:85 - disconnecting trader
TraderConnectionManager:137 - EXIT INPUT
TraderConnectionManager:177 - EXIT OUTPUT
Storage footprint for one exchange
kB
350000
300000
250000
200000
150000
100000
50000
0
1
2
3
4
5
6
7
8
9
10
Trader Num.
Resident set size
Total VM size
11
12
13
14
15
Clock Skew
•
Linux and the Sun box all have ntpd running
•
All Windows boxes all have the Windows time service running
•
Clock Skew we observed on the Linux box (from the NTP log):
……
25
25
25
25
25
25
25
……
•
Apr
Apr
Apr
Apr
Apr
Apr
Apr
04:56:57
05:01:13
05:02:18
05:28:11
05:32:28
05:33:31
05:47:29
ntpd[3151]:
ntpd[3151]:
ntpd[3151]:
ntpd[3151]:
ntpd[3151]:
ntpd[3151]:
ntpd[3151]:
time reset -0.780931 s
synchronized to LOCAL(0), stratum 10
synchronized to 172.16.200.1, stratum 1
time reset +0.504970 s
synchronized to LOCAL(0), stratum 10
synchronized to 172.16.200.1, stratum 1
time reset +0.241215 s
Testing Scenario:
–
Stop ntpd on Linux
–
Change time, for example: date +%T -s "17:07“
–
Test
Time difference, Clock delta, Transaction time
700
600
500
400
300
200
100
0
1
2
3
4
5
6
7
8
9
Time difference
Clock Delta
Transaction how long ago
Time difference
10