Accountable Virtual Machines
Andreas Haeberlen
University of Pennsylvania
Paarijaat Aditya
Rodrigo Rodrigues
Peter Druschel
Max Planck Institute for Software Systems (MPI-SWS)
Max
Planck
Institute
for
Software Systems
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
1
Scenario: Multiplayer game
I'd like
to play
a game
Charlie
Network
Bob
Alice

Alice decides to play a game of Counterstrike
with Bob and Charlie
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
2
What Alice sees
Movie
Alice
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
3
Could Bob be cheating?
Charlie
Ammo
35
36
37
Network
Bob
Alice

In Counterstrike, ammunition is local state


© 2010 Andreas Haeberlen
Bob can manipulate counter and prevent it from decrementing
Such cheats (and many others) do exist, and are being used
OSDI (October 4, 2010)
4
This talk is not (just) about cheating!
Software
Alice

Bob
Cheating is a serious problem in itself


Network
Multi-billion-dollar industry
But we address a more general problem:



© 2010 Andreas Haeberlen
Alice relies on software that runs on a third-party machine
Examples: Competitive system (auction), federated system...
How does Alice know if the software running as intended?
OSDI (October 4, 2010)
5
Goal: Accountability
Software
Alice

Bob
We want Alice to be able to



Network
Detect when the remote machine is faulty
Obtain evidence of the fault that would convince a third party
Challenges:

Alice and Bob may not trust each other


Neither Alice nor Bob may understand how the software works

© 2010 Andreas Haeberlen
Possibility of intentional misbehavior (example: cheating)
Binary only - no specification of the correct behavior
OSDI (October 4, 2010)
6
Outline

Problem: Detecting faults on remote machines

Example: Cheating in multiplayer games

Solution: Accountable Virtual Machines

Evaluation


NEXT
Using earlier example (cheating in Counterstrike)
Summary
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
7
Overview
Virtual
machine
image
Accountable
Virtual Machine
(AVM)
Accountable
Virtual Machine
Monitor (AVMM)
AVM
Log
AVMM
Network
Alice must
Bob
Alice
trust her own
What if Bob
reference
How
caninAlice
find
this
manipulates
 Bob runs Alice's software image
an
AVM
image if it exists?
execution,
the log?
 AVM maintains a log of network in-/outputs

Alice can check this log with a reference image


© 2010 Andreas Haeberlen
AVM correct: Reference image can produce same network
outputs when started in same state and given same inputs
AVM faulty: Otherwise
OSDI (October 4, 2010)
8
Tamper-evident logging
AVM
474: SEND(Alice, Firing)
473: SEND(Charlie, Got ammo)
AVMM
Firing

471: SEND(Charlie, Moving left)
Moving right
...
Message log is tamper-evident [SOSP'07]



472: RECV(Alice, Got medipack)
Log is structured as a hash chain
Messages contain signed authenticators
Result: Alice can either...


© 2010 Andreas Haeberlen
... detect that the log has been tampered with, or
... get a complete log with all the observable messages
OSDI (October 4, 2010)
9
Execution logging
AVM
AVMM


474:
474: SEND(Alice,
SEND(Alice, Firing)
Firing)
473:
button clicked
473: Mouse
SEND(Charlie,
Got ammo)
472: SEND(Charlie, Got ammo)
472: RECV(Alice,
RECV(Alice, Got
Got medipack)
medipack)
471:
470:
network interrupt
471: Got
SEND(Charlie,
Moving left)
469: SEND(Charlie, Moving left)
...
How does Alice know whether the log matches
a correct execution of her software image?
Idea: AVMM can specify an execution



© 2010 Andreas Haeberlen
AVMM additionally logs all nondeterministic inputs
AVM correct: Can replay inputs to get execution
AVM faulty: Replay inevitably (!) fails
OSDI (October 4, 2010)
10
SEND(Alice, Firing)
SEND(Alice, Firing)
SEND(Alice, Firing)
Mouse button clicked
SEND(Alice, Got medipack)
Mouse moved left
AVM
371:
370:
369:
368:
367:
366:
373: SEND(Alice, Firing)
372: SEND(Alice, Firing)
371: SEND(Alice, Firing)
370: SEND(Alice, Firing)
369: SEND(Alice, Firing)
368: Mouse button clicked
367: SEND(Alice, Got medipack)
366: Mouse moved left
...
Evidence
Modification
AVM
Auditing and replay
AVMM
AVMM
Network
Bob
Alice
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
11
AVM properties

Strong accountability




If it runs
in a VM, it
will work
Works for arbitrary, unmodified binaries


Detects faults
Produces evidence
No false positives
Nondeterministic events can be captured by AVM Monitor
Alice does not have to trust Bob, the AVMM,
or any software that runs on Bob's machine


© 2010 Andreas Haeberlen
If Bob tampers with the log, Alice can detect this
If Bob's AVM is faulty, ANY log Bob could produce would
inevitably cause a divergence during replay
OSDI (October 4, 2010)
12
Outline

Problem: Detecting faults on remote machines

Example: Cheating in multiplayer games

Solution: Accountable Virtual Machines

Evaluation


NEXT
Using earlier example (cheating in Counterstrike)
Summary
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
13
Methodology

We built a prototype AVMM



Based on logging/replay engine in VMware Workstation 6.5.1
Extended with tamper-evident logging and auditing
Evaluation: Cheat detection in games




© 2010 Andreas Haeberlen
Setup models competition / LAN party
Three players playing Counterstrike 1.6
Nehalem machines (i7 860)
Windows XP SP3
OSDI (October 4, 2010)
14
Evaluation topics


Effectiveness against real cheats
Overhead








Disk space (for the log)
Time (auditing, replay)
Network bandwidth (for authenticators) Please refer to
Computation (signatures)
the paper for
Latency (signatures)
additional results!
Impact on game performance
Online auditing
Spot checking tradeoffs

© 2010 Andreas Haeberlen
Using a different application: MySQL on Linux
OSDI (October 4, 2010)
15
AVMs can detect real cheats
98: RECV(Alice, Hit)
Missed)
Fire@(3,9))
97: SEND(Alice, Fire@(2,7))
96: Mouse button clicked
95: Interrupt received
94: RECV(Alice, Jumping)
...
BC=53
BC=59
BC=52
BC=54
BC=47
BC=49
BC=44
BC=37
...
EIP=0xb382
EIP=0x861e
EIP=0x3633
EIP=0x2d16
EIP=0xc490
EIP=0xc43e
EIP=0x6771
EIP=0x570f
...
AVM
Event timing (for replay)
AVMM
Bob's log

If the cheat needs to be installed in the AVM
to be effective, AVM can trivially detect it


© 2010 Andreas Haeberlen
Reason: Event timing + control flow change
Examined real 26 cheats from the Internet; all detectable
OSDI (October 4, 2010)
16
AVMs can detect real cheats
?
?
?


?
BC=
BC=
BC=
BC=
BC=53
BC=52
BC=
BC=47
BC=44
BC=
BC=37
...
?
?
?
EIP=
EIP=
EIP=
EIP=
EIP=0xb382
EIP=0x3633
EIP=
EIP=0xc490
EIP=0x6771
EIP=
EIP=0x570f
...
?
?
?
AVM
99: RECV(Alice, Hit)
98: SEND(Alice, Fire@(2,7))
97: Mouse button clicked
96: RECV(Alice,
Mouse moveMissed)
right 1 inch
95: SEND(Alice, Fire@(3,9))
94: Mouse button
clicked
move up
1 inch
93: Interrupt received
92: RECV(Alice, Jumping)
...
AVMM
?
Couldn't cheaters adapt their cheats?
There are three types of cheats:
1.
2.
3.
© 2010 Andreas Haeberlen
Detection impossible (Example: Collusion)
Detection not guaranteed, but evasion technically difficult
Detection guaranteed (15% of the cheats in our sample)
OSDI (October 4, 2010)
17
Impact on frame rate
Average frame rate
200
158fps
-13%
150
100
No fps cap
Window mode
800x600
Softw. rendering
50
0

Different machines
with different players
-11%
Bare
hardware
VMware
VMware
(no logging) (logging)
AVMM
(no crypto)
AVMM
Frame rate is ~13% lower than on bare hw


© 2010 Andreas Haeberlen
137fps is still a lot! 60--80fps generally recommended
11% due to logging; additional cost for accountability is small
OSDI (October 4, 2010)
18
Average log growth (MB/minute)
Cost of auditing

12
Added by
accountability
10
8
~8 MB
per minute
2.47 MB
per minute
(compressed)
6
4
2
0
VMware
AVMM
When auditing a player after a one-hour game,


© 2010 Andreas Haeberlen
How big is the log we have to download? 148 MB
How much time is needed for replay?
~ 1 hour
OSDI (October 4, 2010)
19
Replay
Replay
Logging
Game
Online auditing
Average frame rate
200
150
Alice
100
50
Bob
0

No online
auditing
One audit
per player
Charlie
Two audits
per player
Idea: Stream logs to auditors during the game


© 2010 Andreas Haeberlen
Result: Detection within seconds after fault occurs
Replay can utilize unused cores; frame rate penalty is low
OSDI (October 4, 2010)
20
Summary

Accountable Virtual Machines (AVMs) offer
strong accountability for unmodified binaries



Useful when relying on software executing on remote
machines: Federated system, multiplayer games, ...
No trusted components required
AVMs are practical


Prototype implementation based on VMware Workstation
Evaluation: Cheat detection in Counterstrike
Questions?
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
21
Thank you!
Our enthusiastic Counterstrike volunteers
© 2010 Andreas Haeberlen
OSDI (October 4, 2010)
22