HTTP-Tester- A web appliance
testing tool
Kiran Komaravolu
Dept of Computer Science and Engg
Lehigh University
What is the need to test Web
Appliances

Need to use best components.



Compare performance of different web devices.
Verify if a device is performs as per
specifications.
Choosing between different service providers.
Related Work





The Ethereal network analyser
(www.ethereal.com).
TCPDUMP/PCAP (www.tcpdump.org).
Httpflow (Grevnin and Davison)
Simultaneous proxy evaluation. (Davison).
Medusa Proxy. (Koletsou and Voelker).
Http-Tester




Replays captured requests to an existing server
to one that is being evaluated.
No active deployment needed.
Does not affect existing services.
Can be used to evaluate any Web appliance
running HTTP.
HTTP-Tester

Topology setup
HTTP-Tester

Components

The Sniffer.


Request Replicator.


Logs all packets onto a packet database.
Replays all requests to the “tested device”
The Post Processor.

Reads the packet logs, extracts HTTP requests and
responses and correlates them.
HTTP

HTTP Message Structure

A command line. (GET, POST, OK etc.)


GET http://www.google.com / HTTP 1.0
Zero or more header lines.

Content-Type: text/html; charset=iso-8859-1
Control Return + Line Feed.
 Optional Message Body


Response message
HTTP Transactions.

HTTP 1.0
No persistent connections.
 A new connection is opened up for every request
made, and is closed down after response sent.


HTTP 1.1
Supports Persistent Connections.
 Client may make a series of requests on a
connection. Server returns responses in same order.
 Supports chunk encoding.

The Sniffer



Blindly captures all relevant network packets and
logs them.
The pcap packet capture library has been used
to read packets off the wire.
Berkeley DB package has been used to log the
packets into a ‘db’ database.

The job of efficient memory and disk usage has been
delegated to DB.
Request Replicator




Replays all requests flowing into original device
onto the tested device.
Works in parallel with the sniffer module.
Looks for TCP SYN packets and determines the
beginning of a request flow. And opens up a
new connection to the second device.
The payload every next packet sent to on this
connection is sent to the second device.
Request Replicator.


This way the second server sees the same
workload as the first one.
The replies sent by the server are ignored. They
are flushed out as the socket buffers overflow.
The sniffer module though logs the TCP
packets.
Post Processing.



Extracts HTTP requests/responses from the
packet logs.
Every request is mapped onto its response. The
same request response pair on the second
connection is also identified.
Httpflow routines used to extract HTTP
requests from packet stream.
Identifying Requests.




We know which way requests flow.
Assumption: First TCP packet of every request
has one of GET, HEAD,OPTIONS, DELETE,
POST, PUT.
End of headers is determined by a CRLF.
content-length field is specified in case of
PUT/POST requests.
Identifying responses


Response headers extrated by looking for a
CRLF.
End of Response message determined in three
ways:
Server sends a FIN or RST packet.
 Response headers contain a “content-length” field.
 Response is chunk encoded. Each chunk has its own
length. Zero length chunk indicates end of stream.

Correlating requests and responses

HTTP 1.0