The files:
client.cpp
server.cpp
utils.cpp
utils.h
makefile

A modified HTTP protocol

The purpose of this study was to modify the HTTP protocol in such a way that a connection can be maintained between client and server, so that the socket overhead is not incurred on every request.

By creating a server which sends out blocks of meaningless data and a client that times the server's responses, we set out to determine what advantages our modified protocol could give us. Our results were very surprising.

The surprise came as a result of our variation of the maximum number of requests the server would handle before closing the connection. We considered the case in which the server will handle only one connection to be equivalent to the basic protocol in which one request is made over one socket.

This chart details the average time a 40-request, 1000 bytes/request transaction took from start to finish for each of eight values for the maximum number of requests allowed per connection.

Requests per connection 1 2 5 10 20 40 100 1000
Total Time (seconds) 3.344933 4.175920 6.644457 7.421635 7.911480 8.205384 8.304167 8.321051
Time in socket overhead 1.576493 0.772862 0.309357 0.155422 0.077426 0.038416 0.038957 0.038667
Time in data transfer 1.714552 3.348379 6.281412 7.212502 7.776691 8.083009 8.214653 8.231001
% time in socket overhead 47.13% 18.51% 4.66% 2.09% 0.98% 0.47% 0.47% 0.46%
% time in data transfer 51.26% 80.18% 94.54% 97.18% 98.30% 98.51% 98.92%98.92%

As expected, the absolute and percentage time spent performing socket overhead go down as more requests are allowed per connection. But contrary to our expectations, the total time for the transaction actually increases! The same amount of data is being transferred in each case, so we find it difficult to explain this property.

The client is unaware of how many requests it is allowed, and hence knows it must start a new connection only when it finds the old socket closed, so there is no optimization in the code for establishing a new connection -- in fact, it is rather inefficient.

Our best theory is that it has something to do with caching -- perhaps the kernel implementation of sockets favors the temporal locality of a socket that is addressed very often, or perhaps the large false-entity-body array has more time to grow stale in cache as more messages are sent.


Other results of the modified formula behave rather more as we expect. Here are five times for 24 requests to a server which allows 6 per connection and returns 1000-byte blocks:

 time 1time 2time 3 time 4 time 5 mean time mean % std. dev variance
total 4.320664 4.281201 4.132568 4.181701 4.251109 4.233449   0.07591676 0.00576335
socket 0.159324 0.154431 0.155289 0.155766 0.153869 0.155736 3.68% 0.00213677 0.00000457
data 3.963454 4.079005 3.923029 3.973342 4.044535 3.996673 94.41% 0.06352284 0.00403515

The variance is quite low, which suggests to us that a small number of timing samples gives us a reasonable representation.

Another odd result comes our way, however, when we vary the entity-body size. Using a 40-request client and a server which allows each of 1 and 20 requests, we get the following results:

  1 byte
1 max
1 byte
20 max
10 bytes
1 max
10 bytes
20 max
100 bytes
1 max
100 bytes
20 max
1000 bytes
1 max
1000 bytes
20 max
10000 bytes
1 max
10000 bytes
20 max
100000 bytes
1 max
100000 bytes
20 max
total 3.281748 8.025768 3.262640 7.848265 3.482473 8.063645 3.324586 7.893638 8.317987 4.037560 18.350649 8.414878
socket 1.597931 0.078072 1.563907 0.078133 1.779464 0.077380 1.554903 0.077083 1.551568 0.077228 1.554517 0.077485
data 1.632528 7.889292 1.640133 7.712802 1.644048 7.922992 1.715597 7.757549 6.711609 3.903344 16.737920 8.286843

As you can see, the inversion noted earlier holds through 1000 bytes, but at an entity size of 10000 bytes, the relation suddenly flips so that larger number of requests per connection result in shorter total times -- as we had originally predicted. It would appear that the relationship should converge somewhere between 2000 and 5000 bytes, but we were unable to discern the equation from these numbers. (If we could derive the equation from our code structure, we would know why it does this.) Given enough time, we could find the midpoint by binary searching, but that would not explain much.

The socket overhead times are on the order of 20 times less for the 20-request connections, as expected. But why is the data transfer time longer?


The protocol itself is a stripped-down version of HTTP/1.0. The client has no idea how many connections it is allowed, and the server does not tell it, so no additional information is exchanged. The client generates a valid HTTP 1.0 request, but the server ignores its contents completely. Similarly, the response of the server retains most HTTP information, but it is not used by the client.

The client accepts at runtime an integer prescribing the number of requests it is to make. The server accepts the entity size, the number of connections allowed concurrently, and the length of the queue of requests it can handle.


Some miscellaneous notes.

The same false entity body is used for all server responses; it is a large block of the character 'x'. A new response header is generated for each one, though, for realistic overhead. The client similarly generates a great many identical requests to simulate a realistic amount of traffic.

For testing purposes, we ran the server on everest.cs.umbc.edu, just to get a little physical distance.

We don't know a good way to test the server serving 5 or 10 clients concurrently. Our experiments with two suggest the obvious -- each is slowed down, but only a little.

We tried to correct for network and CPU loads by running the tests as close in time as possible, and at a low-usage hour.

All actions are timed using gettimeofday() on the client side.


Summary and Conclusions

The two main variables -- number of requests allowed per connection and size of entity bodies -- define the running time of transactions in a manner that seems to be far from linear. At one end, large values for the size of the entity body seem to have the effect that more requests per connection can speed up transactions, but the reverse holds true for small values of the entity body size.

More study is needed to determine the exact nature of this relationship.