@@ -0,0 +1,81 @@
7+When a TCP transport is created a new server is started which listens on a
8+given port number on the local host. In order to support multiple connections
9+the transport maintains a set of channels, one per connection, represented as a
10+pair of a `Chan [ByteString]` and a list of pairs `(ThreadId, Socket)` of
11+threads that are listening on this channel. A source end then corresponds to a
12+hostname (the hostname used by clients to identity the local host), port
13+number, and channel ID; a receive end simply corresponds to a channel ID.
15+When `mkTransport` creates a new transport it spawns a thread that listens for
16+incoming connections, running `procConnections` (see below). The set of
17+channels (connections) associated with the transport is initialized to be
20+`newConnectionWith` creates a new channel and add its to the transport channel
21+map (with an empty list of associated threads).
23+To serialize the source end we encode the triple of the local host name, port
24+number, and channel ID, and to deserialize we just decode the same triple
25+(deserialize does not need any other properties of the TCP transport).
27+To connect to the source end we create a new socket, connect to the server at
28+the IP address specified in the TCPConfig, and send the channel number over the
29+connection. Then to `closeSourceEnd` we simply close the socket, and to send a
30+bunch of byte strings we output them on the socket.
32+To receive from the target end we just read from the channel associated with
33+the target end. To `closeTargetEnd` we find kill all threads associated with
34+the channel and close their sockets.
36+When somebody connects to server (running `procConnections`), he first sends a
37+channel ID. `procConnections` then spawns a new thread running
38+`procMessages` which listens for bytestrings on the socket and output them on
39+the specified channel. The ID of this new thread (and the socket it uses) are
40+added to the channel map of the transport.
42+`closeTransport` kills the server thread and all threads that were listening on
43+the channels associated with the transport, and closes all associated sockets.
48+A series of benchmarks has shown that
50+* The use of `-threaded` triples the latency.
52+* Prepending a header to messages has a negligible effect on latency, even when
53+ sending very small packets. However, the way that we turn the length from an
54+ `Int32` to a `ByteString` _does_ have a significant impact; in particular,
55+ using `Data.Serialize` is very slow (and using Blaze.ByteString not much
56+ better). This is fast:
58+ foreign import ccall unsafe "htonl" htonl :: CInt -> CInt
60+ encodeLength :: Int32 -> IO ByteString
63+ pokeByteOff p 0 (htonl (fromIntegral i32))
65+* We do not need to use `blaze-builder` or related;
66+ `Network.Socket.Bytestring.sendMany` uses vectored I/O. On the client side
67+ doing a single `recv` to try and read the message header and message, rather
68+ one to read the header and one to read the payload improves latency, but only
71+* Indirection through an `MVar` or a `Chan` does not have an observable effect
74+* When two nodes _A_ and _B_ communicate, latency is worse when they
75+ communicate over two pairs of sockets (used unidirectionally) rather than one
76+ pair (used bidirectionally) by about 20%. This is not improved by using
77+ `TCP_NODELAY`, and might be because [acknowledgements cannot piggyback with
78+ payload][1] this way. It might thus be worthwhile to try and reuse TCP
79+ connections (or use UDP).
81+[1]: http://lists.freebsd.org/pipermail/freebsd-hackers/2009-March/028006.html "2 uni-directional TCP connection good?"