a962871 · distributed-process

git.iko.soy/distributed-process

Start work on the updated N.T failure semantics

commita962871ce699b03b860fdcd5fcc436c02419a3d9 authorEdsko de Vries <edsko@edsko.net> dateFri 24 Aug 2012, 17:22 (14 years ago) parent686ad9e

13 files changed+87−45

compare with master

unifiedsplit

distributed-process-simplelocalnet/ChangeLog+4 distributed-process-simplelocalnet/distributed-process-simplelocalnet.cabal+5−5 distributed-process/ChangeLog+5 distributed-process/distributed-process.cabal+6−6 distributed-process/src/Control/Distributed/Process/Node.hs+7−6 network-transport-tcp/ChangeLog+4 network-transport-tcp/network-transport-tcp.cabal+3−3 network-transport-tcp/src/Network/Transport/TCP.hs+2−5 network-transport-tcp/tests/TestTCP.hs+11−11 network-transport-tcp/tests/TestTransport.hs+4−4 network-transport/ChangeLog+4 network-transport/network-transport.cabal+1−1 network-transport/src/Network/Transport.hs+31−4

distributed-process-simplelocalnet/ChangeLog+4

@@ -1,3 +1,7 @@

1+<<date>> Edsko de Vries <edsko@well-typed.com> 0.2.0.6

3+* Use new version of network-transport

152012-08-22 Edsko de Vries <edsko@well-typed.com> 0.2.0.5

37* Don't assume slaves are still alive in findSlaves

distributed-process-simplelocalnet/distributed-process-simplelocalnet.cabal+5−5

@@ -1,5 +1,5 @@

11Name: distributed-process-simplelocalnet

2−Version: 0.2.0.5

2+Version: 0.2.0.6

33Cabal-Version: >=1.8

44Build-Type: Simple

55License: BSD3

@@ -32,8 +32,8 @@ Library

3232 binary >= 0.5 && < 0.6,

3333 containers >= 0.4 && < 0.6,

3434 transformers >= 0.2 && < 0.4,

35− network-transport >= 0.2 && < 0.3,

36− network-transport-tcp >= 0.2 && < 0.3,

35+ network-transport >= 0.3 && < 0.4,

36+ network-transport-tcp >= 0.3 && < 0.4,

3737 distributed-process >= 0.2 && < 0.4

3838 Exposed-modules: Control.Distributed.Process.Backend.SimpleLocalnet,

3939 Control.Distributed.Process.Backend.SimpleLocalnet.Internal.Multicast

@@ -54,8 +54,8 @@ Test-Suite TestSimpleLocalnet

5454 binary >= 0.5 && < 0.6,

5555 containers >= 0.4 && < 0.6,

5656 transformers >= 0.2 && < 0.4,

57− network-transport >= 0.2 && < 0.3,

58− network-transport-tcp >= 0.2 && < 0.3,

57+ network-transport >= 0.3 && < 0.4,

58+ network-transport-tcp >= 0.3 && < 0.4,

5959 distributed-process >= 0.2 && < 0.4

6060 Extensions: RankNTypes,

6161 DeriveDataTypeable

distributed-process/ChangeLog+5

@@ -1,3 +1,8 @@

1+<<date>> Edsko de Vries <edsko@well-typed.com> 0.3.2

3+* Improved treatment of network failure, using new failure semantics of

4+Network.Transport.

162012-08-16 Edsko de Vries <edsko@well-typed.com> 0.3.1

38* Fix memory leaks

distributed-process/distributed-process.cabal+6−6

@@ -1,5 +1,5 @@

11Name: distributed-process

2−Version: 0.3.1

2+Version: 0.3.2

33Cabal-Version: >=1.8

44Build-Type: Simple

55License: BSD3

@@ -36,7 +36,7 @@ flag th

3636Library

3737 Build-Depends: base >= 4.4 && < 5,

3838 binary >= 0.5 && < 0.6,

39− network-transport >= 0.2 && < 0.3,

39+ network-transport >= 0.3 && < 0.4,

4040 stm >= 2.3 && < 2.5,

4141 transformers >= 0.2 && < 0.4,

4242 mtl >= 2.0 && < 2.2,

@@ -84,8 +84,8 @@ Test-Suite TestCH

8484 random >= 1.0 && < 1.1,

8585 ansi-terminal >= 0.5 && < 0.6,

8686 distributed-process >= 0.3 && < 0.4,

87− network-transport >= 0.2 && < 0.3,

88− network-transport-tcp >= 0.2 && < 0.3,

87+ network-transport >= 0.3 && < 0.4,

88+ network-transport-tcp >= 0.3 && < 0.4,

8989 binary >= 0.5 && < 0.6,

9090 network >= 2.3 && < 2.4

9191 Other-modules: TestAuxiliary

@@ -104,8 +104,8 @@ Test-Suite TestClosure

104104 ansi-terminal >= 0.5 && < 0.6,

105105 distributed-static >= 0.2 && < 0.3,

106106 distributed-process >= 0.3 && < 0.4,

107− network-transport >= 0.2 && < 0.3,

108− network-transport-tcp >= 0.2 && < 0.3,

107+ network-transport >= 0.3 && < 0.4,

108+ network-transport-tcp >= 0.3 && < 0.4,

109109 bytestring >= 0.9 && < 0.11,

110110 network >= 2.3 && < 2.4

111111 Other-modules: TestAuxiliary

distributed-process/src/Control/Distributed/Process/Node.hs+7−6

@@ -345,23 +345,24 @@ handleIncomingMessages node = go Set.empty Map.empty Map.empty Set.empty

345345 (Map.delete cid procs)

346346 (Map.delete cid chans)

347347 (Set.delete cid ctrls)

348− NT.ErrorEvent (NT.TransportError (NT.EventConnectionLost (Just theirAddr) cids) _) -> do

348+ NT.ErrorEvent (NT.TransportError (NT.EventConnectionLost theirAddr) _) -> do

349349 -- [Unified table 9, rule node_disconnect]

350350 let nid = NodeIdentifier $ NodeId theirAddr

351351 writeChan ctrlChan NCMsg

352352 { ctrlMsgSender = nid

353353 , ctrlMsgSignal = Died nid DiedDisconnect

354354 }

355+ -- TODO: we should remove the incoming transactions that were lost

356+ -- TODO: and we should remove the outgoing connections for which

357+ -- we have an implicit reconnect

358+ {-

355359 let notRemoved k = k `notElem` cids

356360 go (Set.filter notRemoved uninitConns)

357361 (Map.filterWithKey (const . notRemoved) procs)

358362 (Map.filterWithKey (const . notRemoved) chans)

359363 (Set.filter notRemoved ctrls)

360− NT.ErrorEvent (NT.TransportError (NT.EventConnectionLost Nothing _) _) ->

361− -- TODO: We should treat an asymetrical connection loss (incoming

362− -- connection broken, but outgoing connection still potentially ok)

363− -- as a fatal error on the part of the remote node (like (**), above)

364− fail "handleIncomingMessages: TODO"

364+ -}

365+ go uninitConns procs chans ctrls

365366 NT.ErrorEvent (NT.TransportError NT.EventEndPointFailed str) ->

366367 fail $ "Cloud Haskell fatal error: end point failed: " ++ str

367368 NT.ErrorEvent (NT.TransportError NT.EventTransportFailed str) ->

network-transport-tcp/ChangeLog+4

@@ -1,3 +1,7 @@

1+<<date>> Edsko de Vries <edsko@well-typed.com> 0.3.0

3+* Implement new disconnection semantics

152012-08-20 Edsko de Vries <edsko@well-typed.com> 0.2.0.3

37* Allow for colons in hostnames (for IPv6)

network-transport-tcp/network-transport-tcp.cabal+3−3

@@ -1,5 +1,5 @@

11Name: network-transport-tcp

2−Version: 0.2.0.3

2+Version: 0.3.0

33Cabal-Version: >=1.8

44Build-Type: Simple

55License: BSD3

@@ -22,7 +22,7 @@ Source-Repository head

2222

2323Library

2424 Build-Depends: base >= 4.3 && < 5,

25− network-transport >= 0.2 && < 0.3,

25+ network-transport >= 0.3 && < 0.4,

2626 data-accessor >= 0.2 && < 0.3,

2727 containers >= 0.4 && < 0.6,

2828 bytestring >= 0.9 && < 0.11,

@@ -37,7 +37,7 @@ Test-Suite TestTCP

3737 Type: exitcode-stdio-1.0

3838 Main-Is: TestTCP.hs

3939 Build-Depends: base >= 4.3 && < 5,

40− network-transport >= 0.2 && < 0.3,

40+ network-transport >= 0.3 && < 0.4,

4141 data-accessor >= 0.2 && < 0.3,

4242 containers >= 0.4 && < 0.6,

4343 bytestring >= 0.9 && < 0.11,

network-transport-tcp/src/Network/Transport/TCP.hs+2−5

@@ -745,8 +745,7 @@ modifyRemoteState (ourEndPoint, theirEndPoint) match =

745745 handleIOException ex vst = do

746746 tryCloseSocket (remoteSocket vst)

747747 putMVar theirState (RemoteEndPointFailed ex)

748− let incoming = IntSet.elems $ vst ^. remoteIncoming

749− code = EventConnectionLost (Just $ remoteAddress theirEndPoint) incoming

748+ let code = EventConnectionLost (remoteAddress theirEndPoint)

750749 err = TransportError code (show ex)

751750 writeChan (localChannel ourEndPoint) $ ErrorEvent err

752751

@@ -1085,9 +1084,7 @@ handleIncomingMessages (ourEndPoint, theirEndPoint) = do

10851084 relyViolation (ourEndPoint, theirEndPoint)

10861085 "handleIncomingMessages:prematureExit"

10871086 RemoteEndPointValid vst -> do

1088− let code = EventConnectionLost

1089− (Just $ remoteAddress theirEndPoint)

1090− (IntSet.elems $ vst ^. remoteIncoming)

1087+ let code = EventConnectionLost (remoteAddress theirEndPoint)

10911088 writeChan ourChannel . ErrorEvent $ TransportError code (show err)

10921089 forM_ (vst ^. pendingCtrlRequests) $ flip putMVar (Left err)

10931090 return (RemoteEndPointFailed err)

network-transport-tcp/tests/TestTCP.hs+11−11

@@ -98,11 +98,11 @@ testEarlyDisconnect nextPort = do

9898

9999 -- TEST 1: they connect to us, then drop the connection

100100 do

101− ConnectionOpened cid _ addr <- receive endpoint

101+ ConnectionOpened _ _ addr <- receive endpoint

102102 True <- return $ addr == theirAddr

103103

104− ErrorEvent (TransportError (EventConnectionLost (Just addr') [cid']) _) <- receive endpoint

105− True <- return $ addr' == theirAddr && cid' == cid

104+ ErrorEvent (TransportError (EventConnectionLost addr') _) <- receive endpoint

105+ True <- return $ addr' == theirAddr

106106

107107 return ()

108108

@@ -124,8 +124,8 @@ testEarlyDisconnect nextPort = do

124124 Received cid' ["pong"] <- receive endpoint

125125 True <- return $ cid == cid'

126126

127− ErrorEvent (TransportError (EventConnectionLost (Just addr') [cid'']) _) <- receive endpoint

128− True <- return $ addr' == theirAddr && cid'' == cid

127+ ErrorEvent (TransportError (EventConnectionLost addr') _) <- receive endpoint

128+ True <- return $ addr' == theirAddr

129129

130130 return ()

131131

@@ -235,7 +235,7 @@ testEarlyCloseSocket nextPort = do

235235 ConnectionClosed cid'' <- receive endpoint

236236 True <- return $ cid'' == cid

237237

238− ErrorEvent (TransportError (EventConnectionLost (Just addr') []) _) <- receive endpoint

238+ ErrorEvent (TransportError (EventConnectionLost addr') _) <- receive endpoint

239239 True <- return $ addr' == theirAddr

240240

241241 return ()

@@ -634,7 +634,7 @@ testReconnect nextPort = do

634634

635635 -- But a send will fail because the server has closed the connection again

636636 Left (TransportError SendFailed _) <- send conn1 ["ping"]

637− ErrorEvent (TransportError (EventConnectionLost _ []) _) <- receive endpoint

637+ ErrorEvent (TransportError (EventConnectionLost _) _) <- receive endpoint

638638

639639 -- But a subsequent call to connect should reestablish the connection

640640 Right conn2 <- connect endpoint theirAddr ReliableOrdered defaultConnectHints

@@ -695,7 +695,7 @@ testUnidirectionalError nextPort = do

695695

696696 -- But when we send we find the error

697697 Left (TransportError SendFailed _) <- send conn1 ["ping"]

698− ErrorEvent (TransportError (EventConnectionLost _ []) _) <- receive endpoint

698+ ErrorEvent (TransportError (EventConnectionLost _) _) <- receive endpoint

699699

700700 -- A call to connect should now re-establish the connection

701701 Right conn2 <- connect endpoint theirAddr ReliableOrdered defaultConnectHints

@@ -709,7 +709,7 @@ testUnidirectionalError nextPort = do

709709 -- We now find the error when we attempt to close the connection

710710 Nothing <- timeout 500000 $ receive endpoint

711711 close conn2

712− ErrorEvent (TransportError (EventConnectionLost _ []) _) <- receive endpoint

712+ ErrorEvent (TransportError (EventConnectionLost _) _) <- receive endpoint

713713 Right conn3 <- connect endpoint theirAddr ReliableOrdered defaultConnectHints

714714 send conn3 ["ping"]

715715 takeMVar serverGotPing

@@ -721,7 +721,7 @@ testUnidirectionalError nextPort = do

721721 -- Now we notice the problem when we try to connect

722722 Nothing <- timeout 500000 $ receive endpoint

723723 Left (TransportError ConnectFailed _) <- connect endpoint theirAddr ReliableOrdered defaultConnectHints

724− ErrorEvent (TransportError (EventConnectionLost _ []) _) <- receive endpoint

724+ ErrorEvent (TransportError (EventConnectionLost _) _) <- receive endpoint

725725 Right conn4 <- connect endpoint theirAddr ReliableOrdered defaultConnectHints

726726 send conn4 ["ping"]

727727 takeMVar serverGotPing

@@ -746,7 +746,7 @@ testInvalidCloseConnection nextPort = do

746746

747747 -- At this point the client sends an invalid request, so we terminate the

748748 -- connection

749− ErrorEvent (TransportError (EventConnectionLost _ [_]) _) <- receive endpoint

749+ ErrorEvent (TransportError (EventConnectionLost _) _) <- receive endpoint

750750

751751 putMVar serverDone ()

752752

network-transport-tcp/tests/TestTransport.hs+4−4

@@ -606,7 +606,7 @@ testCloseEndPoint transport _ = do

606606 send conn ["pong"]

607607

608608 ConnectionClosed cid'' <- receive endpoint ; True <- return $ cid == cid''

609− ErrorEvent (TransportError (EventConnectionLost (Just addr') []) _) <- receive endpoint ; True <- return $ addr' == theirAddr

609+ ErrorEvent (TransportError (EventConnectionLost addr') _) <- receive endpoint ; True <- return $ addr' == theirAddr

610610

611611 Left (TransportError SendFailed _) <- send conn ["pong2"]

612612

@@ -694,7 +694,7 @@ testCloseTransport newTransport = do

694694 evs <- replicateM 3 $ receive endpoint

695695 let expected = [ ConnectionClosed cid1

696696 , ConnectionClosed cid2

697− , ErrorEvent (TransportError (EventConnectionLost (Just theirAddr2) []) "")

697+ , ErrorEvent (TransportError (EventConnectionLost theirAddr2) "")

698698 ]

699699 True <- return $ any (== expected) (permutations evs)

700700

@@ -813,8 +813,8 @@ testSendException newTransport = do

813813 -- This will have been as a failure to send by endpoint1, which will

814814 -- therefore have closed the socket. In turn this will have caused endpoint2

815815 -- to report that the connection was lost

816− ErrorEvent (TransportError (EventConnectionLost _ []) _) <- receive endpoint1

817− ErrorEvent (TransportError (EventConnectionLost _ [_]) _) <- receive endpoint2

816+ ErrorEvent (TransportError (EventConnectionLost _) _) <- receive endpoint1

817+ ErrorEvent (TransportError (EventConnectionLost _) _) <- receive endpoint2

818818

819819 -- A new connection will re-establish the connection

820820 Right conn2 <- connect endpoint1 (address endpoint2) ReliableOrdered defaultConnectHints

network-transport/ChangeLog+4

@@ -1,3 +1,7 @@

1+<<date> Edsko de Vries <edsko@well-typed.com> 0.3.0

3+* Clarify disconnection

152012-07-16 Edsko de Vries <edsko@well-typed.com> 0.2.0.2

37* Base 4.6 compatible test suites

network-transport/network-transport.cabal+1−1

@@ -1,5 +1,5 @@

11Name: network-transport

2−Version: 0.2.0.2

2+Version: 0.3.0

33Cabal-Version: >=1.6

44Build-Type: Simple

55License: BSD3

network-transport/src/Network/Transport.hs+31−4

@@ -230,8 +230,35 @@ data EventErrorCode =

230230 EventEndPointFailed

231231 -- | Transport-wide fatal error

232232 | EventTransportFailed

233− -- | Some incoming connections were closed abruptly.

234− -- If an endpoint address is specified, then all connections to and

235− -- from that endpoint are now lost

236− | EventConnectionLost (Maybe EndPointAddress) [ConnectionId]

233+ -- | We lost connection to another endpoint

234+ --

235+ -- Although "Network.Transport" provides multiple independent lightweight

236+ -- connections between endpoints, those connections cannot /fail/

237+ -- independently: once one connection has failed, /all/ connections, in

238+ -- both directions, must now be considered to have failed; they fail as a

239+ -- "bundle" of connections, with only a single "bundle" of connections per

240+ -- endpoint at any point in time.

241+ --

242+ -- That is, suppose there are multiple connections in either direction

243+ -- between A and B, and A receives a notification that it has lost

244+ -- connection to B. Then A should not be able to send any further messages

245+ -- to B on existing connections.

246+ --

247+ -- Although B may not realize /immediately/ that its connection to A has

248+ -- been broken, messages sent by B on existing connections should not be

249+ -- delivered, and B must eventually get an EventConnectionLost message,

250+ -- too.

251+ --

252+ -- Moreover, this event must be posted before A has successfully

253+ -- reconnected (in other words, if B notices a reconnection attempt from A,

254+ -- it must post the EventConnectionLost before acknowledging the connection

255+ -- from A) so that B will not receive events about new connections or

256+ -- incoming messages from A without realizing that it got disconnected.

257+ --

258+ -- If B attempts to establish another connection to A before it realized

259+ -- that it got disconnected from A then it's okay for this connection

260+ -- attempt to fail, and the EventConnectionLost to be posted at that point,

261+ -- or for the EventConnectionLost to be posted and for the new connection

262+ -- to be considered the first connection of the "new bundle".

263+ | EventConnectionLost EndPointAddress

237264 deriving (Show, Typeable, Eq)