bc1d5e7 · project-m36 · git.iko.soy

git.iko.soy/project-m36

Merge branch 'remote_import'

commitbc1d5e739721a1ce95d28f123ec3f40f6a92c6ac authorAgentM <agentm@themactionfaction.com> dateSat 3 Apr 2021, 23:07 (6 years ago) parent480e3a2 73bfbb9

29 files changed+199−106

compare with master

unifiedsplit

Changelog.markdown+4

@@ -1,3 +1,7 @@

1+# 2021-04-01 (v0.9.3)

3+* add new ":importtutd <URI> <SHA256 hash>" feature to import TutorialD from a local file or a HTTP/HTTPS URI

15# 2020-12-27 (v0.9.0)

37* replace unmaintained distributed-process-client-server RPC package with new curryer RPC package

docs/15_minute_tutorial.markdown+1−1

@@ -28,7 +28,7 @@ Note that TutorialD is strongly-typed- no type coercions take place automaticall

2828### Load the Canonical Chris Date Example

2929

3030```

31−TutorialD (master/main): :importexample date

31+TutorialD (master/main): :importexample cjdate

3232```

3333

3434```:importexample``` (do not skip typing the preceding colon), loads a built-in example schema and executes it to change the state of the database. In this case, the relations "s" (supplier), "p" (product), and "sp" (suppliers-products) are now loaded.

docs/import_export_csv.markdown+2−2

@@ -9,7 +9,7 @@ If a relation variable cannot be exported due to data types which are not suppor

99### Example Export

1010

1111```

12−TutorialD (master/main): :importexample date

12+TutorialD (master/main): :importexample cjdate

1313TutorialD (master/main): :exportcsv s "/tmp/csv"

1414```

1515```

@@ -37,7 +37,7 @@ s#,sname,status,city

3737S6,Samson,100,New York

3838```

3939```

40−TutorialD (master/main): :importexample date

40+TutorialD (master/main): :importexample cjdate

4141TutorialD (master/main): :showexpr s

4242┌──────────┬────────┬───────────┬───────────────┐

4343│city::Text│s#::Text│sname::Text│status::Integer│

docs/reaching_out_of_the_tarpit.markdown+1−1

@@ -57,7 +57,7 @@ Project:M36 implements multiple interfaces:

5757To understand how Project:M36 stacks up against the fictional programming language shown in the paper, we have implemented the [example from the paper](/examples/out_of_the_tarpit.tutd). The language used in the script is a dialect of Chris Date's TutorialD language. It can be loaded into an interactive ```tutd``` session like so:

5858

5959```

60−TutorialD (master/main): :importexample date

60+TutorialD (master/main): :importexample cjdate

6161TutorialD (master/main): :showexpr p

6262┌───────────────────┬────────────┬───────────────────┬────────────┬────────────┐

6363│address::Address │agent::Agent│dateRegistered::Day│photo::Text │price::Price│

docs/tutd_cheatsheet.markdown+3−1

@@ -47,7 +47,9 @@ Database context expressions take the current database context as input and alte

4747

4848|Database Context Expression |Description |

4949|------------------------------------------------------|---------------------------------------|

50−|`:importexample date`|Imports a precooked schema from Chris Date's books resulting in relvars `s`, `sp`, and `p`|

50+|`:importexample cjdate`|Imports a precooked schema from Chris Date's books resulting in relvars `s`, `sp`, and `p`|

51+|`:importtutd "file:///home/agentm/project-m36/scripts/emp.tutd"`|Imports TutorialD from file URI|

52+|`:importtutd "https://raw.githubusercontent.com/agentm/project-m36/master/scripts/emp.tutd" "effe32b247586dc3ac0079fc241b9618d41d189afcaeb7907edbe5a8b45992a4"`| Imports TutorialD from HTTP or HTTPS URI|

5153|`newrelvar:=relation{tuple{age 3}}`|Assign a new relation variable named `newrelvar`|

5254|`undefine s`|Remove a relation variable|

5355|`insert s relation{tuple{city "Boston",s# "S10",sname "Gonzalez",status 10}}`| Append to an existing relation variable with another relational expression|

docs/tutd_tutorial.markdown+19−1

@@ -19,7 +19,7 @@ TutorialD (master/main):

1919The examples in this tutorial can be executed after loading Chris Date's sample relations using:

2020

2121```

22−:importexample date

22+:importexample cjdate

2323```

2424

2525Each section assumes a fresh instance of the Date examples. The relation variables created in the script are:

@@ -30,6 +30,24 @@ Each section assumes a fresh instance of the Date examples. The relation variabl

3030|p|parts|

3131|sp|contains mapping of which suppliers supply which parts|

3232

33+### Importing TutorialD from Other Sources

34+

35+The `tutd` console can also run TutorialD from a local file or from a web server.

36+

37+To import TutorialD from a file:

38+

39+```

40+:importtutd "file:///home/agentm/project-m36/scripts/emp.tutd"

41+```

42+

43+or from a web server:

44+

45+```

46+:importtutd "https://raw.githubusercontent.com/agentm/project-m36/master/scripts/DateExamples.tutd" "db9f3d9fe06d0a29b8355e045b89ec94d6428e3a3de93def7ca77bf0298b7010"

47+```

48+

49+The second argument is an optional SHA256 hash in hexadecimal form. This can be used to validate that the script has not changed since the last import. Importing TutorialD from the web is in general safe because it is effectively sandboxed with "Safe Haskell" for functions, so the script is contained security-wise, however, nothing prevents the script from running forever. Downloading a schema from the web is a great way to share schemas and to kick start your project without starting from scratch.

50+

3351### Types

3452

3553TutorialD is strongly-typed. The basic built-in types are:

examples/out_of_the_tarpit.tutd+4−4

@@ -3,9 +3,9 @@

33data Address = Address Text

44data Agent = Agent Text

55data Name = Name Text

6−data Price = Price Double

6+data Price = Price Integer --dollars

77data RoomType = Kitchen | Bathroom | LivingRoom

8−data PriceBand = Low | Medium | High | Premium

8+data PriceBand = Low | Med | High | Premium

99data AreaCode = City | Suburban | Rural

1010data SpeedBand = VeryFast | Fast | Medium | Slow

1111

@@ -57,9 +57,9 @@ constraint noOffersAfterSale (offer join decision where ^dayEarlierThan(@offerDa

5757--1. identify offerPrices > 2000 (arbitrarily-chosen high price band which is not specified in the paper)

5858--2. remove the properties which have been sold

5959--3. group and count sub-relation and restrict by greater than 50

60−constraint noMoreThan50PremiumProperties ((((property where ^gt(offerPrice,2000.0)) minus ((property join decision){address,agent,dateRegistered,photo,price})) group ({} as x) : {z:=gt(count(@x),50)}) {z}){} equals false

60+constraint noMoreThan50PremiumProperties ((((property where ^gt(@price,Price 2000)) minus ((property join decision){address,agent,dateRegistered,photo,price})) group ({} as x) : {z:=gt(count(@x),50)}) {z}){} equals false

6161

62−constraint noBidderCanMakeMoreThan10BidsOnAProperty (((offer group ({all but bidderAddress} as bidder)) : {bidCount:=gt(count(@bidder),10)}) where ^gt(bidCount,10)) {} equals false

62+constraint noBidderCanMakeMoreThan10BidsOnAProperty (((offer group ({all but bidderAddress} as bidder)) : {bidCount:=gt(count(@bidder),10)}) where ^gt(@bidCount,10)) {} equals false

6363

6464-- sample data

6565

jupyter/TutorialD Notebook Walkthrough.ipynb+1−1

@@ -38,7 +38,7 @@

3838 }

3939 ],

4040 "source": [

41− ":importexample date"

41+ ":importexample cjdate"

4242 ]

4343 },

4444 {

project-m36.cabal+12−6

@@ -1,6 +1,6 @@

11Cabal-Version: 2.2

22Name: project-m36

3−Version: 0.9.2

3+Version: 0.9.3

44License: MIT

55--note that this license specification is erroneous and only labeled MIT to appease hackage which does not recognize public domain packages in cabal >2.2- Project:M36 is dedicated to the public domain

66Build-Type: Simple

@@ -35,7 +35,7 @@ Flag haskell-scripting

3535 Default: True

3636

3737Library

38− Build-Depends: base>=4.8 && < 4.15, ghc-paths, mtl, containers, unordered-containers, hashable, haskeline, directory, MonadRandom, random-shuffle, uuid >= 1.3.12, cassava >= 0.4.5.1 && < 0.6, text, bytestring, deepseq, deepseq-generics, vector, parallel, monad-parallel, exceptions, transformers, gnuplot, filepath, zlib, directory, temporary, stm, time, hashable-time, old-locale, rset, attoparsec, either, base64-bytestring, data-interval, extended-reals, aeson >= 1.1, path-pieces, conduit, resourcet, http-api-data, semigroups, QuickCheck, quickcheck-instances, list-t, stm-containers >= 0.2.15, foldl, optparse-applicative, Glob, cryptohash-sha256, text-manipulate >= 0.2.0.1 && < 0.4, winery, curryer-rpc, network, async, vector-instances, recursion-schemes

38+ Build-Depends: base>=4.8 && < 4.15, ghc-paths, mtl, containers, unordered-containers, hashable, haskeline, directory, MonadRandom, random-shuffle, uuid >= 1.3.12, cassava >= 0.4.5.1 && < 0.6, text, bytestring, deepseq, deepseq-generics, vector, parallel, monad-parallel, exceptions, transformers, gnuplot, filepath, zlib, directory, temporary, stm, time, hashable-time, old-locale, rset, attoparsec, either, base64-bytestring, data-interval, extended-reals, aeson >= 1.1, path-pieces, conduit, resourcet, http-api-data, semigroups, QuickCheck, quickcheck-instances, list-t, stm-containers >= 0.2.15, foldl, optparse-applicative, Glob, cryptohash-sha256, text-manipulate >= 0.2.0.1 && < 0.4, winery, curryer-rpc, network, async, vector-instances, recursion-schemes, streamly >= 0.7.2

3939 if flag(haskell-scripting)

4040 Build-Depends: ghc >= 8.2 && < 8.11

4141 CPP-Options: -DPM36_HASKELL_SCRIPTING

@@ -184,7 +184,13 @@ Executable tutd

184184 list-t,

185185 parser-combinators,

186186 curryer-rpc,

187− prettyprinter

187+ prettyprinter,

188+ cryptohash-sha256,

189+ --due to decode signature change

190+ base16-bytestring >= 1.0.0.0,

191+ http-conduit,

192+ modern-uri,

193+ http-types

188194 Other-Modules: TutorialD.Interpreter,

189195 TutorialD.Interpreter.Base,

190196 TutorialD.Interpreter.DatabaseContextExpr,

@@ -264,7 +270,7 @@ Executable bigrel

264270

265271Common commontest

266272 Default-Language: Haskell2010

267− Build-Depends: base, HUnit, Cabal, containers, hashable, unordered-containers, mtl, vector, time, hashable-time, bytestring, uuid, stm, deepseq, deepseq-generics,parallel, cassava, attoparsec, gnuplot, directory, temporary, haskeline, megaparsec, text, base64-bytestring, data-interval, filepath, transformers, stm-containers, list-t, websockets, optparse-applicative, network, aeson, project-m36, random, MonadRandom, semigroups, parser-combinators, winery, curryer-rpc, prettyprinter, base64-bytestring

273+ Build-Depends: base, HUnit, Cabal, containers, hashable, unordered-containers, mtl, vector, time, hashable-time, bytestring, uuid, stm, deepseq, deepseq-generics,parallel, cassava, attoparsec, gnuplot, directory, temporary, haskeline, megaparsec, text, base64-bytestring, data-interval, filepath, transformers, stm-containers, list-t, websockets, optparse-applicative, network, aeson, project-m36, random, MonadRandom, semigroups, parser-combinators, winery, curryer-rpc, prettyprinter, base64-bytestring, modern-uri, http-types, http-conduit, base16-bytestring,cryptohash-sha256

268274 Default-Extensions: OverloadedStrings

269275 GHC-Options: -Wall -threaded

270276 Hs-Source-Dirs: ./src/bin, ., test/

@@ -419,7 +425,7 @@ Test-Suite test-scripts

419425

420426Executable project-m36-websocket-server

421427 Default-Language: Haskell2010

422− Build-Depends: base, aeson, path-pieces, either, conduit, http-api-data, template-haskell, websockets, aeson, optparse-applicative, project-m36, containers, bytestring, text, vector, uuid, megaparsec, haskeline, mtl, directory, base64-bytestring, random, MonadRandom, time, semigroups, attoparsec, parser-combinators, prettyprinter, network

428+ Build-Depends: base, aeson, path-pieces, either, conduit, http-api-data, template-haskell, websockets, aeson, optparse-applicative, project-m36, containers, bytestring, text, vector, uuid, megaparsec, haskeline, mtl, directory, base64-bytestring, random, MonadRandom, time, semigroups, attoparsec, parser-combinators, prettyprinter, network, modern-uri, http-types, cryptohash-sha256, http-conduit, base16-bytestring

423429 Main-Is: ProjectM36/Server/WebSocket/websocket-server.hs

424430 Other-Modules: ProjectM36.Client.Json, ProjectM36.Server.RemoteCallTypes.Json, ProjectM36.Server.WebSocket, TutorialD.Interpreter, TutorialD.Interpreter.Base, TutorialD.Interpreter.DatabaseContextExpr, TutorialD.Interpreter.DatabaseContextIOOperator, TutorialD.Interpreter.Export.Base, TutorialD.Interpreter.Export.CSV, TutorialD.Interpreter.Import.Base, TutorialD.Interpreter.Import.BasicExamples, TutorialD.Interpreter.Import.CSV, TutorialD.Interpreter.Import.TutorialD, TutorialD.Interpreter.InformationOperator, TutorialD.Interpreter.RODatabaseContextOperator, TutorialD.Interpreter.RelationalExpr, TutorialD.Interpreter.TransactionGraphOperator, TutorialD.Interpreter.Types, TutorialD.Interpreter.TransGraphRelationalOperator, TutorialD.Interpreter.SchemaOperator, TutorialD.Printer

425431 GHC-Options: -Wall -threaded

@@ -468,7 +474,7 @@ Test-Suite test-client-simple

468474Executable handles

469475 Default-Language: Haskell2010

470476 Default-Extensions: OverloadedStrings

471− Build-Depends: base, HUnit, Cabal, containers, hashable, unordered-containers, mtl, vector, time, hashable-time, bytestring, uuid, stm, deepseq, deepseq-generics,parallel, cassava, attoparsec, gnuplot, directory, temporary, haskeline, megaparsec, text, base64-bytestring, data-interval, filepath, optparse-applicative, stm-containers, list-t, ghc, ghc-paths, transformers, project-m36, random, MonadRandom, semigroups, parser-combinators, prettyprinter

477+ Build-Depends: base, HUnit, Cabal, containers, hashable, unordered-containers, mtl, vector, time, hashable-time, bytestring, uuid, stm, deepseq, deepseq-generics,parallel, cassava, attoparsec, gnuplot, directory, temporary, haskeline, megaparsec, text, base64-bytestring, data-interval, filepath, optparse-applicative, stm-containers, list-t, ghc, ghc-paths, transformers, project-m36, random, MonadRandom, semigroups, parser-combinators, prettyprinter, modern-uri, base16-bytestring, http-conduit, cryptohash-sha256, http-types

472478 main-is: benchmark/Handles.hs

473479 Other-Modules: TutorialD.Interpreter,

474480 TutorialD.Interpreter.Base,

scripts/DateExamples.tutd+2−2

@@ -20,7 +20,7 @@ sp := relation{tuple{s# "S1",p# "P1",qty 300},

2020 tuple{s# "S3",p# "P2",qty 200},

2121 tuple{s# "S4",p# "P4",qty 300},

2222 tuple{s# "S4",p# "P5",qty 400}};

23−key s_pkey {s#} s

24−key p_key {p#} p

23+key s_pkey {s#} s;

24+key p_key {p#} p;

2525foreign key s_sp_fk sp{s#} in s{s#};

2626foreign key p_sp_fk sp{p#} in p{p#};

scripts/multiline.tutdnew file+2

@@ -0,0 +1,2 @@

1+x:=relation{

2+tuple{ x 2}} -- nice

src/bin/ProjectM36/Server/RemoteCallTypes/Json.hs+3

@@ -164,6 +164,9 @@ instance FromJSON SchemaError

164164instance ToJSON MergeError

165165instance FromJSON MergeError

166166

167+instance ToJSON ImportError'

168+instance FromJSON ImportError'

169+

167170instance ToJSON DatabaseContextFunctionError

168171instance FromJSON DatabaseContextFunctionError

169172

src/bin/ProjectM36/Server/WebSocket/websocket-client.html+1−1

@@ -126,7 +126,7 @@ margin-bottom: 5pt;

126126 <li>:showrelvars</li>

127127 <li>:showexpr true</li>

128128 <li>:showexpr false</li>

129− <li>:importexample date</li>

129+ <li>:importexample cjdate</li>

130130 <li>:showexpr s join sp</li>

131131 <li>update s where city="Paris" (status:=10)</li>

132132 <li>:showexpr s</li>

src/bin/TutorialD/Interpreter.hs+1

@@ -216,6 +216,7 @@ evalTutorialDInteractive sessionId conn safe interactive expr = case expr of

216216 unsafeError = pure $ DisplayErrorResult "File I/O operation prohibited."

217217 barf :: RelationalError -> IO TutorialDOperatorResult

218218 barf (ScriptError (OtherScriptCompilationError errStr)) = pure (DisplayErrorResult (T.pack errStr))

219+ barf (ParseError err) = pure (DisplayErrorResult err)

219220 barf err = return $ DisplayErrorResult (T.pack (show err))

220221 eHandler io = do

221222 eErr <- io

src/bin/TutorialD/Interpreter/Base.hs+17−2

@@ -17,7 +17,7 @@ import ProjectM36.Relation

1717import ProjectM36.DataFrame

1818

1919#if MIN_VERSION_megaparsec(6,0,0)

20−import Text.Megaparsec.Char

20+import Text.Megaparsec.Char

2121import qualified Text.Megaparsec.Char.Lexer as Lex

2222import Text.Megaparsec

2323import Data.Void

@@ -44,6 +44,7 @@ import Data.List.NonEmpty as NE

4444import Data.Time.Clock

4545import Data.Time.Format

4646import Control.Monad (void)

47+import Data.Char

4748

4849#if !MIN_VERSION_megaparsec(7,0,0)

4950anySingle :: Parsec Void Text (Token Text)

@@ -81,8 +82,9 @@ type ParseStr = Text

8182type ParseStr = String

8283#endif

8384

85+-- consumes only horizontal spaces

8486spaceConsumer :: Parser ()

85−spaceConsumer = Lex.space (void spaceChar) (Lex.skipLineComment "--") (Lex.skipBlockComment "{-" "-}")

87+spaceConsumer = Lex.space space1 (Lex.skipLineComment "--") (Lex.skipBlockComment "{-" "-}")

8688

8789opChar :: Parser Char

8890opChar = oneOf (":!#$%&*+./<=>?\\^|-~" :: String)-- remove "@" so it can be used as attribute marker without spaces

@@ -135,6 +137,9 @@ arrow = symbol "->"

135137semi :: Parser Text

136138semi = symbol ";"

137139

140+nline :: Parser Text

141+nline = (T.singleton <$> newline) <|> crlf

142+

138143integer :: Parser Integer

139144#if MIN_VERSION_megaparsec(6,0,0)

140145integer = Lex.signed (pure ()) Lex.decimal <* spaceConsumer

@@ -200,6 +205,9 @@ normalQuotedString = quote *> (T.pack <$> manyTill Lex.charLiteral quote)

200205quotedString :: Parser Text

201206quotedString = try tripleQuotedString <|> normalQuotedString

202207

208+quoted :: Parser a -> Parser a

209+quoted = between quote quote

210+

203211uuidP :: Parser U.UUID

204212uuidP = do

205213 uuidStart <- count 8 hexDigitChar

@@ -228,3 +236,10 @@ colonOp :: Text -> Parser ()

228236colonOp opStr = do

229237 _ <- string opStr <* (void spaceChar <|> eof) <* spaceConsumer

230238 pure ()

239+

240+hex :: Parser Text

241+hex = takeWhileP (Just "hexadecimal")

242+ (\c ->

243+ isDigit c

244+ || (c >= 'a' && c <= 'f'))

245+

src/bin/TutorialD/Interpreter/DatabaseContextExpr.hs+3−2

@@ -53,8 +53,9 @@ multilineSep :: Parser T.Text

5353multilineSep = newline >> pure "\n"

5454

5555multipleDatabaseContextExprP :: Parser DatabaseContextExpr

56−multipleDatabaseContextExprP =

57− MultipleExpr <$> sepBy1 databaseContextExprP semi

56+multipleDatabaseContextExprP = do

57+ exprs <- filter (/= NoOperation) <$> sepBy1 databaseContextExprP semi

58+ pure (MultipleExpr exprs)

5859

5960insertP :: Parser DatabaseContextExpr

6061insertP = do

src/bin/TutorialD/Interpreter/Import/Base.hs+8−3

@@ -2,6 +2,8 @@

22module TutorialD.Interpreter.Import.Base where

33import ProjectM36.Base

44import ProjectM36.Error

5+import Text.URI (URI)

6+import Data.Text (Text)

57#if __GLASGOW_HASKELL__ < 804

68import Data.Monoid

79#endif

@@ -12,11 +14,14 @@ data RelVarDataImportOperator = RelVarDataImportOperator RelVarName FilePath (Re

1214instance Show RelVarDataImportOperator where

1315 show (RelVarDataImportOperator rv path _) = "RelVarDataImportOperator " <> show rv <> " " <> path

1416

17+type HashVerification = Maybe Text

18+

1519-- | import data into a database context

16−data DatabaseContextDataImportOperator = DatabaseContextDataImportOperator FilePath (FilePath -> IO (Either RelationalError DatabaseContextExpr))

20+data DatabaseContextDataImportOperator = DatabaseContextDataImportOperator URI HashVerification (URI -> HashVerification -> IO (Either RelationalError DatabaseContextExpr))

1721

1822instance Show DatabaseContextDataImportOperator where

19− show (DatabaseContextDataImportOperator path _) = "DatabaseContextDataImportOperator " <> path

23+ show (DatabaseContextDataImportOperator uri hash _) =

24+ "DatabaseContextDataImportOperator " <> show uri <> " " <> show hash

2025

2126-- perhaps create a structure to import a whole transaction graph section in the future

2227

@@ -24,4 +29,4 @@ evalRelVarDataImportOperator :: RelVarDataImportOperator -> TypeConstructorMappi

2429evalRelVarDataImportOperator (RelVarDataImportOperator relVarName path importFunc) tConsMap attrs = importFunc relVarName tConsMap attrs path

2530

2631evalDatabaseContextDataImportOperator :: DatabaseContextDataImportOperator -> IO (Either RelationalError DatabaseContextExpr)

27−evalDatabaseContextDataImportOperator (DatabaseContextDataImportOperator path importFunc) = importFunc path

32+evalDatabaseContextDataImportOperator (DatabaseContextDataImportOperator uri hash importFunc) = importFunc uri hash

src/bin/TutorialD/Interpreter/Import/BasicExamples.hs+2−2

@@ -20,8 +20,8 @@ importBasicExampleOperatorP :: Parser ImportBasicExampleOperator

2020importBasicExampleOperatorP = do

2121 reservedOp ":importexample"

2222 example <- identifier

23− if example == "date" then

23+ if example == "cjdate" then

2424 pure ImportBasicDateExampleOperator

2525 else

2626 fail "Unknown example name"

27−

27+

src/bin/TutorialD/Interpreter/Import/CSV.hs+3−3

@@ -14,10 +14,10 @@ importCSVRelation relVarName tConsMap attrs pathIn = do

1414 --TODO: handle filesystem errors

1515 csvData <- try (BS.readFile pathIn) :: IO (Either IOError BS.ByteString)

1616 case csvData of

17− Left err -> return $ Left (ImportError $ T.pack (show err))

17+ Left err -> pure $ Left (ImportError (ImportFileError (T.pack (show err))))

1818 Right csvData' -> case csvAsRelation attrs tConsMap csvData' of

19− Left err -> return $ Left (ParseError $ T.pack (show err))

20− Right csvRel -> return $ Right (Insert relVarName (ExistingRelation csvRel))

19+ Left err -> pure $ Left (ParseError $ T.pack (show err))

20+ Right csvRel -> pure $ Right (Insert relVarName (ExistingRelation csvRel))

2121

2222importCSVP :: Parser RelVarDataImportOperator

2323importCSVP = do

src/bin/TutorialD/Interpreter/Import/TutorialD.hs+70−17

@@ -9,25 +9,78 @@ import qualified Data.Text as T

99import Control.Exception

1010import qualified Data.ByteString as BS

1111import qualified Data.Text.Encoding as TE

12−--import a file containing TutorialD database context expressions

12+import qualified Text.URI as URI

13+import Text.URI (URI)

14+import Crypto.Hash.SHA256

15+import Data.ByteString.Base16 as B16

16+import Network.HTTP.Simple

17+import Network.HTTP.Types

18+--import a file containing TutorialD database context expressions via local files or HTTP/HTTPS

1319

14−importTutorialD :: FilePath -> IO (Either RelationalError DatabaseContextExpr)

15−importTutorialD pathIn = do

16− eTutdBytes <- try (BS.readFile pathIn) :: IO (Either IOError BS.ByteString)

17− case eTutdBytes of

18− Left err -> return $ Left (ImportError $ T.pack (show err))

19− Right tutdBytes ->

20− case TE.decodeUtf8' tutdBytes of

21− Left err -> pure (Left (ImportError $ T.pack (show err)))

22− Right tutdUtf8 ->

23− case parse multipleDatabaseContextExprP "import" tutdUtf8 of

24− --parseErrorPretty is new in megaparsec 5

25− Left err -> pure (Left (PM36E.ParseError (T.pack (show err))))

26− Right expr -> pure (Right expr)

27−

20+importTutorialDFromFile :: URI -> HashVerification -> IO (Either RelationalError DatabaseContextExpr)

21+importTutorialDFromFile fileURI mHash = do

22+ case URI.uriPath fileURI of

23+ Just (False, _) -> do

24+ eTutdBytes <- try (BS.readFile filePath) :: IO (Either IOError BS.ByteString)

25+ case eTutdBytes of

26+ Left err -> pure $ Left (ImportError (ImportFileError (T.pack (show err))))

27+ Right tutdBytes ->

28+ case validateBytes tutdBytes mHash of

29+ Left err -> pure (Left err)

30+ Right tutdBytes' -> importTutorialDBytes tutdBytes'

31+ _ -> pure (Left (ImportError (InvalidFileURIError (URI.render fileURI))))

32+ where

33+ filePath = drop (length ("file://" :: String)) (URI.renderStr fileURI)

34+

35+

36+importTutorialDBytes :: BS.ByteString -> IO (Either RelationalError DatabaseContextExpr)

37+importTutorialDBytes tutdBytes =

38+ case TE.decodeUtf8' tutdBytes of

39+ Left err -> pure (Left (ImportError (ImportFileDecodeError (T.pack (show err)))))

40+ Right tutdUtf8 ->

41+ case parse (multipleDatabaseContextExprP <* eof) "import" tutdUtf8 of

42+ --parseErrorPretty is new in megaparsec 5

43+ Left err -> pure (Left (PM36E.ParseError (T.pack (errorBundlePretty err))))

44+ Right expr -> pure (Right expr)

45+

46+importTutorialDViaHTTP :: URI -> HashVerification -> IO (Either RelationalError DatabaseContextExpr)

47+importTutorialDViaHTTP uri mHash = do

48+ reqURI <- parseRequest (URI.renderStr uri)

49+ response <- httpBS reqURI

50+ let scode = getResponseStatus response

51+ if scode == ok200 then

52+ let byteBody = getResponseBody response in

53+ case validateBytes byteBody mHash of

54+ Right byteBody' -> do

55+ importTutorialDBytes byteBody'

56+ Left err -> pure (Left err)

57+ else

58+ pure (Left (ImportError (ImportDownloadError (T.pack ("download failed: " <> show scode)))))

59+

60+validateBytes :: BS.ByteString -> HashVerification -> Either RelationalError BS.ByteString

61+validateBytes tutdBytes mHash =

62+ case mHash of

63+ Nothing -> pure tutdBytes

64+ Just expectedHashText ->

65+ case B16.decode (TE.encodeUtf8 expectedHashText) of

66+ Left _ -> Left (ImportError (InvalidSHA256Error expectedHashText))

67+ Right expectedHash | hashBytes == expectedHash -> pure tutdBytes

68+ | otherwise ->

69+ let p = TE.decodeUtf8 . B16.encode in

70+ Left (ImportError (SHA256MismatchError (p expectedHash) (p hashBytes)))

71+ where

72+ hashBytes = hash tutdBytes

73+

74+

2875tutdImportP :: Parser DatabaseContextDataImportOperator

2976tutdImportP = do

3077 reserved ":importtutd"

31− path <- quotedString

78+ uri <- quoted URI.parser

3279 spaceConsumer

33− return $ DatabaseContextDataImportOperator (T.unpack path) importTutorialD

80+ mhash <- optional (quoted hex)

81+ handler <- case URI.unRText <$> URI.uriScheme uri of

82+ Nothing -> fail "URI scheme missing"

83+ Just "file" -> pure importTutorialDFromFile

84+ Just scheme | scheme `elem` ["http", "https"] -> pure importTutorialDViaHTTP

85+ | otherwise -> fail ( "unsupported URI scheme: " <> show scheme)

86+ pure $ DatabaseContextDataImportOperator uri mhash handler

src/bin/TutorialD/Interpreter/RelationalExpr.hs+1

@@ -216,6 +216,7 @@ consumeAtomExprP consume = try functionAtomExprP <|>

216216 attributeAtomExprP <|>

217217 try nakedAtomExprP <|>

218218 relationalAtomExprP

219+

219220

220221attributeAtomExprP :: Parser (AtomExprBase a)

221222attributeAtomExprP = do

src/lib/ProjectM36/Error.hs+10−2

@@ -81,7 +81,7 @@ data RelationalError = NoSuchAttributeNamesError (S.Set AttributeName)

8181 | RelationValuedAttributesNotSupportedError [AttributeName]

8282 | NotificationNameInUseError NotificationName

8383 | NotificationNameNotInUseError NotificationName

84− | ImportError T.Text -- really? This should be broken out into some other error type- this has nothing to do with relational algebra

84+ | ImportError ImportError'

8585 | ExportError T.Text

8686 | UnhandledExceptionError String

8787 | MergeTransactionError MergeError

@@ -144,4 +144,12 @@ data SchemaError = RelVarReferencesMissing (S.Set RelVarName) |

144144 RelVarInReferencedMoreThanOnce RelVarName |

145145 RelVarOutReferencedMoreThanOnce RelVarName

146146 deriving (Show, Eq, Generic, Typeable, NFData)

147−

147+

148+

149+data ImportError' = InvalidSHA256Error T.Text

150+ | SHA256MismatchError T.Text T.Text

151+ | InvalidFileURIError T.Text

152+ | ImportFileDecodeError T.Text

153+ | ImportFileError T.Text

154+ | ImportDownloadError T.Text

155+ deriving (Show, Eq, Generic, Typeable, NFData)

src/lib/ProjectM36/Serialise/Error.hs+1

@@ -12,3 +12,4 @@ deriving via WineryVariant MergeError instance Serialise MergeError

1212deriving via WineryVariant ScriptCompilationError instance Serialise ScriptCompilationError

1313deriving via WineryVariant PersistenceError instance Serialise PersistenceError

1414deriving via WineryVariant SchemaError instance Serialise SchemaError

15+deriving via WineryVariant ImportError' instance Serialise ImportError'

stack.ghc8.0.yamldeleted−48

@@ -1,48 +0,0 @@

1−# This file was automatically generated by stack init

2−# For more information, see: http://docs.haskellstack.org/en/stable/yaml_configuration/

3−

4−# Specifies the GHC version and set of packages available (e.g., lts-3.5, nightly-2015-09-21, ghc-7.10.2)

5−resolver: lts-8.5

6−

7−# Local packages, usually specified by relative directory name

8−packages:

9−- '.'

10−# Packages to be pulled from upstream that are not in the resolver (e.g., acme-missiles-0.3)

11−extra-deps:

12−- git: https://github.com/agentm/curryer.git

13− commit: 1e60c0580ca33d6ae1e97aaddc32189fdfe3ef6c

14−- data-accessor-transformers-0.2.1.7

15−- data-interval-1.2.0

16−- extended-reals-0.2.1.0

17−- gnuplot-0.5.4.1

18−- text-1.2.2.1

19−- aeson-1.2.1.00

20−- deferred-folds-0.6.12

21−- rset-1.0.0

22−

23−# Override default flag values for local packages and extra-deps

24−flags: {}

25−

26−# Extra package databases containing global packages

27−extra-package-dbs: []

28−

29−# Control whether we use the GHC we find on the path

30−# system-ghc: true

31−

32−# Require a specific version of stack, using version ranges

33−# require-stack-version: -any # Default

34−# require-stack-version: >= 1.0.0

35−

36−# Override the architecture used by stack, especially useful on Windows

37−# arch: i386

38−# arch: x86_64

39−

40−# Extra directories used by stack for building

41−# extra-include-dirs: [/path/to/dir]

42−# extra-lib-dirs: [/path/to/dir]

43−

44−# Allow a newer minor version of GHC than the snapshot specifies

45−# compiler-check: newer-minor

46−

47−allow-newer: true

48−

stack.ghc8.6.yaml+4

@@ -28,6 +28,10 @@ extra-deps:

2828- prettyprinter-1.7.0

2929- prettyprinter-ansi-terminal-1.1.2

3030- fusion-plugin-types-0.1.0

31+- base16-bytestring-1.0.1.0

32+- modern-uri-0.3.4.0

33+- cryptohash-sha256-0.11.102.0

34+- http-types-0.12.3

3135

3236# Override default flag values for local packages and extra-deps

3337flags:

stack.ghc8.8.yaml+2

@@ -30,6 +30,8 @@ extra-deps:

3030- primitive-unlifted-0.1.3.0

3131- prettyprinter-1.7.0

3232- prettyprinter-ansi-terminal-1.1.2

33+- base16-bytestring-1.0.1.0

34+- modern-uri-0.3.4.0

3335

3436# Override default flag values for local packages and extra-deps

3537flags:

test/TutorialD/Interpreter.hs+1−1

@@ -362,7 +362,7 @@ testComments :: Test

362362testComments = TestCase $ do

363363 (sessionId, dbconn) <- dateExamplesConnection emptyNotificationCallback

364364 mapM_ (executeTutorialD sessionId dbconn) [

365− ":branch testbranch --test comment\n",

365+ ":branch testbranch --test comment",

366366 ":jumphead {- test comment -} master"]

367367

368368-- create a graph and query from two disparate contexts

test/TutorialD/Interpreter/Import/TutorialD.hs+16−4

@@ -2,19 +2,21 @@ import Test.HUnit

22import ProjectM36.Base

33import TutorialD.Interpreter.Import.TutorialD

44import System.Exit

5+import qualified Data.Text as T

56import System.IO.Temp

67import qualified Data.Map as M

78import System.IO

89import qualified Data.ByteString as BS

910import qualified Data.Text.Encoding as TE

11+import Text.URI

1012

1113main :: IO ()

1214main = do

13− tcounts <- runTestTT $ TestList [testTutdImport]

15+ tcounts <- runTestTT $ TestList [testTutdFileImport, testTutdHTTPSImport]

1416 if errors tcounts + failures tcounts > 0 then exitFailure else exitSuccess

1517

16−testTutdImport :: Test

17−testTutdImport = TestCase $

18+testTutdFileImport :: Test

19+testTutdFileImport = TestCase $

1820 withSystemTempFile "m.tutd" $ \tempPath handle -> do

1921 BS.hPut handle (TE.encodeUtf8 "x:=relation{tuple{a 5,b \"spam\"}}; y:=relation{tuple{b \"漢字\"}}")

2022 hClose handle

@@ -24,6 +26,16 @@ testTutdImport = TestCase $

2426 ("b", NakedAtomExpr $ TextAtom "spam")])]),

2527 Assign "y" (MakeRelationFromExprs Nothing

2628 $ TupleExprs () [TupleExpr (M.fromList [("b", NakedAtomExpr (TextAtom "漢字"))])])]

27− imported <- importTutorialD tempPath

29+ fileURI <- mkURI (T.pack ("file://" <> tempPath))

30+ imported <- importTutorialDFromFile fileURI Nothing

2831 assertEqual "import tutd" (Right expectedExpr) imported

2932

33+testTutdHTTPSImport :: Test

34+testTutdHTTPSImport = TestCase $ do

35+ uri <- mkURI "https://raw.githubusercontent.com/agentm/project-m36/master/test/TutorialD/Interpreter/Import/httpimporttest.tutd"

36+ let hash = "effe32b247586dc3ac0079fc241b9618d41d189afcaeb7907edbe5a8b45992a4"

37+ expected = Right (MultipleExpr [Assign "x" (RelationVariable "true" ()),Assign "y" (RelationVariable "false" ())])

38+ actual <- importTutorialDViaHTTP uri (Just hash)

39+ assertEqual "github https" expected actual

40+

41+

test/scripts.hs+5−2

@@ -4,6 +4,8 @@ import TutorialD.Interpreter.Import.TutorialD

44import System.Directory

55import Data.List (isSuffixOf)

66import System.Exit

7+import qualified Data.Text as T

8+import Text.URI

810testList :: IO Test

911testList = do

@@ -20,7 +22,8 @@ main = do

2022

2123testScript :: FilePath -> Test

2224testScript tutdFile = TestCase $ do

23− eImport <- importTutorialD tutdFile

25+ fileURI <- mkURI (T.pack ("file://" <> tutdFile))

26+ eImport <- importTutorialDFromFile fileURI Nothing

2427 case eImport of

2528 Left err -> assertFailure ("tutd import failure in " ++ tutdFile ++ ": " ++ show err)

26− Right _ -> pure ()

29+ Right _ -> pure ()