on master help sign in
update docs to prepare for release of out-of-the-tarpit essay
commit054da5ac3231cc235889ff24281f10da4a546298 authorAgentM <agentm@themactionfaction.com> dateFri 1 Apr 2016, 23:44 (11 years ago) parentad7a300
4 files changed+405−15
README.markdown+3−11 docs/15_minute_tutorial.markdown+289 docs/reaching_out_of_the_tarpit.markdown+29−4 examples/out_of_the_tarpit.hs+84
README.markdown+3−11
@@ -25,6 +25,7 @@ Project:M36 supports multiple frontends which target different audiences.
25251. [Installation and Introduction to Project:M36](docs/introduction_to_projectm36.markdown)
26261. [Introduction to the Relational Algebra](docs/introduction_to_the_relational_algebra.markdown)
27271. [TutorialD Tutorial](docs/tutd_tutorial.markdown)
28+1. [15 Minute Tutorial](docs/15_minute_tutorial.markdown)
28291. [Transaction Graph Operators](docs/transaction_graph_operators.markdown)
29301. [On NULL (in SQL)](docs/on_null.markdown)
30311. [Persistent Library Driver](docs/persistent_library_driver.markdown)
@@ -33,6 +34,7 @@ Project:M36 supports multiple frontends which target different audiences.
33341. [ACID Database Properties](docs/acid_assessment.markdown)
34351. [Serving Remote ProjectM36 Databases](docs/server_mode.markdown)
35361. [Using Notifications](docs/using_notifications.markdown)
37+1. [Reaching "Out of the Tarpit" with Project:M36](docs/reaching_out_of_the_tarpit.markdown)
3638
3739## Development
3840
@@ -44,14 +46,4 @@ Important modules:
44462. Relational Expressions - represents a composable abstract syntax tree for the relational algebra
45473. Tutorial D Interpreter - one available frontend to evaluate relational expressions
46484. ProjectM36.Client- user-facing native Haskell library
47−
48−Other modules include:
49−
50−* displaying a relation as HTML
51−* displaying a relation in the terminal
52−* displaying a relation using gnuplot
53−* relational expression static optimizer
54−* relational expression cost-based optimizer
55−* relational error enumeration
56−* import of CSV as a relation and export of a relation to CSV
57−* import of TutorialD from files
49+5. Database.Persist.ProjectM36 - persistent library database driver
docs/15_minute_tutorial.markdownnew file+289
@@ -0,0 +1,289 @@
1+# Project:M36 in 15 Minutes
2+
3+Thank you for evaluating Project:M36, a new relational algebra engine written in Haskell. This tutorial will give you a general overview of the features of the database management system (DBMS). This tutorial assumes basic knowledge of your terminal's shell.
4+
5+## Prerequisites
6+
7+* any Linux
8+* GHC (Haskell compiler)
9+* cabal (package downloader) installed
10+
11+## Download and Install
12+
13+```bash
14+git clone https://github.com/agentm/project-m36.git
15+cabal sandbox init
16+cabal install --dependencies-only
17+cabal run tutd
18+```
19+
20+## Some Basic TutorialD
21+
22+When you run the ```tutd``` interpreter, you are interacting with an in-memory database using a dialect of the TutorialD language which is described in numerous book by Chris J. Date. It is a language specifically designed for human interaction with the relational algebra.
23+
24+Note that TutorialD is strongly-typed- no type coercions take place automatically. For example, an integer will never be interpreted as a string or vice versa.
25+
26+### Load the Canonical Chris Date Example
27+
28+```
29+TutorialD (master): :importtutd "scripts/DateExamples.tutd"
30+```
31+
32+```:importtutd``` (do not skip typing the preceding colon), loads a file which contains a script written in the TutorialD dialect used by Project:M36 and executes it to change the state of the database. In this case, the relations "s" (supplier), "p" (product), and "sp" (suppliers-products) are now loaded.
33+
34+### Execute Relational Expressions
35+
36+```
37+TutorialD (master): :showexpr s
38+┌──────────┬────────┬───────────┬───────────┐
39+│city::Text│s#::Text│sname::Text│status::Int│
40+├──────────┼────────┼───────────┼───────────┤
41+│"Paris" │"S2" │"Jones" │10 │
42+│"Paris" │"S3" │"Blake" │30 │
43+│"London" │"S1" │"Smith" │20 │
44+│"London" │"S4" │"Clark" │20 │
45+│"Athens" │"S5" │"Adams" │30 │
46+└──────────┴────────┴───────────┴───────────┘
47+```
48+
49+Use ```:showexpr``` (note the preceding colon again) followed by a TutorialD relational expression to execute the expression against the database's current state.
50+
51+A relational expression operates on the current database state (also known as "database context") and returns a relation containing the result. Note that relational expressions do not modify the state of the database.
52+
53+You will need to [learn TutorialD](docs/tutd_tutorial.markdown) to create relational expressions here, but here are a few quick, simple examples:
54+
55+#### Joins
56+
57+```
58+TutorialD (master): :showexpr s join sp
59+┌──────────┬────────┬────────┬────────┬───────────┬───────────┐
60+│city::Text│p#::Text│qty::Int│s#::Text│sname::Text│status::Int│
61+├──────────┼────────┼────────┼────────┼───────────┼───────────┤
62+│"London" │"P3" │400 │"S1" │"Smith" │20 │
63+│"London" │"P4" │300 │"S4" │"Clark" │20 │
64+│"Paris" │"P2" │200 │"S3" │"Blake" │30 │
65+│"Paris" │"P1" │300 │"S2" │"Jones" │10 │
66+│"London" │"P5" │400 │"S4" │"Clark" │20 │
67+│"London" │"P6" │100 │"S1" │"Smith" │20 │
68+│"London" │"P5" │100 │"S1" │"Smith" │20 │
69+│"Paris" │"P2" │400 │"S2" │"Jones" │10 │
70+│"London" │"P4" │200 │"S1" │"Smith" │20 │
71+│"London" │"P1" │300 │"S1" │"Smith" │20 │
72+│"London" │"P2" │200 │"S1" │"Smith" │20 │
73+└──────────┴────────┴────────┴────────┴───────────┴───────────┘
74+```
75+
76+All joins in TutorialD are similar to "NATURAL JOIN" in SQL.
77+
78+#### Restriction
79+
80+```
81+TutorialD (master): :showexpr s where status=20
82+┌──────────┬────────┬───────────┬───────────┐
83+│city::Text│s#::Text│sname::Text│status::Int│
84+├──────────┼────────┼───────────┼───────────┤
85+│"London" │"S1" │"Smith" │20 │
86+│"London" │"S4" │"Clark" │20 │
87+└──────────┴────────┴───────────┴───────────┘
88+```
89+
90+Restrictions are similar to the "WHERE" clause in SQL.
91+
92+#### Projection
93+
94+```
95+TutorialD (master): :showexpr p{color,p#,weight}
96+┌───────────┬────────┬───────────┐
97+│color::Text│p#::Text│weight::Int│
98+├───────────┼────────┼───────────┤
99+│"Red" │"P6" │19 │
100+│"Red" │"P1" │12 │
101+│"Blue" │"P5" │12 │
102+│"Green" │"P2" │17 │
103+│"Red" │"P4" │14 │
104+│"Blue" │"P3" │17 │
105+└───────────┴────────┴───────────┘
106+```
107+
108+Projection is equivalent to "SELECT <columns>" in SQL.
109+
110+#### Attribute Rename
111+
112+```
113+TutorialD (master): :showexpr s rename {city as town}
114+┌────────┬───────────┬───────────┬──────────┐
115+│s#::Text│sname::Text│status::Int│town::Text│
116+├────────┼───────────┼───────────┼──────────┤
117+│"S2" │"Jones" │10 │"Paris" │
118+│"S3" │"Blake" │30 │"Paris" │
119+│"S4" │"Clark" │20 │"London" │
120+│"S1" │"Smith" │20 │"London" │
121+│"S5" │"Adams" │30 │"Athens" │
122+└────────┴───────────┴───────────┴──────────┘
123+```
124+
125+Renaming attributes is equivalent to "SELECT city AS town..." in SQL.
126+
127+#### Relational "true" and "false"
128+
129+The relational algebra includes two fundamental named relations: "true" and "false". (Chris Date calls them "TABLE_DUM" and "TABLE_DEE" which are poor names.)
130+
131+"true" is the name of the relation with no attributes and a set of one tuple of no attributes.
132+
133+```
134+TutorialD (master): :showexpr true
135+┌┐
136+││
137+├┤
138+└┘
139+```
140+
141+"false" is the name of the relation with no attributes and an empty tuples set.
142+
143+```
144+TutorialD (master): :showexpr false
145+┌┐
146+││
147+└┘
148+```
149+
150+These relations cannot be represented in SQL databases but are fundamental to the mathematics of the relational algebra. Any relation projected against no attributes will return "true" if the relation contained any tuples or "false" otherwise.
151+
152+### Execute Database Context Expressions
153+
154+Database context expressions change the state of the database, much like "CREATE TABLE", "INSERT", "UPDATE", or "DELETE" in SQL. Database expressions in the interpreter are *not* prefixed by a colon because they are the default operation type.
155+
156+#### Create a Relation Variable
157+
158+```
159+TutorialD (master): x:=relation{name Text, age Int}{tuple{name "Steve", age 40},tuple{name "Mike", age 31}}
160+TutorialD (master): :showexpr x
161+┌────────┬──────────┐
162+│age::Int│name::Text│
163+├────────┼──────────┤
164+│40 │"Steve" │
165+│31 │"Mike" │
166+└────────┴──────────┘
167+```
168+
169+This is similar to SQL's "CREATE TABLE" operator.
170+
171+#### Insert Additional Tuples
172+
173+```
174+TutorialD (master): x:=relation{name Text, age Int}{tuple{name "Steve", age 40},tuple{name "Mike", age 31}}
175+TutorialD (master): :showexpr x
176+┌────────┬──────────┐
177+│age::Int│name::Text│
178+├────────┼──────────┤
179+│40 │"Steve" │
180+│31 │"Mike" │
181+└────────┴──────────┘
182+TutorialD (master): insert x relation{tuple{name "Bob", age 24}}
183+TutorialD (master): :showexpr x
184+┌────────┬──────────┐
185+│age::Int│name::Text│
186+├────────┼──────────┤
187+│40 │"Steve" │
188+│24 │"Bob" │
189+│31 │"Mike" │
190+└────────┴──────────┘
191+```
192+
193+Note that the relational insertion is identical to "x:=x union relation{...}".
194+
195+#### Update Tuples in a Relation Variable
196+
197+```
198+TutorialD (master): :showexpr s
199+┌──────────┬────────┬───────────┬───────────┐
200+│city::Text│s#::Text│sname::Text│status::Int│
201+├──────────┼────────┼───────────┼───────────┤
202+│"Paris" │"S2" │"Jones" │10 │
203+│"Paris" │"S3" │"Blake" │30 │
204+│"London" │"S1" │"Smith" │20 │
205+│"London" │"S4" │"Clark" │20 │
206+│"Athens" │"S5" │"Adams" │30 │
207+└──────────┴────────┴───────────┴───────────┘
208+TutorialD (master): update s where city="London" (status:=35)
209+TutorialD (master): :showexpr s
210+┌──────────┬────────┬───────────┬───────────┐
211+│city::Text│s#::Text│sname::Text│status::Int│
212+├──────────┼────────┼───────────┼───────────┤
213+│"London" │"S4" │"Clark" │35 │
214+│"London" │"S1" │"Smith" │35 │
215+│"Paris" │"S2" │"Jones" │10 │
216+│"Paris" │"S3" │"Blake" │30 │
217+│"Athens" │"S5" │"Adams" │30 │
218+└──────────┴────────┴───────────┴───────────┘
219+```
220+#### Deleting Tuples from a Relation Variable
221+
222+```
223+TutorialD (master): :showexpr s
224+┌──────────┬────────┬───────────┬───────────┐
225+│city::Text│s#::Text│sname::Text│status::Int│
226+├──────────┼────────┼───────────┼───────────┤
227+│"Paris" │"S2" │"Jones" │10 │
228+│"Paris" │"S3" │"Blake" │30 │
229+│"London" │"S1" │"Smith" │20 │
230+│"London" │"S4" │"Clark" │20 │
231+│"Athens" │"S5" │"Adams" │30 │
232+└──────────┴────────┴───────────┴───────────┘
233+TutorialD (master): delete s where sname="Adams"
234+TutorialD (master): :showexpr s
235+┌──────────┬────────┬───────────┬───────────┐
236+│city::Text│s#::Text│sname::Text│status::Int│
237+├──────────┼────────┼───────────┼───────────┤
238+│"Paris" │"S2" │"Jones" │10 │
239+│"Paris" │"S3" │"Blake" │30 │
240+│"London" │"S1" │"Smith" │20 │
241+│"London" │"S4" │"Clark" │20 │
242+└──────────┴────────┴───────────┴───────────┘
243+```
244+
245+### Execute Transaction Graph Expressions
246+
247+Unlike typical SQL DBMSs, Project:M36 maintains a graph of transactions, similar to how git stores source patches. This allows the database context to be branched (with a name), merged, and enables recalling the database context from past transactions.
248+
249+#### Committing
250+
251+```
252+TutorialD (master): :commit
253+```
254+
255+#### Rollback
256+
257+```
258+TutorialD (master): :rollback
259+```
260+
261+#### Branching
262+
263+```
264+TutorialD (master): :branch experimental
265+TutorialD (experimental): employee:=relation{name Text, salary Int}
266+TutorialD (experimental): :commit
267+```
268+
269+Note how the prompt changes to reflect the current branch's name.
270+
271+#### Jump to Branch
272+
273+```
274+TutorialD (experimental): :jumphead master
275+TutorialD (master): :showexpr employee
276+ERR: RelVarNotDefinedError "employee"
277+```
278+
279+Now we have jumped back to the master branch and we can confirm that the "employee" relation variable is not present here despite being committed. It was, in fact, committed only on the "experimental" branch.
280+
281+#### Other operations
282+
283+Of course, Project:M36 supports [aggregate operators](docs/tutd_tutorial.markdown#aggregate-queries), [value functions](docs/tutd_tutorial.markdown#extend), [database constraints](docs/tutd_tutorial.markdown#constraints) including candidate and foreign keys, and other operators not mentioned in this brief tutorial.
284+
285+## Conclusion
286+
287+The TutorialD dialect used by Project:M36 is ideal for learning about the relational algebra because it is quick to write and grammatically unambiguous.
288+
289+The TutorialD language is not, however, suitable for production use because such use would be susceptible to the same string injection attacks which plague SQL. For integration with middleware, please try the [persistent library driver](docs/persistent_library_driver.markdown) or [the native Haskell interface](docs/projectm36_client_library.markdown).
docs/reaching_out_of_the_tarpit.markdown+29−4
@@ -53,14 +53,23 @@ Project:M36 implements multiple interfaces:
5353* a native Haskell API
5454* a database driver for the [persistent framework](http://www.yesodweb.com/book/persistent)
5555
56−To understand how Project:M36 stacks up against the fictional programming language shown in the paper, we have implemented the example from the paper.
56+### Real Estate Database Example
5757
58−feeders
59−observers
58+To understand how Project:M36 stacks up against the fictional programming language shown in the paper, we have implemented the [example from the paper](examples/out_of_the_tarpit.tutd). The language used in the script is a dialect of Chris Date's TutorialD language.
59+
60+### Feeders and Observers
61+
62+The "Out of the Tarpit" paper distinguishes between pushing data to the database and pulling data out of the database ([p.46](http://shaffner.us/cs/papers/tarpit.pdf)): "feeders" continually update the state of the database to best reflect reality and "observers" request notifications on relational state changes and take action on them.
63+
64+SQL developers are already intimately familiar with "feeders"- these are merely INSERT and UPDATE statements issues to the database via some middleware.
65+
66+However, most database do not support and adequate notion of observers. The purpose of an observer is to notify interested parties about relevant changes to the state of the database so that clients relying on the database are viewing the most "up-to-date" information from the database. Project:M36 implements this requirement by allowing clients to register relational expressions by name and return relational expression. When the result of the relational expression changes in a tracked state (branch) of the database, an asynchronous notification including the name and the result of return relational expression is sent to the client. The client can then use this information to update its own user interface or take further automated actions.
67+
68+Some SQL databases include a similar feature; in PostgreSQL, this is activated by [LISTEN](http://www.postgresql.org/docs/9.0/static/sql-listen.html)/[NOTIFY](http://www.postgresql.org/docs/9.0/static/sql-notify.html). However, PostgreSQL offers no provision for triggering notifications based on the state of the database- this crucial feature is left to the database developer. Furthermore, the return payload can only be a string, not the result of a relational expression evaluation.
6069
6170## Good Riddance to Old Baggage
6271
63−Project:M36 is a clean-room implementation of a relational algebra engine; it does not implement SQL or any other legacy standard. While this means that Project:M36 is incompatible with other RDBMS, it means that Project:M36 can avoid the substantial mistakes of previous implementations, such as:
72+Project:M36 is a clean-room implementation of a relational algebra engine; it does not implement SQL or any other legacy standard. While this means that Project:M36 is incompatible with other RDBMS, it means that Project:M36 can avoid the substantial pitfalls of previous implementations, such as:
6473
6574* [three-valued logic (NULL)](docs/on_null.markdown)
6675* non-relational database constructs such as row and column ordering or OUTER JOINs
@@ -68,3 +77,19 @@ Project:M36 is a clean-room implementation of a relational algebra engine; it do
6877* direct table-to-file mapping which create perverse model-data dependencies
6978* key-value storage: a key-value mapping is valued merely for its performance characteristics, not because it models reality well
7079* restrictive, legacy type systems which impede modeling
80+* unnecessarily complex SQL grammar
81+
82+## Future Directions
83+
84+Project:M36 is just getting started as an ambitious open-source project- please join us! Immediate future features include:
85+
86+* transaction graph branch merging (never bother with "master-to-master" conflicts again)
87+* making maximum use of parallelization
88+* static and statistics-based optimizations
89+* multiple, simultaneous storage representations for relations
90+* more network access methods, including a WebSocket driver
91+* support for relations with infinitely many tuples
92+
93+## Try It!
94+
95+To learn more about Project:M36, try the [15 minute tutorial](docs/15_minute_tutorial.markdown).
examples/out_of_the_tarpit.hsnew file+84
@@ -0,0 +1,84 @@
1+-- the Out-of-the-Tarpit example in Haskell and Project:M36
2+{-# LANGUAGE OverloadedStrings, DeriveAnyClass #-}
3+import ProjectM36.Client
4+import qualified Data.Text as T
5+import ProjectM36.Atom
6+import Data.Proxy
7+
8+type Text = T.Text
9+
10+--custom types cannot yet be made due to hardcoding in the Atom Binary implementation
11+type Address = Text
12+type Agent = Text
13+type Name = Text
14+type Price = Double
15+type FileName = Text
16+data RoomType = Kitchen | Bathroom | LivingRoom deriving (Atomable)
17+roomTypeAtomType = atomTypeForProxy (Proxy :: RoomType)
18+
19+data PriceBand = Low | Medium | High | Premium
20+priceBandAtomType = atomTypeForProxy (Proxy :: PriceBand)
21+
22+data AreaCode = City | Suburban | Rural
23+areaCodeAtomType = atomTypeForProxy (Proxy :: AreaCode)
24+
25+data SpeedBand = VeryFastBand | FastBand | MediumBand | SlowBand
26+speedBandAtomType = atomTypeForProxy (Proxy :: SpeedBand)
27+
28+
29+main :: IO ()
30+main = do
31+ let connInfo = InProcessConnectionInfo emptyNotificationCallback
32+ check x = case x of
33+ Left err -> error (show err)
34+ Right x' -> x'
35+ eConn <- connectProjectM36 connInfo
36+ let conn = check eConn
37+
38+ eSessionId <- createSessionAtHead "master" conn
39+ let sessionId = check eSessionId
40+
41+ createSchema sessionId conn
42+
43+createSchema :: Connection -> IO ()
44+createSchema conn = do
45+ let propertyAttrs = [Attribute "address" addressAtomType,
46+ Attribute "price" priceAtomType
47+ Attribute "photo" fileNameAtomType,
48+ Attribute "dateRegistered" dateAtomType]
49+ offerAttrs = [Attribute "address" addressAtomType,
50+ Attribute "offerPrice" priceAtomType,
51+ Attribute "offerDate" dateAtomType,
52+ Attribute "bidderName" nameAtomType,
53+ Attribute "bidderAddress" addressAtomType,
54+ Attribute "decisionDate" dateAtomType,
55+ Attribute "accepted" boolAtomType]
56+ decisionAttrs = [Attribute "address" addressAtomType,
57+ Attribute "offerDate" dateAtomType,
58+ Attribute "bidderName" nameAtomType,
59+ Attribute "bidderAddress" addressAtomType,
60+ Attribute "decisionDate" dateAtomType,
61+ Attribute "accepted" boolAtomType]
62+ roomAttrs = [Attribute "address" addressAtomType,
63+ Attribute "roomName" stringAtomType,
64+ Attribute "width" doubleAtomType,
65+ Attribute "breadth" doubleAtomType,
66+ Attribute "type" roomTypeAtomType]
67+ floorAttrs = [Attribute "address" addressAtomType,
68+ Attribute "roomName" stringAtomType,
69+ Attribute "floor" intAtomType]
70+ commissionAttrs = [Attribute "priceBand" priceBandAtomType,
71+ Attribute "areaCode" areaCodeAtomType,
72+ Attribute "saleSpeed" speedBandAtomType,
73+ Attribute "commission" doubleAtomType]
74+ relvarMap = [("property", propertyAttrs),
75+ ("offer", offerAttrs),
76+ ("decision", decisionAttrs),
77+ ("room", roomAttrs),
78+ ("floor", floorAttrs),
79+ ("commission", commissionAttrs)]
80+ dbcontextExprs = map (\(name, attrs) -> Define name (attributesFromList attrs)) relvarMap
81+ mapM executeDatabaseContextExpr dbcontextExprs
82+
83+
84+