on master help sign in
add basic, but better organized markdown-based documentation
commite8c5c06a632a8685c3ab21fb7adc48c84f091e6f authorAgentM <agentm@themactionfaction.com> dateSun 4 Oct 2015, 22:09 (11 years ago) parent16ef0ed
7 files changed+938
docs/committed_database_state.dot+7 docs/index.markdown+6 docs/initial_database_state.dot+5 docs/introduction_to_projectm36.markdown+101 docs/introduction_to_the_relational_algebra.markdown+341 docs/transaction_graph_operators.markdown+47 docs/tutd_tutorial.markdown+431
docs/committed_database_state.dotnew file+7
@@ -0,0 +1,7 @@
1+digraph G {
2+ base[label="base transaction",shape=rectangle];
3+ discon[label="mutable database context"];
4+ fresh[label="newly committed transaction",shape=rectangle];
5+ fresh -> base;
6+ discon -> fresh;
7+}
docs/index.markdownnew file+6
@@ -0,0 +1,6 @@
1+# Contents
2+1. [Introduction to the Relational Algebra](introduction_to_the_relational_algebra.markdown)
3+1. [Introduction to ProjectM36](introduction_to_projectm36.markdown)
4+1. [TutorialD Tutorial](tutd_tutorial.markdown)
5+1. [Persistent Library Driver](persistent_library_driver.markdown)
6+1. [ProjectM36.Client Library](projectm36_client_library.markdown)
docs/initial_database_state.dotnew file+5
@@ -0,0 +1,5 @@
1+digraph G {
2+ base[label="base transaction",shape=rectangle];
3+ new[label="mutable database context"];
4+ new -> base;
5+}
docs/introduction_to_projectm36.markdownnew file+101
@@ -0,0 +1,101 @@
1+# Introduction to Project:M36
2+
3+## What is Project:M36?
4+Project:M36 is a relational-algebra-based database management system. It can be used a production-level database backend or for learning more about the relational algebra.
5+
6+The principled mathematical approach to Project:M36's design sets it apart from competing DBMSs. Specifically, Project:M36 adheres strongly to the mathematics of the relational algebra and makes no compromises nor apologies in doing so. To that end, Project:M36 is implemented in the programming language "Haskell" which is itself based on the lambda calculus.
7+
8+## The Relational Algebra Connection
9+
10+The relational algebra is the mathematical model underpinning Project:M36. When compared to other "relational" DBMSs, by strictly adhering to the relational algebra, Project:M36 offers:
11+
12+* predictable, type-safe composability of relational operators
13+* more opportunities for interesting optimizations
14+* elimination of odd and easy-to-forget corner cases (such as found when dealing with SQL's NULL)
15+
16+## Software Dependencies & Installation
17+
18+Requirements:
19+
20+* [GHC 7.10 or greater](https://www.haskell.org/downloads)
21+* cabal-install 1.20 or greater (normally part of the Haskell installation
22+* POSIX filesystem
23+
24+Compilation steps:
25+
26+* git clone https://github.com/agentm/project-m36
27+* cd project-m36
28+* cabal sandbox init
29+* cabal install
30+
31+At this point, the TutorialD interactive interpreter can be run using ```cabal run tutd```.
32+
33+
34+## Accessing Project:M36 Databases
35+
36+Project:M36 implements a generic relational algebra DBMS backend with multiple data access frontends. Such frontends currently include:
37+
38+* TutorialD: Chris Date's interactive language for learning about the relational algebra
39+* ProjectM36.Client: a Haskell library for native access
40+* Database.Persist.ProjectM36: a driver for the Haskell persistent library
41+
42+More frontends are expected to be supported in the future.
43+
44+## Beyond the Relational Algebra
45+
46+In addition to being a relational algebra engine, Project:M36 supports features which one has come to expect from a modern DBMS.
47+
48+### Transaction Graph
49+
50+Most DBMSs provide a way to allow the user to specify atomic database state changes. These changes are often wrapped up as "transactions" which, when applied to the database state, offer the illusion of sequential and grouped, atomic changes applied to the databases. Often the only state which can be queried is the snapshot at the time the transaction is opened.
51+
52+Project:M36 extends this transaction model with the "transaction graph". This feture can be adequately described as version control for transactions. In the legacy DBMS model, the transactions are added to a linear transaction stream where each client is contending over the "head" of the stream- the latest transactions. In the transaction graph model, clients can add named branches and add transactions to the branches, then merge changes back, if necessary.
53+
54+This feature is useful, for example:
55+* during testing- create a testing branch which production will never see. This obviates the need for a separate production data set.
56+* during A/B testing- deploy a feature to some users, but not all.
57+* for implementation experimentation- try new features against production data without fear of data destruction
58+* for time travel- jump to transactions in the past and examine the state of the database
59+* for point-in-time recovery- undo an unlimited number of transactions to a known good state
60+* for multi-master replication- merge potentially-conflicting databases
61+
62+Naturally, this design implies that data is never deleted because all transactions are saved, but the graph can be compressed and pruned. The cost in complexity of this feature is merely in defining merge strategies and is an area of ongoing research.
63+
64+### Fine-grained Durability
65+
66+In any application, some data is more important than other data. Many DBMSs, however, make it difficult to separate such data and, thus, all data is equally expensive to save and retain. This often leads the database designer to leave certain data out of the database or store it in a second DBMS.
67+
68+Project:M36 allows the database client to specify what level of durability safety is required on a per-transaction basis. For example, if a specific transaction has less essential data, the transaction can be marked and committed to the database at a later time. However, further transactions on that branch must be marked with the same low durability level or the entire branch's durability is increased.
69+
70+Decreasing a transaction's durability can reduce IO contention at the cost of reduced certainty that the transaction will be restored if the DBMS software or hardware fails.
71+
72+### Native Haskell Compatibility
73+
74+Any Haskell data type which implements the ```Atom``` typeclass can be manipulated as a value in the database.
75+
76+Haskell functions which operate on those values can also be added to the database.
77+
78+### Parallelism
79+
80+Project:M36 benefits from Haskell's excellent support for parallelism. Because every relational algebra query is logically pure in the referential transparency sense, every query can take advantage of multiple CPU cores. In the future, distributed query execution will also be possible.
81+
82+### Function-based Relations
83+
84+Relations can also be defined by functions, thereby taking effectively no storage to represent and no IO to generate or query. Any function which generates supported values without duplicates can be represented as a relation, including functions which generate an potentially infinite number of values.
85+
86+### Static- and Cost-based Optimizers
87+
88+The relational algebra allows all optimizations to be written as algebraic transformations. The query execution graph is represented by the same algebraic constructs as the original query.
89+
90+### CSV support
91+
92+Though the CSV ("comma-separated values") format cannot represent all possible relations, it is nonetheless a basic format for exchange of tabular data. Project:M36 supports import and export of relations as CSV data whenever possible.
93+
94+### Relations as Plots
95+
96+N-attributed relations can be visualized as plots where each tuple represents a point in the plot. Project:M36 can generate gnuplot graphs from 2- and 3-attributed relations.
97+
98+
99+
100+
101+
docs/introduction_to_the_relational_algebra.markdownnew file+341
@@ -0,0 +1,341 @@
1+# Introduction to the Relational Algebra
2+## What is the relational algebra?
3+### History
4+The relational algebra was originally developed by [E. F. Codd](https://en.wikipedia.org/wiki/Edgar_F._Codd) working at IBM in the late 1960s as a proposed method by which database management system could present a unified interface to organization and querying of data. This relational model has been refined over the decades since and represents the state-of-the-art for modeling data used in computerized information processing.
5+
6+### Definition
7+As implied by the name "algebra", the relational algebra is a mathematical model for modeling data and the links between data; it is not a specific software design or software product. A specific database management system may choose to label itself "relational", but this need not imply that it adheres to the relational algebra, rather, many existing "relational" software products find a medium ground with legacy features. SQL-based database management systems in particular do not adhere to the relational algebra.
8+
9+The purpose of a database is to model a small sliver of reality. Data present in the database make boolean logic statements about the nature of our reality, while data not present in the database makes statements to the contrary. This is called the "closed world assumption". For example, if we have a database of employees, the database can unequivocally state whether or not "Steve" is an employee. If that answer does not align with reality, that is because the database's notion of the current state has not been updated. After all, a database can only model reality- it cannot be directly tied into reality.
10+
11+A relational algebra system must support the notion of relations and relational operators. What makes the relational model "algebraic" is the ability to compose the relational operators to make arbitraily complex operator expressions.
12+
13+Let us begin by starting with the smallest units in the relational model and work upwards.
14+
15+#### Value
16+
17+A value is a typed, singular unit of data.
18+
19+Common examples could be a number indicating a person's age, the number of dollars in a bank account, or a street address used for delivery. In all cases, a value must be of a specific type and must support the notion of equality.
20+
21+As an example, it doesn't make sense to ask if a person's age is equal to a street address; the relational model can enforce this- this is called "type safety" and is the motivation for using types.
22+
23+Note that a value can also be a relation. This creates a recursively-defined data structure. More on this later.
24+
25+Examples:
26+* A number:
27+
28+|3|
29+|-|
30+
31+* A name:
32+
33+|Adam|
34+|----|
35+
36+### Attribute
37+
38+An attribute is a pair containing an attribute name along with an attribute type. An attribute name allows the relational model to uniquely identify values in a tuple or relation, therefore the attribute name must be unique within a relation and tuple.
39+
40+Examples:
41+* An "Address" attribute name is of type "Text":
42+
43+(Address, Text)
44+
45+* An "Current Balance" attribute name is of type "Dollar Amount":
46+
47+(Current Balance, Dollar Amount)
48+
49+#### Tuple
50+
51+A tuple is a set of attribute names mapped to values. The tuple represents a statement of "truth" in the database. For example, a tuple containing ((Name, Bob),(Address, 123 Main St.),(Age, 45)) could be used to state the logical proposition "A person named Bob lives at 123 Main St. and is 45 years old." However, that is not the only interpretation; the same tuple could be used to represent the proposition "a pet tarantula named Bob is treated by a veterinarian who was aged 45". Current database management systems make no attempt to interpret or store the propositions, so the interpretation is at the sole discretion of the human modeler.
52+
53+Examples:
54+* "Cindy deposited $45.30 into account C at 4:15 PM":
55+
56+((Name, Cindy), (Amount, +$43.30), (Time, 4:15 PM))
57+
58+* "A car with license plate XYZ-123 failed the inspection due to rust":
59+
60+((License Number, XYZ-123), (Reason, Rust))
61+
62+In the last example, the fact that the car failed the inspection is implied by the existence of the tuple in a relation.
63+
64+#### Relation
65+
66+A relation is a container of values organized by tuples of the same type. Specifically, a relation is a set of attributes (a relation header) mapped over a set of tuples (a relation body). Note that this implies that tuples cannot appear more than once since a set can only hold unique items.
67+
68+When a tuple appears in a relation, a relation's attributes and respective types must match that of all of its tuples. Thus, a relation's header summarizes the attribute names and types of all of its constituent tuples.
69+
70+If a tuple represents a logical proposition, then a relation represents a logical predicate. Thus, a relation contains all the propositions born from the predicate which are true. The absence of a tuple in a relation implies a false proposition.
71+
72+Since both header and body are sets of attributes and tuples, respectively, neither the header nor body have any ordering. However, when we approximate a relation, we must choose an arbitrary ordering.
73+
74+A relation can also be thought of as an n-dimensional graph (each dimension defined by an attribute), where each value represents one dimension of a point in the graph.
75+
76+Examples:
77+* Predicate: "A supplier SNO supplies part PNO in quantity QTY"
78+* Propositions: "Supplier S1 supplies part P1 in quantity 300", "Supplier S2 supplies part P1 in quantity 250"
79+
80+|(SNO, Supplier Number Type)|(PNO,Parts Number Type)|(QTY, Quantity Type)|
81+|---|---|---|
82+|S1 |P1 |300|
83+|S2 |P1 |250|
84+
85+* Predicate: "A person named Name is an employee and works in department Department."
86+* Propositions: "A person named Steve is an employee and works in the Accounting department."
87+
88+|(Name, Name Type)|(Department, Department Type)|
89+|-|-|
90+|Steve|Accounting|
91+
92+Note that the absence of a tuple representing "Bob" indicates that Bob is not an employee and not that, for example, Bob could be an employee in an unspecified department.
93+
94+### Relation Variables
95+
96+In order to reference relations with a specific, albeit arbitrary name, relation variables map such names to relations, much like an integer variable references a specific, immutable integer. Relation variables vary over state changes, thus a single relation variable may reference an arbitrary number of immutable relations over its lifetime. However, each relation variable can only reference relations of the same type. The type of a relation can be represented by a relation's header (its set of attributes).
97+
98+When a relation variable is changed, it references a new, immutable relation; relations themselves cannot be modified/mutated.
99+
100+Examples:
101+* At state T1, Bob is an employee. At state T2, Bob and Steve are employees.
102+
103+T1: (Employees, (Name, Name Type)) ->
104+
105+|(Name, Name Type)|
106+|-|
107+|Bob|
108+
109+T2: (Employees, (Name, Name Type)) ->
110+
111+|(Name, Name Type)|
112+|-|
113+|Bob|
114+|Steve|
115+
116+To reiterate, the transition from T1 to T2 replaced the "Employees" relation variable reference found at T1 with the reference to the new relation found at T2.
117+
118+#### Relational Operators
119+
120+Since we refer to the relational algebra as a closed algebra, all relational operators necessarily operate on and generate relations. The purpose of the operators is allow data modelers to answer questions (queries) based on the current state of the database. For example, a modeler may wish to know the ages of the employees working in the accounting department. The modeler can then compose the relational operators together to answer this and virtually any other conceivable query.
121+
122+##### Project
123+
124+The projection operator's input is a set of attribute names and a relation and outputs the relation with the input set of attribute names. Logically, projection removes attributes from a relation (header and body).
125+
126+Example:
127+* What are the names of all my employees?
128+
129+Project ("Name") in
130+
131+|(Name, NameType)|(Department, Department Type)|(Room Number, Room Number Type|
132+|-|-|-|
133+|Steve|Marketing|445|
134+|Bob|Printing|112|
135+
136+===>
137+
138+|(Name, NameType)|
139+|-|
140+|Steve|
141+|Bob|
142+
143+##### Restrict
144+
145+The restriction operator's input is a relation along with a predicate whose purpose is to filter the relation body's tuple set. This operator answers queries such as "which employees work in the marketing department?" or "which employees' names start with the letter 'S'?"
146+
147+Examples:
148+* Find all fruit products.
149+
150+Restrict (tuples where attribute "Fruit" is true) in
151+
152+|(Product Name, Product Name Type)|(Is Fruit, Fruit Boolean Flag)|(Price, Price Type)|
153+|-|-|-|
154+|Orange|True|$2.50|
155+|Pineapple|True|$5.00|
156+|Cucumber|False|$3.00|
157+
158+===>
159+
160+|(Product Name, Product Name Type)|(Is Fruit, Fruit Boolean Flag)|(Price, Price Type)|
161+|-|-|-|
162+|Orange|True|$2.50|
163+|Pineapple|True|$5.00|
164+
165+##### Rename
166+
167+The rename operator takes a relation argument and a pair of attribute names (A1,A2) and returns a new relation with all appearances of attribute name A1 now appearing as attribute name A2.
168+
169+This operator is mostly useful in composing other relational operators together.
170+
171+Examples:
172+* Rename the "Age" attribute to be "Years Old".
173+
174+|(Name, Name Type)|(Age, Age Type)|
175+|-|-|
176+|Bob|42|
177+|Jen|55|
178+
179+===>
180+
181+|(Name, Name Type)|(Years Old, Age Type)|
182+|-|-|
183+|Jen|55|
184+|Bob|42|
185+
186+Note that rename does not alter the type of the attribute in question.
187+
188+##### Join
189+
190+One of the most essential-to-understand operators is the join operator. Unlike the previous two unary (single relation argument) operators, join operates on two (or potentially more) relations. The purpose of a join is to merge multiple relations into one relation in order to determine the relationships between data across relations. For example, there are good reasons (which have not yet been explained) why a relation representing a company's products and records of when those products were shipped should be two disparate relations. The join operators merges relation A and relation B by returning a new relation with the attribute sets of both relation headers merged and with a body of the tuples merged if-and-only-if the in-common attributes' values are equal.
191+
192+In the case where no attributes are in common between the relations, the join operator becomes a cross-product operator; this is not a special case, rather a logical consequence of the definition.
193+
194+Examples:
195+* Find all course information and their respective instructors' information. An instructor may teach more than one course, but each course has one instructor.
196+
197+Join (Instructor relation)
198+
199+|(Instructor Name, Instructor Name Type)|(Office Number, Office Number Type)|
200+|-|-|-|
201+|Steve|Office 45|
202+|Bob|Office 30|
203+
204+(Course Relation)
205+
206+|(Course Name, Course Name Type)|(Instructor, Instructor Name Type)|(Course Room Number, Room Number Type)|
207+|-|-|-|
208+|English 1|Steve|Room 104|
209+|French 2|Bob|Room 103|
210+|English 2|Steve|Room 104|
211+
212+["Instructor Name" is the in-common attribute]
213+===>
214+
215+|(Instructor Name, Instructor Name Type)|(Office Number, Office Number Type)|(Course Name, Course Name Type)|(Course Room Number, Room Number Type)|
216+|-|-|-|-|
217+|Steve|Office 45|English 1|Room 104|
218+|Steve|Office 45|English 2|Room 104|
219+|Bob|Office 30|French 2|Room 103|
220+
221+Note that "Steve" and his "Office 45" appear twice in the resultant relation. This is called "data duplication".
222+
223+##### Union
224+
225+The union operator takes two (or more) relations of the same type (relations with identical headers) and returns a new relation whose body is a sum of the tuples from both relations. Note that tuples cannot appear twice, so (A union A) always equals A.
226+
227+Examples:
228+* Find all parents and children.
229+
230+union (Parents relation)
231+
232+|(Name, Name Type)|(Age, Age Type)|
233+|-|-|
234+|Bob|45|
235+|Cindy|32|
236+
237+(Children relation)
238+
239+|(Name, Name Type)|(Age, Age Type)|
240+|-|-|
241+|Steve|12|
242+|Tom|11|
243+
244+===>
245+
246+|(Name, Name Type)|(Age, Age Type)|
247+|-|-|
248+|Steve|12|
249+|Cindy|32|
250+|Bob|45|
251+|Tom|11|
252+
253+##### Group
254+
255+##### Ungroup
256+
257+### Displaying Relations
258+
259+Note that in the above operator examples, the order of the tuples appears to shift around. Since a relation body is a set of tuples, any ordering, such as required to represent a relation in two-dimenstional space as a table, is completely arbitrary. Note that the following two relations are two representations of *same* relation:
260+
261+|(Name, Name Type)|(Age, Age Type)|
262+|-|-|
263+|Bob|40|
264+|Sam|41|
265+|Cindy|42|
266+
267+and
268+
269+|(Name, Name Type)|(Age, Age Type)|
270+|-|-|
271+|Sam|41|
272+|Cindy|42|
273+|Bob|40|
274+
275+The relation header is a set of attributes, so the attributes also have no ordering. Here is a third representation of the above relation:
276+
277+|(Age, Age Type)|(Name, Name Type)|
278+|-|-|
279+|42|Cindy|
280+|40|Bob|
281+|41|Sam|
282+
283+In general, an n-dimensional relation can be represented by an n-dimensional graph whereby each point in n-dimensional space is represented by each tuple.
284+
285+## Why is the relational algebra important?
286+
287+Unlike hierarchical or "document" databases, the relational algebra offers the best option for managing data sets while allowing queries of arbitrary complexity. The relational algebra is built on solid mathematics instead of design-by-commitee or legacy mistakes. Still, there are many exciting opportunities to expand study of the relational algebra; for example, in query optimization, data modeling principles, and distributed databases.
288+
289+## Why do existing databases not adhere to the relational algebra?
290+
291+Database management systems existed before the relational algebra existed or was fleshed out. Many database management products built "relational features" into pre-existing database software with legacy requirements where, inevitably, impedance mismatches led to divergence from the mathematics of the relational algebra. Today, the most prominent "relational" databases are based on SQL which is maintained as a standard by a commitee of legacy database product companies, not mathematicians. While SQL continues to evolve, it is quite clear it will never adhere to the mathematical principles of the relational algebra due to legacy concerns.
292+
293+## What is the advantage of adhering to the relational algebra?
294+
295+For most companies, its data, such as customer, employee, sales, and product information is its lifeblood. To entrust that data to unprincipled database management systems or worse, proprietary database software, is self-defeating. By providing a firm mathematical foundation on which to build their internal information technology infrastructure, the relational algebra reduces cross-product integration headaches, missing feature workarounds, and reliance on proprietary technology.
296+
297+## What is the advantage of having relation-valued attributes?
298+
299+Since a value can store any (immutable) type, it can certainly hold a relation. At the very least, the mathematics does not exclude the possibility. This does create a recursively-defined data structure (Relation -> Tuple -> Value -> Relation), but it turns out that relations as values can be quite useful.
300+
301+For example, in SQL, if one wants to collect some aggregate data over some tuples while still returning the individual tuples, in the best case, the aggregate information must be repeated for every individual tuple. In the worst case, two separate queries are required since nested tables are not supported.
302+
303+In the relational algebra, The aggregate value can be placed alongside the relation value from whic the aggregate was derived.
304+
305+Example:
306+
307+* Find the average salary per department along with the employees in each department.
308+
309+(Employees relation)
310+
311+|(Name, Name Type)|(Salary, Salary Type)|(Department, Department Type)|
312+|-|-|-|
313+|Steve|$32000|Marketing|
314+|Bob|$40000|Marketing|
315+|Sam|$41000|Sales|
316+
317+(relational algebra magic)
318+
319+====>
320+
321+|(Department, Department Type)|(Average Salary, Salary Type)|(Employees, (Name, Name Type))|
322+|-|-|-|
323+|Marketing|$36000|<table><tr><th>(Name, Name Type)</th></tr><tr><td>Steve</td></tr><tr><td>Bob</td></tr></table>|
324+|Sales|$41000|<table><tr><th>(Name, Name Type)</th></tr><tr><td>Sam</td></tr></table>|
325+
326+## Suggested Reading
327+
328+Chris Date has been researching the relational model and its consequences for over forty years. Date has published his findings in various papers but also summarizes his work in a series of introductory and advanced books. Date freely admits that his work on the relational algebra has been corrected over time, so it is best to read his latest works.
329+
330+Despite that fact that modern SQL databases do not adhere to the principles of relational algebra doesn't mean that the principles cannot be applied. Date often discusses and compares SQL constructions with his own "TutorialD" language. Project:M36 also supports TutorialD.
331+
332+Introductory texts:
333+* [SQL and Relational Theory:
334+How to Write Accurate SQL Code](http://shop.oreilly.com/product/0636920022879.do)
335+
336+
337+Advanced texts:
338+* [Database Design & Relational Theory](http://shop.oreilly.com/product/0636920025276.do)
339+* [Database Explorations](http://bookstore.trafford.com/Products/SKU-000177853/Database-Explorations.aspx)
340+* [View Updating and Relational Theory](http://shop.oreilly.com/product/0636920028437.do)
341+* [Time and Relational Theory](http://shop.oreilly.com/product/9780128006313.do)
docs/transaction_graph_operators.markdownnew file+47
@@ -0,0 +1,47 @@
1+## Transaction Operators
2+
3+Relational operators generate new relations and database context operators alter the relations referenced by relation variables. Finally, transaction operators alter the state of the database's transaction graph. This includes adding new transactions, branching, and changing the current database context to point at an arbitrary transaction in the past (time-travel).
4+
5+#### The Database Context
6+
7+Every database operation occurs within a database context. The database context is the state of the relation variables, constraints, and user-definable functions. While past transactions committed to the transaction graph are immutable, new transactions can be created from database contexts based on any transaction in the transaction graph.
8+
9+When a new database is created, a new transaction graph is created for it with one base transaction. The base transaction is special because it is the only transaction in the graph without a parent. At this point, the editable database context necessarily references the singular transaction. It can be considered a mutable copy of the database context stored by the first transaction.
10+
11+As the user interactively alters the database context such as by adding a new relation variable, the database context is no longer equal to that of the first transactions. This is called "database context divergence" because many of the same components exist in both the updated context and the context of the first transaction.
12+
13+[Initial database starting state](http://g.gravizo.com/g?
14+digraph G {
15+ base[label="base transaction",shape=rectangle];
16+ new[label="mutable database context"];
17+ new -> base;
18+})
19+
20+Once the user decides that the changes in the updated database context are worth adding to the transaction graph, the user issues a "commit" request. The commit operation converts the current database context into a new transaction which is then added to the graph with the parent transaction set to the same parent transaction as the database context. The user's mutable database context is shifted to be a copy of the context of the newly-committed transaction.
21+
22+[Freshly committed transaction](http://g.gravizo.com/g?
23+digraph G {
24+ base[label="base transaction",shape=rectangle];
25+ discon[label="mutable database context"];
26+ fresh[label="newly committed transaction",shape=rectangle];
27+ fresh -> base;
28+ discon -> fresh;
29+})
30+
31+It can be handy to attempt to modify the database without altering the data that others may wish to continue to use and modify. If a set of transactions may diverge from another use-case for the data, the user may choose to "branch" the transaction graph. Branching allows the user to assign an arbitrary name to a series of transactions and then potentially merge the set of transactions back to the originating branch. Project:M36 creates a "master" branch when the transaction graph is first created.
32+
33+Note that this transaction graph approach is different from that of most SQL DBMSs which have a strictly linear transaction graph where all the contention is at the head of the graph and neither time-travel nor branching are supported.
34+
35+#### Commit
36+
37+To convert a mutable database context into a transaction, it must be "committed" to the transaction graph.
38+
39+```
40+TutorialD (master): :commit
41+```
42+
43+Once committed, the transaction's database context becomes immutable. Internally, Project:M36 uniquely identifies each transaction with a UUID.
44+
45+#### Rollback
46+
47+Sometimes, the user determines that the changes in the mutable database context can be discarded and the context reset to be a copy of its parent's transaction. This is called a "rollback" because it rolls back to a previous known state.
docs/tutd_tutorial.markdownnew file+431
@@ -0,0 +1,431 @@
1+# TutorialD Tutorial for Project:M36
2+
3+TutorialD is an logic language designed by Chris Date for interacting with any relational algebra engine. Project:M36 supports this language for the purposes of learning about the relational algebra. TutorialD is not recommended for use in production systems.
4+
5+It is presumed that the audience for this tutorial has at least some minimal background with an SQL-based DBMS.
6+
7+### Starting the TutorialD Interpreter
8+
9+From the project-m36 source directory, execute ```cabal run tutd``` to start the interactive TutorialD interpreter. You will be greeted by the interactive prompt:
10+
11+```
12+TutorialD (master):
13+```
14+
15+"master" refers to the name of the branch to which the current transaction will be potentially committing.
16+
17+### Types
18+
19+TutorialD is strongly-typed. The basic built-in types are:
20+
21+|Type Name|Explanation|Example|
22+|-|-|-|
23+|char|arbitrary text|"The Old Man and the Sea"|
24+|int|any integer|-4|
25+|datetime|timestamp UTC|"2015-02-02 04:05:02"::datetime|
26+|double|floating point number|3.1459|
27+|bool|boolean value|t|
28+
29+With regards to boolean values, be sure not to conflate ```t``` or ```f``` as a boolean value with ```true``` and ```false``` which are relation variables.
30+
31+### Using Relation Variables
32+
33+Relation Variables are named cells which reference relations within a transaction state. The relation can be replaced with another relation through assignment.
34+
35+Definition of a relation variable:
36+```
37+products :: {name char, age int}
38+```
39+This expression creates a new relation variable in the current context which can refer to any relation with the header of "name" of type "char" and "age" of type int.
40+
41+Assignment to a relation variable:
42+```
43+products := relation{tuple{name "Mike",age 6},tuple{name "Sam",age 10}}
44+```
45+This expression assigns a relation containing two tuples to the previously-defined "products" relation.
46+
47+```tutd``` includes two relations by default:
48+* ```true```: the relation with no attributes and a body of one empty-attributed tuple
49+* ```false```: the relation with no attributes and a body of no tuples
50+
51+Chris Date refers to these relations as "TABLE_DUM" and "TABLE_DEE" but such arbitrary names are not so recognizable. ```true``` and ```false``` can be easily remembered as the answer to the question: "Does this relation have any tuples?" which can be formulated in TutorialD as the projection of a relation asking for no attributes:
52+```
53+products{}
54+```
55+
56+### Date's Sample Relations
57+
58+The examples in this tutorial can be executed after loading Chris Date's sample relations using:
59+
60+```
61+:importtutd "scripts/DateExamples.tutd"
62+```
63+
64+Each section assumes a fresh instance of the Date examples. The relation variables created in the script are:
65+
66+|Relation Variable Name|Description|
67+|-|-|
68+|S|suppliers|
69+|P|parts|
70+|SP|contains mapping of which suppliers supply which parts|
71+
72+### Relational Expressions
73+
74+A relational expression combines relational operators to create a new relation. ```tutd``` can display the result of executing a relational expression with ":showexpr".
75+
76+```
77+TutorialD (master): :showexpr P
78+┌──────┬─────┬──┬─────┬──────┐
79+│CITY │COLOR│P#│PNAME│WEIGHT│
80+├──────┼─────┼──┼─────┼──────┤
81+│London│Red │P6│Cog │19 │
82+│Paris │Blue │P5│Cam │12 │
83+│London│Red │P1│Nut │12 │
84+│London│Red │P4│Screw│14 │
85+│Oslo │Blue │P3│Screw│17 │
86+│Paris │Green│P2│Bolt │17 │
87+└──────┴─────┴──┴─────┴──────┘
88+```
89+
90+### Using Relational Operators
91+
92+Relational operators generate new relations from existing relations. The operators form the closed algebra against relations.
93+
94+#### Rename
95+
96+The unary rename operator outputs a new relation with chosen attributes renamed. In SQL, the equivalent is to use "AS" in the column name list of SELECTS.
97+
98+```
99+TutorialD (master): :showexpr S rename {CITY as TOWN}
100+┌──┬─────┬──────┬──────┐
101+│S#│SNAME│STATUS│TOWN │
102+├──┼─────┼──────┼──────┤
103+│S3│Blake│30 │Paris │
104+│S4│Clark│20 │London│
105+│S5│Adams│30 │Athens│
106+│S1│Smith│20 │London│
107+│S2│Jones│10 │Paris │
108+└──┴─────┴──────┴──────┘
109+```
110+
111+#### Projection
112+
113+Projection is applied by adding curly braces and attribute names following a relational expression. Projection in SQL is applied by adding a list of column names to a SELECT statement.
114+
115+For example:
116+
117+```
118+TutorialD (master): :showexpr P{COLOR,CITY}
119+┌──────┬─────┐
120+│CITY │COLOR│
121+├──────┼─────┤
122+│London│Red │
123+│Paris │Blue │
124+│Oslo │Blue │
125+│Paris │Green│
126+└──────┴─────┘
127+```
128+
129+#### Restriction
130+
131+Restriction is applied using a "where" clause, much like in SQL.
132+
133+```
134+TutorialD (master): :showexpr P where COLOR="Blue" and CITY="Paris"
135+┌─────┬─────┬──┬─────┬──────┐
136+│CITY │COLOR│P#│PNAME│WEIGHT│
137+├─────┼─────┼──┼─────┼──────┤
138+│Paris│Blue │P5│Cam │12 │
139+└─────┴─────┴──┴─────┴──────┘
140+```
141+
142+The restriction predicate can be built from "and", "not", and "or". Boolean atom functions can also appear in a restriction as long as they are preceded by "^". For example:
143+
144+```
145+TutorialD (master): :showexpr S where ^lt(@STATUS,20)
146+┌─────┬──┬─────┬──────┐
147+│CITY │S#│SNAME│STATUS│
148+├─────┼──┼─────┼──────┤
149+│Paris│S2│Jones│10 │
150+└─────┴──┴─────┴──────┘
151+```
152+
153+#### Join
154+
155+Joins are binary operators and are applied with the "join" keyword. The equivalent in SQL would be a "NATURAL JOIN". To join attributes which are not identically-named, use the rename operator
156+
157+```
158+TutorialD (master): :showexpr S join SP
159+┌──────┬──┬───┬──┬─────┬──────┐
160+│CITY │P#│QTY│S#│SNAME│STATUS│
161+├──────┼──┼───┼──┼─────┼──────┤
162+│London│P6│100│S1│Smith│20 │
163+│London│P3│400│S1│Smith│20 │
164+│London│P5│400│S4│Clark│20 │
165+│London│P1│300│S1│Smith│20 │
166+│Paris │P2│200│S3│Blake│30 │
167+│Paris │P1│300│S2│Jones│10 │
168+│London│P5│100│S1│Smith│20 │
169+│London│P4│300│S4│Clark│20 │
170+│Paris │P2│400│S2│Jones│10 │
171+│London│P2│200│S1│Smith│20 │
172+│London│P4│200│S1│Smith│20 │
173+└──────┴──┴───┴──┴─────┴──────┘
174+```
175+
176+#### Union
177+
178+The binary union operator merges tuples from both relations if-and-only-if the attributes of both relations are identical. The SQL equivalent is the "UNION" operator. Unlike SQL, however, duplicate tuples are not tolerated, so there is no "UNION ALL" equivalent in the relational algebra.
179+
180+```
181+TutorialD (master): :showexpr S union S
182+┌──────┬──┬─────┬──────┐
183+│CITY │S#│SNAME│STATUS│
184+├──────┼──┼─────┼──────┤
185+│Paris │S3│Blake│30 │
186+│London│S4│Clark│20 │
187+│Athens│S5│Adams│30 │
188+│London│S1│Smith│20 │
189+│Paris │S2│Jones│10 │
190+└──────┴──┴─────┴──────┘
191+```
192+
193+The union of any relation with itself is itself.
194+
195+#### Extend
196+
197+The extend unary operator adds an attribute to a relation's header and body and is represented by a colon ":". The equivalent in SQL is a subselect in a SELECT column list. Typically, extend is used to add information derived from relation.
198+
199+```
200+TutorialD (master): :showexpr S:{STATUS2:=add(10,@STATUS)}
201+┌──────┬──┬─────┬──────┬───────┐
202+│CITY │S#│SNAME│STATUS│STATUS2│
203+├──────┼──┼─────┼──────┼───────┤
204+│Paris │S2│Jones│10 │20 │
205+│Athens│S5│Adams│30 │40 │
206+│Paris │S3│Blake│30 │40 │
207+│London│S1│Smith│20 │30 │
208+│London│S4│Clark│20 │30 │
209+└──────┴──┴─────┴──────┴───────┘
210+```
211+
212+The "@" is required in the above query in order to distinguish the attribute's name from a relation's name.
213+
214+Supported atom functions include:
215+
216+|Function|Purpose|
217+|-|-|
218+|add(int,int)|Return the sum of two integers.|
219+|not(bool)|Invert a boolean expression.|
220+|lt(int,int)|Returns boolean true atom if he first argument is less than the second argument.|
221+|lte(int,int)|Returns boolean true atom if he first argument is less than or equal to the second argument.|
222+|gt(int,int)|Returns boolean true atom if he first argument is greater than the second argument.|
223+|gte(int,int)|Returns boolean true atom if he first argument is greater than or equal to the second argument.|
224+
225+#### Group
226+
227+The unary group operator groups the argument attributes into subrelations for each ungrouped set of attributes. SQL does not support tables as values, though SQL does support a notion of grouping by equal values. However, SQL's "GROUP BY" relies on tuple ordering in a table. A relations' tuples are never ordered.
228+
229+Grouping is useful for aggregate operations (see below) or for summarizing data against a set of attributes; for example, "display all employees grouped by boss name".
230+
231+```
232+TutorialD (master): :showexpr S group ({SNAME,STATUS,S#} as subrel)
233+┌──────┬─────────────────┐
234+│CITY │subrel │
235+├──────┼─────────────────┤
236+│Athens│┌──┬─────┬──────┐│
237+│ ││S#│SNAME│STATUS││
238+│ │├──┼─────┼──────┤│
239+│ ││S5│Adams│30 ││
240+│ │└──┴─────┴──────┘│
241+│London│┌──┬─────┬──────┐│
242+│ ││S#│SNAME│STATUS││
243+│ │├──┼─────┼──────┤│
244+│ ││S1│Smith│20 ││
245+│ ││S4│Clark│20 ││
246+│ │└──┴─────┴──────┘│
247+│Paris │┌──┬─────┬──────┐│
248+│ ││S#│SNAME│STATUS││
249+│ │├──┼─────┼──────┤│
250+│ ││S3│Blake│30 ││
251+│ ││S2│Jones│10 ││
252+│ │└──┴─────┴──────┘│
253+└──────┴─────────────────┘
254+```
255+
256+#### Ungroup
257+
258+The unary ungroup operator "unwraps" subrelations in a relation-valued attribute. There is no equivalent in SQL.
259+
260+```
261+TutorialD (master): :showexpr (S group ({SNAME,STATUS,S#} as subrel)) ungroup subrel
262+┌──────┬──┬─────┬──────┐
263+│CITY │S#│SNAME│STATUS│
264+├──────┼──┼─────┼──────┤
265+│Paris │S3│Blake│30 │
266+│London│S4│Clark│20 │
267+│Athens│S5│Adams│30 │
268+│London│S1│Smith│20 │
269+│Paris │S2│Jones│10 │
270+└──────┴──┴─────┴──────┘
271+```
272+
273+## State Operators
274+
275+While relational operators compose to relational expressions representing queries of the database, state operators change the state of the database.
276+
277+### Relation Variable State Changes
278+
279+#### Define and Assign
280+
281+The relation assignment mentioned above certainly changes the state of the database.
282+
283+```
284+TutorialD (master): newrel:=relation{tuple{age 3}}
285+TutorialD (master): :showexpr newrel
286+┌───┐
287+│age│
288+├───┤
289+│3 │
290+└───┘
291+```
292+
293+All further operators are convenience operators and could be implemented with simple assignment. SQL has no equivalent to relational assignment; instead, one must issue "CREATE TABLE" commands and insert rows into the resultant tables.
294+
295+#### Undefine
296+
297+To signal that a relation variable should be forgotten, undefine it.
298+
299+```
300+TutorialD (master): undefine S
301+TutorialD (master): :showexpr S
302+ERR: RelVarNotDefinedError "S"
303+```
304+
305+#### Insert
306+
307+The insert operators accepts a relation variable and a relation of the same type (a relation having an identical header), and replaces the relation referenced by the relation variable with the union of the previous relation value and the argument's relation value.
308+
309+```
310+TutorialD (master): insert S relation{tuple{CITY "Boston",S# "S10",SNAME "Gonzalez",STATUS 10}}
311+TutorialD (master): :showexpr S
312+┌──────┬───┬────────┬──────┐
313+│CITY │S# │SNAME │STATUS│
314+├──────┼───┼────────┼──────┤
315+│Paris │S3 │Blake │30 │
316+│London│S4 │Clark │20 │
317+│Athens│S5 │Adams │30 │
318+│London│S1 │Smith │20 │
319+│Paris │S2 │Jones │10 │
320+│Boston│S10│Gonzalez│10 │
321+└──────┴───┴────────┴──────┘
322+```
323+
324+This insertion operation is equivalent to:
325+
326+```
327+S:=S union relation{tuple{CITY "Boston",S# "S10",SNAME "Gonzalez",STATUS 10}}
328+```
329+#### Update
330+
331+The update operator accepts a relation argument and a predicate, optionally filters and creates new tuples based on the original relation.
332+
333+```
334+TutorialD (master): update S where STATUS=20 (SNAME:="Mr. Twenty")
335+TutorialD (master): :showexpr S
336+┌──────┬──┬──────────┬──────┐
337+│CITY │S#│SNAME │STATUS│
338+├──────┼──┼──────────┼──────┤
339+│Paris │S3│Blake │30 │
340+│London│S1│Mr. Twenty│20 │
341+│Athens│S5│Adams │30 │
342+│Paris │S2│Jones │10 │
343+│London│S4│Mr. Twenty│20 │
344+└──────┴──┴──────────┴──────┘
345+```
346+
347+This is logically equivalent to:
348+
349+```
350+TutorialD (master): S:=(((S where STATUS=20){CITY,S#,STATUS}):{SNAME:="Mr. Twenty"}) union (S where not STATUS=20)
351+TutorialD (master): :showexpr S
352+┌──────┬──┬──────────┬──────┐
353+│CITY │S#│SNAME │STATUS│
354+├──────┼──┼──────────┼──────┤
355+│Paris │S3│Blake │30 │
356+│London│S1│Mr. Twenty│20 │
357+│Athens│S5│Adams │30 │
358+│Paris │S2│Jones │10 │
359+│London│S4│Mr. Twenty│20 │
360+└──────┴──┴──────────┴──────┘
361+```
362+
363+This query means: "Drop the SNAME attribute from S filtered where STATUS equals 20, extend the resultant relation by SNAME with value 'Mr. Twenty', then union the resultant relation with S filtered by STATUS not equaling 20, and assign that to S."
364+
365+#### Delete
366+
367+The delete operator accepts a relation argument and a filtering predicate, creates a new relation with only the tuples which do not match the predicate, and stores the result in the relation variable argument.
368+
369+```
370+TutorialD (master): delete SP where S#="S4"
371+TutorialD (master): :showexpr SP
372+┌──┬───┬──┐
373+│P#│QTY│S#│
374+├──┼───┼──┤
375+│P3│400│S1│
376+│P4│200│S1│
377+│P2│400│S2│
378+│P2│200│S1│
379+│P6│100│S1│
380+│P1│300│S1│
381+│P2│200│S3│
382+│P1│300│S2│
383+│P5│100│S1│
384+└──┴───┴──┘
385+```
386+
387+This is logically equivalent to:
388+
389+```
390+TutorialD (master): SP:=SP where not S#="S4"
391+```
392+
393+Note the inverted predicate.
394+
395+## Constraints
396+
397+Database constraints are user-specified relational algebra predicates which must be true at all times. If a database expression attempts to violate a database constraint, then the DBMS must reject the expression and ensure that the previous, non-violating state remains.
398+
399+Chris Date identified that all constraints can be represented as inclusion dependencies, but Project:M36 offers some convenience operators.
400+
401+#### Uniqueness Constraints
402+
403+Uniqueness constraints ensure that a set of attributes' values are unique throughout a relational expression. For example, it is useful to ensure that all products' identifiers are unique. The equivalent in SQL is the UNIQUE expression found in table creation.
404+
405+```
406+TutorialD (master): key S_key_constraint {S#} S
407+```
408+
409+This expression creates a constraint named "S_key_constraint" and ensure that the values of the "S#" attribute are unique in the "S" relational expression.
410+
411+#### Foreign Key Constraints
412+
413+Foreign keys are constraints which require that certain values appearing in one relation variable must appear in another. The equivalent in SQL is the FOREIGN KEY() construction in table creation.
414+
415+```
416+TutorialD (master): foreign key S#_in_SP SP{S#} in S{S#}
417+```
418+
419+This expression ensure that any SP{S#} must also appear as a value in S{S#}.
420+
421+#### Other Constraints
422+
423+All other constraints can be represented by a set of inclusion dependencies.
424+
425+```
426+TutorialD (master): constraint S_status_less_than_50 (S where ^lt(50,@STATUS)){} in false
427+```
428+
429+This expression ensures that the relation variable's relation can never include a tuple with STATUS greater than 50.
430+
431+## Aggregate Queries