@@ -0,0 +1,70 @@
1+# Reaching Out of the Tarpit with Project:M36
2+
3+## Introduction
4+
5+[Project:M36](https://github.com/agentm/project-m36) is a relational algebra database management system (RDBMS) whose purpose is to implement the foundation for functional relational programming as described in the excellent ["Out of the Tarpit"](http://shaffner.us/cs/papers/tarpit.pdf) paper. This document describes how Project:M36 meets the design goals laid out in the paper. It is recommended to read the paper before reading this document.
6+
7+## Why is "Out of the Tarpit" Important?
8+
9+The "Out of the Tarpit" paper, written in 2006, consolidates decades of software development research and experience to explain why software written today is chock-full of software bugs. The paper categorizes sources of complexity and then proceeds to propose a software architecture which promises to reduce unnecessary software complexity so that the resultant software is maximally focused on actual business rules.
10+
11+## Laying the Foundations for Reduced Complexity
12+
13+After analyzing sources of software complexity (primarily unnecessarily mutable state and unintended order-of-execution), the not-so-radical proposal of the "Out of the Tarpit" paper is "functional-relational programming", not to be confused with "[functional reactive programming](https://wiki.haskell.org/Functional_Reactive_Programming)". According to the paper, the ideal language would minimize mutable state (such as in functional programming) and eliminate unintended order-of-execution by providing declarative instead of imperative constructs.
14+
15+The central software component of functional-relational programming is a relational algebra engine/database management system (RDBMS). Project:M36 purports to meet the software requirements of such a functional-relational system, perhaps being the first RDBMS to actually target the paper's proposed design.
16+
17+## The Importance of the Relational Model
18+
19+"Out of the Tarpit" recommends the relational model not only in its customary backend role, but throughout the application because the relational model is:
20+
21+* **declarative**: thus removing accidental complexity (such as accidental ordering)
22+* **restricted**: the paper makes it clear that if a programming language allows a programmer to do something less-than-savory, then that misfeature will be used
23+
24+and provides ([p.37](http://shaffner.us/cs/papers/tarpit.pdf)):
25+
26+* **structure**: a closed world system whereby all data is represented by relations
27+* **manipulation**: a means to simulate mutation of the application state over time
28+* **data integrity**: constraints which prevent illogical data constructions
29+* **data independence**: the logical and physical representations of state are clearly separated
30+
31+The relational model is a mathematical model for storing and manipulating "essential state" ([p.25](http://shaffner.us/cs/papers/tarpit.pdf)). As a mathematical model, it not beholden to any specific implementation or quirks.
32+
33+Unfortunately, due to the quirks of legacy implementations, much of what developers learn about the relational model is wrong. Project:M36 aims to rectify this situation by providing a mathematically coherent implementation of the relational model. Contrast the mathematical approach to existing databases which sacrifice or elide a mathematical model in lieu of "performance". Refer to [Knuth on premature optimization](https://en.wikiquote.org/wiki/Donald_Knuth) for additional details.
34+
35+## Chris Date's Contribution
36+
37+Chris Date is a prolific author, researcher, and lecturer on relation algebra and database design in general. Project:M36 would not exist without his writings on the mathematical underpinnings of the relational algebra. Project:M36 does not however implement all Date's recommendations. In particular, Project:M36 implements:
38+
39+* **algebraic data types** inspired by Haskell's types in lieu of Date's convoluted data types based on object-oriented type systems
40+* **transactions in a graph** in lieu of Date's temporal constructs
41+* **a TutorialD dialect** which is not 100% compatible with Date's proposal
42+* **isomorphic schemas** (not yet implemented) in lieu of Date's updateable views
43+
44+## Project:M36 In Practice
45+
46+Values which Project:M36 can store are typed by the powerful functional programming language Haskell. Any Haskell data value can become a database value by implementing a simple typeclass. Thus, it provides a native and seamless interface between Haskell and the relational algebra engine.
47+
48+Project:M36 does not, however, implement a completely new programming language for functional-relational programming, as suggested in the paper. The benefits of a fully-restricted programming language are described in the paper and this feature may be implemented in the future.
49+
50+Project:M36 implements multiple interfaces:
51+
52+* an interactive console implementing [TutorialD](https://en.wikipedia.org/wiki/D_%28data_language_specification%29)
53+* a native Haskell API
54+* a database driver for the [persistent framework](http://www.yesodweb.com/book/persistent)
55+
56+To understand how Project:M36 stacks up against the fictional programming language shown in the paper, we have implemented the example from the paper.
57+
58+feeders
59+observers
60+
61+## Good Riddance to Old Baggage
62+
63+Project:M36 is a clean-room implementation of a relational algebra engine; it does not implement SQL or any other legacy standard. While this means that Project:M36 is incompatible with other RDBMS, it means that Project:M36 can avoid the substantial mistakes of previous implementations, such as:
64+
65+* [three-valued logic (NULL)](docs/on_null.markdown)
66+* non-relational database constructs such as row and column ordering or OUTER JOINs
67+* linear transaction flow and conflict resolution
68+* direct table-to-file mapping which create perverse model-data dependencies
69+* key-value storage: a key-value mapping is valued merely for its performance characteristics, not because it models reality well
70+* restrictive, legacy type systems which impede modeling