@@ -0,0 +1,433 @@
1+# On NULL
2+
3+## Introduction
4+
5+There is no doubt that SQL has made NULL pervasive among database products. However, as Chris Date makes clear through numerous counter-examples, NULL can have no place in the relational algebra. This document summarizes Chris Date's rationale for banishing NULL, provides a rationale for why it cannot appear in a mathematically-coherent relational algebra system, and describes some alternative designs.
6+
7+## NULL is not a value
8+
9+SQL intentionally differentiates NULL from all other values. NULL is not an integer or text or anything else (though it does carry type information). In fact, NULL is literally equal to nothing else.
10+
11+```
12+postgres=# select NULL = NULL;
13+ ?column?
14+----------
15+
16+(1 row)
17+```
18+
19+Whether two NULLs are equal is unknown (NULL). Of course, this leads to three-valued logic: a boolean query can true, false, or unknown (NULL).
20+
21+## Ambiguity
22+
23+The purpose of a database is to model reality in some form, so what does the NULL represent in reality? Consider a nullable integer column; NULL could represent:
24+
25+* a unanswered question (How many children does he have? We forgot to ask.)
26+* a refused answer (How many children does he have? He refused to divulge this.)
27+* not applicable (How many children does the corporation have? Nonsensical.)
28+* information to collect in the future (How many children does he have? Ask him tomorrow.)
29+* the information can be found elsewhere (How many children does he have? Find the children's names with a join on some other table.)
30+* and any other arbitrary meaning
31+
32+The problem is that NULL gives no indication which of the above meanings are intended. Indeed, NULL could indicate that one or even more than one of the meanings could hold. The schema's nullable values makes the intention completely ambiguous.
33+
34+
35+## The Folly of Three-Valued Logic
36+
37+When dealing with NULLs, one inevitably must encounter the three-valued logic implied by the NULL non-value. This logic often comes as a surprise to newcomers as in the following examples:
38+
39+Setup: Create an example table with some NULLs.
40+```
41+postgres=# create table x (a int, b text);
42+CREATE TABLE
43+postgres=# insert into x(a,b) values (1,'one'),(2,'two'),(NULL,'three') returning a,b;
44+ a | b
45+---+-------
46+ 1 | one
47+ 2 | two
48+ | three
49+(3 rows)
50+```
51+Example 1: Return all rows where a is not equal to one.
52+```
53+postgres=# select * from x where a <> 1;
54+ a | b
55+---+-----
56+ 2 | two
57+(1 row)
58+```
59+Example 2: Return all rows where a is equal to itself.
60+```
61+postgres=# select * from x where a=a;
62+ a | b
63+---+-----
64+ 1 | one
65+ 2 | two
66+(2 rows)
67+```
68+Example 3: Return all rows where b is from a list of NULL.
69+```
70+postgres=# select * from x where b = any(NULL);
71+ a | b
72+---+---
73+(0 rows)
74+```
75+Example 4: Return all rows where the (a,b) tuple are in the set of x's tuples.
76+```
77+postgres=# select * from x where (a,b) = any(select * from x);
78+ a | b
79+---+-----
80+ 1 | one
81+ 2 | two
82+(2 rows)
83+```
84+Example 5: Return the natural join of x to itself.
85+```
86+postgres=# select * from x natural join x as y;
87+ a | b
88+---+-----
89+ 1 | one
90+ 2 | two
91+(2 rows)
92+```
93+Example 6: Return rows in x where the same row appears in x.
94+```
95+postgres=# select * from x where exists (SELECT * FROM x as y where (a,b) = (y.a,y.b));
96+ a | b
97+---+-------
98+ 1 | one
99+ 2 | two
100+ | three
101+(3 rows)
102+````
103+
104+All of the above query examples conform to the SQL standard with regards to NULL handling.
105+
106+
107+## Inconsistencies Leading to Errors
108+
109+If the difference between NULL handling in ```ANY``` versus ```EXISTS``` didn't surprise you, perhaps some NULL-in-aggregate-function usage will.
110+
111+Example 7: Aggregate a sum over a in x where one a value is NULL.
112+```
113+postgres=# select sum(a) from x;
114+ sum
115+-----
116+ 3
117+(1 row)
118+```
119+Example 8: Aggregate a sum over two NULLs.
120+```
121+postgres=# select sum(a) from (values (NULL::int),(NULL)) as y(a); --sum of only NULLs is NULL
122+ sum
123+-----
124+
125+(1 row)
126+```
127+Example 9: Count the number of a values in x. Count the number of rows in x.
128+```
129+postgres=# select count(a) from x;
130+ count
131+-------
132+ 2
133+(1 row)
134+postgres=# select count(*) from x;
135+ count
136+-------
137+ 3
138+(1 row)
139+```
140+Example 10: Aggregate the sum of an empty table.
141+```
142+postgres=# create table z(a int);
143+CREATE TABLE
144+postgres=# select sum(a) from z; -- sum of empty table is NULL
145+ sum
146+-----
147+
148+(1 row)
149+```
150+Example 11: NULL multipled by zero plus three.
151+```
152+postgres=# select 0*NULL + 3;
153+ ?column?
154+----------
155+
156+(1 row)
157+```
158+
159+This logic follows into arrays:
160+
161+Example 12: Is NULL equal to any element in the array?
162+```
163+postgres=# select NULL = ANY(array[1,2,3,NULL]);
164+ ?column?
165+----------
166+
167+(1 row)
168+```
169+
170+and row-valued expressions:
171+
172+Example 13: Is the (NULL,1) tuple equal to NULL? Is it not equal to NULL?
173+```
174+postgres=# select (NULL,1) is null;
175+ ?column?
176+----------
177+ f
178+(1 row)
179+
180+postgres=# select (NULL,1) is not null;
181+ ?column?
182+----------
183+ f
184+(1 row)
185+```
186+
187+Are you certain that your application handles NULL correctly? Would you notice if some rows disappeared due to NULL?
188+
189+## Multiple NULL Nonsense
190+
191+SQL does not enforce the relational model constraint that rows (tuples) be unique in a table (relation). So what does the following table mean?
192+
193+Example 14: Create multiple NULLs in a result set.
194+```
195+postgres=# select * from (values (34),(50),(NULL),(NULL)) as employee_age(val);
196+ val
197+-----
198+ 34
199+ 50
200+
201+
202+(4 rows)
203+```
204+
205+What could this construct model in reality? In general, what does a row which is completely NULL mean?
206+
207+## Ordering NULLs
208+
209+Since NULLs are not valued, they don't have a logical sort order. Luckily, SQL allows one to arbitrarily sort NULLs to come before all real values or after all real values. But... so what? Invariably, the receiver of a result set containing NULLs will require special handling for the NULLs.
210+
211+## NOT NULL
212+
213+Suppose you are convinced NULLs in SQL are too convoluted to be used safely. Even if all columns in your schema are marked ```NOT NULL```, NULLs can still make an unwanted appearance. We have seen this with some aggregate functions on empty tables, but any usage of ```OUTER JOIN``` could generate NULLs, too.
214+
215+## Outer Join Ambiguity
216+
217+Example 15: A table with one valued-row and one completely NULL row. Construct NULL-aware left outer join.
218+```
219+postgres=# create table z(a int,b text);
220+CREATE TABLE
221+postgres=# insert into z(a,b) values (2,'two'),(NULL,NULL) returning a,b;
222+ a | b
223+---+-----
224+ 2 | two
225+ |
226+(2 rows)
227+postgres=# select x.a as xa,x.b as xb,z.a as za,z.b as zb from x left join z on x.a is not distinct from z.a;
228+ xa | xb | za | zb
229+----+-------+----+-----
230+ 1 | one | |
231+ 2 | two | 2 | two
232+ | three | |
233+(3 rows)
234+
235+```
236+Does the double-NULL which appears twice reference the actual double-NULL row in ```z``` or is it a missing row which is actually missing in ```z```? There is no way to know from this query. In outer joins, NULLs are overloaded with at least two possible meanings.
237+
238+
239+## Reversed Logic
240+
241+Inevitably, when one attacks the notion of NULL is SQL, database developers become defensive:
242+
243+* You can't have a database properly model reality without NULLs. Missing information is a fact-of-life.
244+* Even if NULLs don't fit the mathematics, they are too valuable to dismiss.
245+* Getting rid of NULLs is a purely academic concern. NULLs don't bother me.
246+
247+However, the onus should be on those holding these viewpoints to justify the existence of a construct which:
248+
249+* adds substantial complexity in the fundamental logic
250+* adds ambiguity to query results
251+* reduces confidence that queries are logically-sound
252+* fails to adequately model the intention of missing information (more on this below)
253+
254+Undoubtedly, representing missing or unknown values *is* a useful construct, but unfortunately SQL's NULL fails to adequately represent this notion.
255+
256+If the SQL designers had wanted to be mathematically rigorous with regards to NULL, every query would return two tables (result sets): one where the query's predicate certainly holds, and a second where the predicate might hold if NULL is interpreted as unknown- a sort of logical superposition.
257+
258+## What was the goal of NULL?
259+
260+Clearly, the notion of NULL would not have been introduced if it served no valid purpose. NULL came from a time where all base database types could fit in CPU registers: integers, ASCII bytes (text), floats. Most programming languages includes these data types in some form, so to include them in SQL was a logical step.
261+
262+Then, someone must have realized that it can be useful to have a tuple/row record an employee's name, date-of-hire, and salary, even if his age, an otherwise required attribute/column, is unknown. This is certainly a fair need to fulfill, so why not put a placeholder in the place of age, indicating "fill this in later". Some programming languages include the notion of null references which literally reference nothing. (The first relational-model-based DBMS was [System R](http://www.cs.berkeley.edu/~brewer/cs262/SystemR.pdf) written in PL/I and included NULL). So, the inclusion of this concept in SQL seemed logical. However, in those same programming languages, attempting to dereference or use a NULL reference in calculations results in a fatal error, terminating the program. Obviously, we don't want to fail to answer queries with missing/unknown information, because it's not logically a fatal condition. Thus, the three-valued logic was born.
263+
264+Unfortunately, the consequences of including three-valued logic were not fully considered. There is no shame in this, as long as we learn from mistakes. Even Codd, the original IBM-based author of the seminal paper describing the relational model, tried to shoehorn in the concept of NULL.
265+
266+## What is the alternative to NULL?
267+
268+Chris Date devotes several chapters to exactly this question in his "[Database Explorations](http://bookstore.trafford.com/Products/SKU-000177853/Database-Explorations.aspx)". Some of his suggestions are not teneable, but are considered for the purpose of discussion. Date's options for representing missing data, summarized, are:
269+
270+* "The Decomposition Approach": normalize/decompose relations horizontally or vertically so that missing data is implied by missing tuples
271+* "The Multirelational Approach": a structure which includes multiple relations with differing headers. This approach is mathematically untenable, but listed for dicussion.
272+* "An Inheritance Approach": use vaguely object-oriented type hierarchies to build new types which may include a missing value. Note that "missing" is indeed a value in this model.
273+* "An Approach Using Relation Valued Attributes": use single- or empty-tupled relation-values to represent present or missing data
274+
275+Date, however, fails to generalize the "inheritance approach" which leads to perhaps the easiest solution. [Algebraic data types](https://en.wikipedia.org/wiki/Algebraic_data_type) are essential in functional programming but make only rare appearances in other programming languages. The core of algebraic data types is that types themselves can be composed using either products or sums of types.
276+
277+Defining a new type can involve taking the product of two other types, as in:
278+```haskell
279+data BookType = Book BookTitle ISBN
280+```
281+where ```BookTitle``` and ```ISBN``` are pre-existing types. Here, the new BookType can be considered a sort of new two-tuple holding two other types.
282+
283+Types can also be composed with a summation:
284+```haskell
285+data HairColorType = HairColor ColorName |
286+ Bald
287+```
288+
289+In this case, ```ColorName``` is a text string describing the color. The values associated with a ```HairColorType``` could be any of the following (as examples):
290+
291+* ```HairColor "Brown"```
292+* ```HairColor "Blonde"```
293+* ```Bald```
294+
295+Here, we witness that a type can be composed of multiple other types in a kind of "or" relationship. Indeed, this is precisely our intention when we wish to mark something as unknown! If a record's "age" attribute is unknown, the value of that record's field should model that intention. Furthermore, we can be explicit about why the value is missing:
296+
297+```haskell
298+data AgeType = PreciseAge NaturalNumberType |
299+ ForgotToAsk |
300+ RefusedToDisclose |
301+ NotApplicable |
302+ ApproximateAge NaturalNumberType NaturalNumberType
303+```
304+and some examples values:
305+
306+* ```PreciseAge 25```
307+* ```NotApplicable```
308+* ```ApproximateAge 30 40``` approximately between 30-40 years old
309+
310+Already, we can see that this single data type can pack far more useful information than any combination of an integer and NULL.
311+
312+### In SQL
313+
314+An algebraic data type may be emulated in SQL, albeit in a convoluted fashion and without type safety. Here is the above example translated using SQL enumerations:
315+
316+```SQL
317+postgres=# create type age_type as enum ('precise_age', 'forgot_to_ask', 'refused_to_disclose', 'not_applicable', 'approximate_age');
318+CREATE TYPE
319+
320+postgres=# create table hospital_patient(name text not null,
321+ age age_type not null,
322+ precise_age int,
323+ approximate_age int[2]);
324+postgres=# insert into hospital_patient(name,age,precise_age,approximate_age) values ('Bob','approximate_age',NULL,array[30,40]);
325+INSERT 0 1
326+postgres=# select name,
327+case when age = 'precise_age' then
328+ 'precisely ' || precise_age::text
329+when age = 'approximate_age' then
330+ 'approximately ' || approximate_age[1] || '-' || approximate_age[2]
331+else
332+ age::text
333+end
334+from hospital_patient;
335+ name | age
336+------+---------------------
337+ Bob | approximately 30-40
338+(1 row)
339+```
340+
341+But all we have really done is add even more NULLs. Retrieving a sensible value is convoluted and not strongly-typed. In addition, the table constraints (not shown) necessary to ensure that, for example, precise_age is NULL if age is "forgot_to_ask" make this scheme even more complicated to implement.
342+
343+### In Project:M36
344+
345+Any Haskell data type can be turned into a value (called an "atom") in Project:M36. The following example illustrates:
346+
347+```haskell
348+{-# LANGUAGE OverloadedStrings,DeriveAnyClass,DeriveGeneric #-}
349+module Hospital where
350+
351+import ProjectM36.Client
352+import ProjectM36.Atom
353+import Data.Typeable
354+import ProjectM36.Relation
355+import Data.Binary
356+import Control.DeepSeq
357+import Data.Text
358+import GHC.Generics
359+import Data.Hashable
360+import ProjectM36.Tuple
361+
362+data AgeType = PreciseAge Int |
363+ ForgotToAsk |
364+ RefusedToDisclose |
365+ NotApplicable |
366+ ApproximateAge Int Int
367+ deriving (Eq,Show,Read,Hashable,Binary,Typeable,NFData,Generic)
368+
369+instance Atomable AgeType
370+
371+failFastMaybe :: (Show a) => Maybe a -> IO ()
372+failFastMaybe (Just err) = error (show err)
373+failFastMaybe Nothing = return ()
374+
375+failFastEither :: (Show a) => Either a b -> IO (b)
376+failFastEither (Left err) = error (show err)
377+failFastEither (Right val) = return val
378+
379+ageAtomType :: AtomType
380+ageAtomType = atomTypeForProxy (Proxy :: Proxy AgeType)
381+
382+runExample :: IO ()
383+runExample = do
384+ let bob_relation_attrs = attributesFromList [Attribute "name" stringAtomType,
385+ Attribute "age" ageAtomType]
386+ relvar_name = "hospital_patient"
387+ age_value_in = ApproximateAge 30 40
388+ bob_relation_err = mkRelationFromList
389+ bob_relation_attrs
390+ [[Atom ("Bob"::Text), Atom age_value_in]]
391+
392+ connerr <- connectProjectM36 (InProcessConnectionInfo NoPersistence)
393+ conn <- failFastEither connerr
394+ bob_relation <- failFastEither bob_relation_err
395+ --create the relation variable
396+ merr <- executeDatabaseContextExpr conn (Assign relvar_name (ExistingRelation bob_relation))
397+ failFastMaybe merr
398+ --query the relation variable
399+ result_err <- executeRelationalExpr conn (RelationVariable relvar_name)
400+ result <- failFastEither result_err
401+ case singletonTuple result of
402+ Nothing -> error "not a singleton relation!"
403+ Just tuple -> do
404+ (Atom age_value_out) <- failFastEither $ atomForAttributeName "age" tuple
405+ putStrLn (show age_value_in)
406+ putStrLn (show age_value_out)
407+ case cast age_value_out of
408+ Nothing -> error "wrong datatype"
409+ Just age_value_out' -> do
410+ putStrLn (show (age_value_in == age_value_out'))
411+
412+```
413+
414+The example demonstrates that the algebraic datatype ```AgeType``` value which goes into the database is an equal value when it is retrieved from the database.
415+
416+Since ```AgeType``` is a normal value, but one which can be created using multiple constructors (```PreciseAge x```, ```ForgotToAsk```, etc.), it can be manipulated using standard database functions without any special consideration for which values indicate "missing-ness".
417+
418+## Summary
419+
420+This document summarizes the inconsistencies and ambiguities when faced with dealing with SQL's NULL. As Date writes, "You can never trust the answers you get from a database with nulls. In my opinion, this state of affairs is a complete showstopper." ([Database In Depth](http://shop.oreilly.com/product/9780596100124.do) p. 55)
421+
422+Regardless, modeling the fact that some information may be missing is valuable, as long as the fact is unambiguous and does not violate the relational algebra. This can be accomplished by several means, but the easiest is to make use of algebraic data types. Unfortunately, SQL products do not support algebraic data types.
423+
424+
425+## Additional Reading
426+
427+Date, C.J. Darwen, Hugh. [Database Explorations: Essays on the Third Manifesto and Related Topics](http://bookstore.trafford.com/Products/SKU-000177853/Database-Explorations.aspx) chapters 23-29
428+
429+Date, C.J. [Database in Depth](http://bookstore.trafford.com/Products/SKU-000177853/Database-Explorations.aspx) pages 53-55
430+
431+
432+
433+