The main reason I wasn't happy with this before is that there was nothing stopping you from writing `materialize query pure`. This returned query would produce invalid SQL if you actually tried to use it, because it would attempt to reference a common table expression outside the scope of the `WITH` statement.
The "solution" here is just to throw a `rebind` around the result such that `materialize` incurs an extra `Table Expr b` constraint, which means that you can't return `Query (Query a)` because `Query a` can't satisfy a `Table Expr` constraint.
This does away with the weird variadic arguments thing we had going on with `function`.
Functions with no arguments are now written as:
```haskell
now :: Expr UTCTime
now = function "now" ()
```
Functions with multiple arguments are now written as:
```haskell
quot :: Sql DBIntegral a => Expr a -> Expr a -> Expr a
quot n d = function "div" (n, d)
```
Single-argument functions are written exactly as before.
This adds a new type `QualifiedName` for named PostgreSQL objects (tables, views, functions and sequences) that can optionally be qualified by a schema. Previously only `TableSchema` could be qualified in this way.
`QualifiedName` has an `IsString` instance so the common case (where the schema is `Nothing`) doesn't have to care about schemas (if `OverloadedStrings` is enabled).
This also refactors `TableSchema` to use `QualifiedName` for its `name` field and drops its `schema` field.
Thanks to @elldritch for the bug report and the inspiration.
The motivation behind this PR is to add support for PostreSQL's `WITH` syntax at the statement level, which gives the ability to, e.g., delete some rows from a table and then re-insert those deleted rows into another table, without any round-trips between the application and the database.
To support this, this PR introduces a new type called `Statement`, which represents a single PostgreSQL statement. It has a `Monad` instance which allows sub-statements (such as `DELETE` and `INSERT` statements) to be composed together and their results bound to values that can be referenced in subsequent sub-statements. These "compound" statements are then rendered as a `WITH` statement.
`select`, `insert`, `update` and `delete` have all been altered to produce the `Statement` type described above instead of the `Hasql.Statement` type.
Some changes were necessary to the `Returning` type. `Returning` previously bundled two different concepts together: whether or not to generate a `RETURNING` clause in the SQL for a manipulation statement, and how to decode the returned rows (if any). It was necessary to break these concepts apart because with `WITH` we need the ability to generate manipulation statements with `RETURNING` clauses that are never actually decoded at all (the results just get passed to the next statement without touching the application).
Now, the `Returning` type is only concerned with whether or not to generate a `RETURNING` clause, and the question of how to decode the returned the result of the statement is handled by the `run` functions. `run` converts a `Statement` into a runnable `Hasql.Statement`, decoding the result of the statement as a list of rows. The other variations, `run_`, `runN`, `run1`, `runMaybe` and `runVector` can be used when you want to decode as something other than a list of rows.
This also gains us support for decoding the result of a query directly to a `Vector` for the first time, which brings a performance improvement over lists for those who need it.
This is another possible "fix" to #168 (as opposed to #242). It doesn't really fix the problem, but it allows us to use two levels of `catListTable` instead of only one. Instead of trying to use Postgres's broken `.f1` syntax, we cast the anonymous record to text, remove the parentheses and quotes and unescape any escaped quotes or backslashes, and then cast the resulting text back to the appropriate type. The reason this only works one level deep is that if the type we cast the text back to is itself an anonymous record, then PostgreSQL doesn't know how to parse the text.
It's kind of ugly and hacky but it does work and otherwise maintains the status quo. Comparison operators on nested lists continue to work as before and we don't need to burden `DBType` with parsing nonsense.
This PR makes a number of changes to how aggregation works in Rel8.
The biggest change is that we drop the `Aggregate` context and we return to the `Profunctor`-based `Aggregator` that Opaleye uses (as in #37). While working with `Profunctor`s is more awkward for many common use-cases, it's ultimately more powerful. The big thing it gives you that we don't currently have is the ability to "post-map" on the result of an aggregation function. Pretend for a moment that Postgres does not have the `avg` function built-in. With the previous Rel8, there is no way to directly write `sum(x) / count(x)`. The best you could do would something like:
```haskell
fmap (\(total, size) -> total / fromIntegral size) $ aggregate $ do
foo <- each fooSchema
pure (sum foo.x, count foo.x)
```
The key thing is that the mapping can only happen after `aggregate` is called. Whereas with the `Profunctor`-based `Aggregator` this is just `(/) <$> sum <*> fmap fromIntegral count`. This isn't too bad if the only thing you want to do is computing the average, but if you're doing a complicated aggregation with several things happening at once then you might need to do several unrelated post-processings after the `aggregate`. We really want a way to bundle up the postmapping with the aggregation itself and have that as a singular composable unit. Another example is the `listAggExpr` function. The only reason Rel8 exports this is because it can't be directly expressed in terms of `listAgg`. With the `Profunctor`-based `Aggregator` it can be, it's just `(id $*) <$> listAgg`, it no longer needs to be a special case.
The original attempt in #37 recognised that it can be awkward to have to write `lmap (.x) sum`, so instead of sum having the type signature `Aggregator (Expr a) (Expr a)`, it had the type signature `(i -> Expr a) -> Aggregator i (Expr a)`, so that you wouldn't have to use `lmap`, you could just type `sum (.x)`. However, there are many ways to compose `Aggregator`s — for example, if you wanted to use combinators from `product-profunctor` to combine aggregators, then you'd rather type `sum ***! count` than `sum id ***! count id`. So in this PR we keep the type of `sum` as `Aggregator (Expr a) (Expr a)`, but we also export `sumOn`, which has the bundled `lmap`.
The other major change is that this PR introduces two forms of aggregation — "semi"-aggregation and "full"-aggregation. Up until now, all aggregation in Rel8 was "semi"-aggregation, but "full"-aggregation feels a bit more natural and Haskelly.
Up until now, the `aggrgegate` combinator in Rel8 would return zero rows if given a query that itself returned zero rows, even if the aggregation functions that comprised it had identity values. So it was very common to see code like `fmap (fromMaybeTable 0) $ optional $ aggregate $ sum <$> _`. Again, we "know" that `0` is the identity value for `sum` and we really want some way to bundle those together and to say "return the identity value if there are zero rows". Rel8 now has this ability — it has both `Aggregator` and `Aggregator1`, with the former having identity values and the latter not. The `aggregate` function now takes an `Aggregator` and returns the identity value when encountering zero rows, whereas the `aggregate1` function takes an `Aggregator1` and behaves as before. `count`, `sum`, `and`, `or`, `listAgg` are `Aggregator`s (with the identity values `0`, `0`, `true`, `false` and `listTable []` respectively) and `groupBy`, `max` and `min` are `Aggregator1`s.
This also means that `many` is now just `aggregate listAgg` instead of `fmap (fromMaybeTable (listTable [])) . optional . aggregate . fmap listAgg`.
It should also be noted that these functions are actually polymorphic — `sum` will actually give you an `Aggregator'` that can be used as either `Aggregator` or `Aggregator1` without needing to explicitly convert between them. Similarly `aggregate1` can take either an `Aggegator` or an `Aggregator1` (though it won't use the identity value of the former).
Aggregation in Rel8 now supports more of the features of PostgresSQL supports. Three new combinators are introduced — `distinctAggregate`, `filterWhere` and `orderAggregateBy`.
Opaleye itself already supported `distinctAggregate` and indeed we used this to implement `countDistinct` as a special case, but we now support using `DISTINCT` on arbitrary aggregation functions.
`filterWhere` is new to both Rel8 and Opaleye. It corresponds to PostgreSQL's `FILTER (WHERE ...)` syntax in aggregations. It also uses the identity value of an `Aggregator` in the case where the given predicate returns zero rows. There is also `filterWhereOptional` which can be used with `Aggregator1`s.
`orderAggregateBy` allows the values within an aggregation to be ordered using a given ordering, mainly non-commutative aggregation functions like `listAgg`.
In #182 we introduced window functions, and also refactored `indexed` to use the new window function support via `rowNumber`.
However, previously `indexed` was zero-based, whereas PostgreSQL's `row_number()` function counts from one, so this change silently changed the behaviour of `indexed`.
This commit restores the previous behaviour.
`Rel8.Tabulate.alignWith` uses `theseTable` to recover the key, which translates to a mess of nested `CASE` expressions. This becomes exponential with repeated uses of `alignWith`. Adding a `rebind` step to `alignWith` to rewrite this `key` expression each time makes this linear.
At some point, possibly due to a change in Opaleye, insertion of default values (using `unsafeDefault`) stopped working in Rel8.
In general, PostgreSQL only allows insertion of default values in insert statements that have the form `INSERT INTO table VALUES (DEFAULT)`. The way Opaleye typically generates SQL, the Rel8 equivalent to that code would come out looking more like `INSERT INTO table SELECT * FROM (VALUES (DEFAULT)) q`. Because the `VALUES` is wrapped in a `SELECT`, the `DEFAULT` becomes invalud PostgreSQL syntax.
To get around this, we special case `VALUES` when generating SQL for INSERT statements. However, something in the Opaleye representation changed and this special case was no longer firing. This commit fixes that.
* Add htraverseP
* Add traverseFieldP
* Define and use fromOpaleyespec
* Use traverseFieldP
* Add htraversePWithField and use it in valuesspec
* Use aggregatorApply
* Update HTable.hs
Co-authored-by: Ollie Charles <ollie@ocharles.org.uk>
Co-authored-by: Shane <shane.obrien@circuithub.com>
`Tabulation` has an instance of `AltTable`, which is intended to uphold the following law:
```haskell
fromQuery a <|>: fromQuery b = fromQuery (a <|>: b)
```
The previous implementation was not actually defined in terms of `Query`'s `<|>:` (i.e., `UNION ALL`), because not every `Tabulation` can be safely `toQuery`'d. Instead it used a combination of `alignWith`, `catNonEmptyTable` and `some` that worked even on "magic" `Tabulation`s.
However, using `unsafePeekQuery`, we can actually "statically" determine if a `Tabulation` is "magic" or not, which means we can selectively switch the implementation to use `Query`'s `<|>:` where possible. This produces a simpler and usually faster query.
This is an alternative to `MaybeTable` that doesn't use a tag columns. It's less flexible (no `Functor` or `Applicative` instance) and is meaningless when used with a table that has no non-nullable columns (so nesting `NullTable` is redundant). But in situations where the underlying `Table` does have non-nullable columns, it can losslessly converted to and from `MaybeTable`.
It is useful for embedding into a base table when you don't want to store the extra tag column in your schema.
Co-authored-by: Ollie Charles <ollie@ocharles.org.uk>
