Previously we set up migrations directly as structs which doesn't match
how users will create them. For the new test framework, we'll instead
write TOML for migrations. This brings the tests closer to how an end
user will interact with Reshape and also lets us write tests to match
the examples in the documentation.
This reduces the amount of boilerplate required to run a test and also
has the added benefit of extending tests as we can now easily test both the
completion and abort paths. So far two tests have been ported over,
create_table and add_column, with the rest to follow.
The framework also improves the output of tests, adding sections clearly
indicating in which part of the migration process a test failed.
Tones down the disclaimer a bit as I feel reasonable certain Reshape
won't lose any data. Also adds some details about locking and downtime
to the introduction.
This makes the format of `add_index` a bit more consistent with
`add_column` which has a nested definition for the column. This also
sets up for us to allow adding indices in `create_table`.
This protects all database access using a Postgres advisory lock. This
way we can avoid problems when two instances of Reshape try to change
the same database at the same time.
With the old fields in the `State` broken out we can now make `State` a
proper enum, replacing the nested `Status`. The naming with both `State`
and `Status` was a bit confusing.
The version is not currently in used but will probably be useful later
when we want Reshape to be backwards compatbility. Having the version
stored in the database will then allow us to perform schema migrations
on the metadata of Reshape.
Before, the entire state was stored as a JSON encoded blob including all
completed migrations. This commits moves the migrations into a dedicated
table, which prevents the encoded from growing indefinitely. The size of
the state probably wouldn't have become a problem but it feels weird for
a schema migration tool to not use a proper schema.
Next, the version stored in the state struct will be moved to a separate
key in the `reshape.data` table and `State` will become an enum with the
same structure as `Status`. The naming right now is a bit confusing.
Until now, using alter_column would cause indices to be lost as they
weren't copied to the temporary column. This fixes that for indices that
consist of one or more actual columns. It won't work for indices on
expressions.
This allows people new to the project to understand how Reshape works before getting into the technical detail of setting it up and using it (which they may or may not want to do just yet).
Some migrations can be completed without needing a transaction for
atomicity. We should avoid transactions for DDL as far as possible to
avoid interfering with other queries and holding locks too long. It's
possible that all actions could be completed without a transaction but
I'm not sure of that yet.
Some migrations also won't work with transactions. For example dropping
an index using `DROP INDEX CONCURRENTLY` doesn't work inside a
transation.
Next step is to perform the same change for aborts.
This builds on the previous commit to make the transactions even more
granular when completing a migration. Keeping the transaction span short
is important to avoid interfering with other queries.
We previously used one transaction across all the migrations being
completed, which is not ideal from a locking perspective. Preferably we
want to keep the transactions as short-lived as possible to avoid
interfering with other queries. The next step will be to use one
transaction for each action.
To achieve this, we need to introduce a new intermediate state called
`Completing` which tracks which migrations have been completed so far.
Changing the default value only affects the new schema, the old schema
will still use the existing default value. This is to make sure a new
default value doesn't break the old application.
The builder will help simplify the tests and avoid boilerplate updates
when need fields are added to the create_table migration. We might want
to add builders all migrations later.
