@@ -2,54 +2,54 @@
22
33[](https://github.com/fabianlindfors/reshape/actions/workflows/test.yaml) [](https://github.com/fabianlindfors/reshape/releases)
44
5+> Also check out [ReshapeDB](https://reshapedb.com), a new database built from the ground up to make zero-downtime schema and data migrations as simple and safe as possible. If you'd like to chat about it, please [reach out](contact@reshapedb.com)!
6+
57Reshape is an easy-to-use, zero-downtime schema migration tool for Postgres. It automatically handles complex migrations that would normally require downtime or manual multi-step changes. During a migration, Reshape ensures both the old and new schema are available at the same time, allowing you to gradually roll out your application. It will also perform all changes without excessive locking, avoiding downtime caused by blocking other queries. For a more thorough introduction to Reshape, check out the [introductory blog post](https://fabianlindfors.se/blog/schema-migrations-in-postgres-using-reshape/).
68
79Designed for Postgres 12 and later.
810
9−*Note: Reshape is **experimental** and should not be used in production. It can (and probably will) break your application.*
10−
1111- [How it works](#how-it-works)
1212- [Getting started](#getting-started)
13− - [Installation](#installation)
14− - [Creating your first migration](#creating-your-first-migration)
15− - [Preparing your application](#preparing-your-application)
16− - [Running your migration](#running-your-migration)
17− - [Using during development](#using-during-development)
13+ - [Installation](#installation)
14+ - [Creating your first migration](#creating-your-first-migration)
15+ - [Preparing your application](#preparing-your-application)
16+ - [Running your migration](#running-your-migration)
17+ - [Using during development](#using-during-development)
1818- [Writing migrations](#writing-migrations)
19− - [Basics](#basics)
20− - [Tables](#tables)
21− - [Create table](#create-table)
22− - [Rename table](#rename-table)
23− - [Remove table](#remove-table)
24− - [Add foreign key](#add-foreign-key)
25− - [Remove foreign key](#remove-foreign-key)
26− - [Columns](#columns)
27− - [Add column](#add-column)
28− - [Alter column](#alter-column)
29− - [Remove column](#remove-column)
30− - [Indices](#indices)
31− - [Add index](#add-index)
32− - [Remove index](#remove-index)
33− - [Enums](#enums)
34− - [Create enum](#create-enum)
35− - [Remove enum](#remove-enum)
36− - [Custom](#custom)
19+ - [Basics](#basics)
20+ - [Tables](#tables)
21+ - [Create table](#create-table)
22+ - [Rename table](#rename-table)
23+ - [Remove table](#remove-table)
24+ - [Add foreign key](#add-foreign-key)
25+ - [Remove foreign key](#remove-foreign-key)
26+ - [Columns](#columns)
27+ - [Add column](#add-column)
28+ - [Alter column](#alter-column)
29+ - [Remove column](#remove-column)
30+ - [Indices](#indices)
31+ - [Add index](#add-index)
32+ - [Remove index](#remove-index)
33+ - [Enums](#enums)
34+ - [Create enum](#create-enum)
35+ - [Remove enum](#remove-enum)
36+ - [Custom](#custom)
37+ - [Complex changes across tables](#complex-changes-across-tables)
3738- [Commands and options](#commands-and-options)
38− - [`reshape migration start`](#reshape-migration-start)
39− - [`reshape migration complete`](#reshape-migration-complete)
40− - [`reshape migration abort`](#reshape-migration-abort)
41− - [`reshape schema-query`](#reshape-schema-query)
42− - [Connection options](#connection-options)
39+ - [`reshape migration start`](#reshape-migration-start)
40+ - [`reshape migration complete`](#reshape-migration-complete)
41+ - [`reshape migration abort`](#reshape-migration-abort)
42+ - [`reshape schema-query`](#reshape-schema-query)
43+ - [Connection options](#connection-options)
4344- [License](#license)
4445
45−
4646## How it works
4747
4848Reshape works by creating views that encapsulate the underlying tables, which your application will interact with. During a migration, Reshape will automatically create a new set of views and set up triggers to translate inserts and updates between the old and new schema. This means that every deployment is a three-phase process:
4949
50501. **Start migration** (`reshape migration start`): Sets up views and triggers to ensure both the new and old schema are usable at the same time.
51512. **Roll out application**: Your application can be gradually rolled out without downtime. The existing deployment will continue using the old schema whilst the new deployment uses the new schema.
52−3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
52+3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
5353
5454If the application deployment fails, you should run `reshape migration abort` which will roll back any changes made by `reshape migration start` without losing data.
5555
@@ -71,7 +71,7 @@ cargo install reshape
7171
7272#### Docker
7373
74−Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
74+Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
7575
7676```shell
7777docker run -v $(pwd):/usr/share/app fabianlindfors/reshape reshape migration start
@@ -167,9 +167,9 @@ Every action has a `type`. The supported types are detailed below.
167167
168168#### Create table
169169
170−The `create_table` action will create a new table with the specified columns, indices and constraints.
170+The `create_table` action will create a new table with the specified columns, indices and constraints. You can optionally provide an `up` option to backfill values from an existing table.
171171
172−*Example: create a `customers` table with a few columns and a primary key*
172+_Example: create a `customers` table with a few columns and a primary key_
173173
174174```toml
175175[[actions]]
@@ -187,13 +187,13 @@ primary_key = ["id"]
187187 type = "TEXT"
188188
189189 # Columns default to nullable
190− nullable = false
190+ nullable = false
191191
192192 # default can be any valid SQL value, in this case a string literal
193− default = "'PLACEHOLDER'"
193+ default = "'PLACEHOLDER'"
194194```
195195
196−*Example: create `users` and `items` tables with a foreign key between them*
196+_Example: create `users` and `items` tables with a foreign key between them_
197197
198198```toml
199199[[actions]]
@@ -226,11 +226,34 @@ primary_key = ["id"]
226226 referenced_columns = ["id"]
227227```
228228
229+_Example: create `profiles` table based on existing `users` table_
230+
231+```toml
232+[[actions]]
233+type = "create_table"
234+name = "profiles"
235+primary_key = ["user_id"]
236+
237+ [[actions.columns]]
238+ name = "user_id"
239+ type = "INTEGER"
240+
241+ [[actions.columns]]
242+ name = "user_email"
243+ type = "TEXT"
244+
245+ # Backfill from `users` table and copy `users.email` to `user_email` column
246+ # This will perform an upsert based on the primary key to avoid duplicate rows
247+ [actions.up]
248+ table = "users"
249+ values = { user_id = "id", user_email = "email" }
250+```
251+
229252#### Rename table
230253
231254The `rename_table` action will change the name of an existing table.
232255
233−*Example: change name of `users` table to `customers`*
256+_Example: change name of `users` table to `customers`_
234257
235258```toml
236259[[actions]]
@@ -243,7 +266,7 @@ new_name = "customers"
243266
244267The `remove_table` action will remove an existing table.
245268
246−*Example: remove `users` table*
269+_Example: remove `users` table_
247270
248271```toml
249272[[actions]]
@@ -255,7 +278,7 @@ table = "users"
255278
256279The `add_foreign_key` action will add a foreign key between two existing tables. The migration will fail if the existing column values aren't valid references.
257280
258−*Example: create foreign key from `items` to `users` table*
281+_Example: create foreign key from `items` to `users` table_
259282
260283```toml
261284[[actions]]
@@ -272,7 +295,7 @@ table = "items"
272295
273296The `remove_foreign_key` action will remove an existing foreign key. The foreign key will only be removed once the migration is completed, which means that your new application must continue to adhere to the foreign key constraint.
274297
275−*Example: remove foreign key `items_user_id_fkey` from `users` table*
298+_Example: remove foreign key `items_user_id_fkey` from `users` table_
276299
277300```toml
278301[[actions]]
@@ -285,9 +308,9 @@ foreign_key = "items_user_id_fkey"
285308
286309#### Add column
287310
288−The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column.
311+The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column. `up` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)).
289312
290−*Example: add a new column `reference` to table `products`*
313+_Example: add a new column `reference` to table `products`_
291314
292315```toml
293316[[actions]]
@@ -301,7 +324,7 @@ table = "products"
301324 default = "10"
302325```
303326
304−*Example: replace an existing `name` column with two new columns, `first_name` and `last_name`*
327+_Example: replace an existing `name` column with two new columns, `first_name` and `last_name`_
305328
306329```toml
307330[[actions]]
@@ -337,7 +360,7 @@ column = "name"
337360down = "first_name || ' ' || last_name"
338361```
339362
340−*Example: extract nested value from unstructured JSON `data` column to new `name` column*
363+_Example: extract nested value from unstructured JSON `data` column to new `name` column_
341364
342365```toml
343366[[actions]]
@@ -352,6 +375,25 @@ up = "data['path']['to']['value'] #>> '{}'"
352375 type = "TEXT"
353376```
354377
378+_Example: duplicate `email` column from `users` to `profiles` table_
379+
380+```toml
381+# `profiles` has `user_id` column which maps to `users.id`
382+[[actions]]
383+type = "add_column"
384+table = "profiles"
385+
386+ [actions.column]
387+ name = "email"
388+ type = "TEXT"
389+ nullable = false
390+
391+ # When `users` is updated in the old schema, we write the email value to `profiles`
392+ [actions.up]
393+ table = "users"
394+ value = "email"
395+ where = "user_id = id"
396+```
355397
356398#### Alter column
357399
@@ -359,7 +401,7 @@ The `alter_column` action enables many different changes to an existing column,
359401
360402When performing more complex changes than a rename, `up` and `down` should be provided. These should be SQL expressions which determine how to transform between the new and old version of the column. Inside those expressions, you can reference the current column value by the column name.
361403
362−*Example: rename `last_name` column on `users` table to `family_name`*
404+_Example: rename `last_name` column on `users` table to `family_name`_
363405
364406```toml
365407[[actions]]
@@ -371,7 +413,7 @@ column = "last_name"
371413 name = "family_name"
372414```
373415
374−*Example: change the type of `reference` column from `INTEGER` to `TEXT`*
416+_Example: change the type of `reference` column from `INTEGER` to `TEXT`_
375417
376418```toml
377419[[actions]]
@@ -386,7 +428,7 @@ down = "CAST(reference AS INTEGER)" # Converts from text value to integer
386428 type = "TEXT" # Previous type was 'INTEGER'
387429```
388430
389−*Example: increment all values of an `index` column by one*
431+_Example: increment all values of an `index` column by one_
390432
391433```toml
392434[[actions]]
@@ -401,7 +443,7 @@ down = "index - 1" # Decrement to revert for old schema
401443 name = "index"
402444```
403445
404−*Example: make `name` column not nullable*
446+_Example: make `name` column not nullable_
405447
406448```toml
407449[[actions]]
@@ -416,7 +458,7 @@ up = "COALESCE(name, 'N/A')"
416458 nullable = false
417459```
418460
419−*Example: change default value of `created_at` column to current time*
461+_Example: change default value of `created_at` column to current time_
420462
421463```toml
422464[[actions]]
@@ -430,11 +472,11 @@ column = "created_at"
430472
431473#### Remove column
432474
433−The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
475+The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. `down` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)) . The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
434476
435477Any indices that cover the column will be removed.
436478
437−*Example: remove column `name` from table `users`*
479+_Example: remove column `name` from table `users`_
438480
439481```toml
440482[[actions]]
@@ -446,13 +488,29 @@ column = "name"
446488down = "'N/A'"
447489```
448490
491+_Example: remove `email` column from `users` table and use column from `profiles` table instead_
492+
493+```toml
494+[[actions]]
495+type = "remove_column"
496+table = "users"
497+column = "email"
498+
499+ # Our application will use the `profiles.email` column instead
500+ # For backwards compatibility, we will write back to the removed `email` column whenever `profiles` is changed
501+ [actions.down]
502+ table = "profiles"
503+ value = "profiles.email"
504+ where = "users.id = profiles.user_id"
505+```
506+
449507### Indices
450508
451509#### Add index
452510
453511The `add_index` action will add a new index to an existing table.
454512
455−*Example: create a `users` table with a unique index on the `name` column*
513+_Example: create a `users` table with a unique index on the `name` column_
456514
457515```toml
458516[[actions]]
@@ -481,7 +539,7 @@ table = "users"
481539 unique = true
482540```
483541
484−*Example: add GIN index to `data` column on `products` table*
542+_Example: add GIN index to `data` column on `products` table_
485543
486544```toml
487545[[actions]]
@@ -500,7 +558,7 @@ table = "products"
500558
501559The `remove_index` action will remove an existing index. The index won't actually be removed until the migration is completed.
502560
503−*Example: remove the `name_idx` index*
561+_Example: remove the `name_idx` index_
504562
505563```toml
506564[[actions]]
@@ -514,7 +572,7 @@ index = "name_idx"
514572
515573The `create_enum` action will create a new [enum type](https://www.postgresql.org/docs/current/datatype-enum.html) with the specified values.
516574
517−*Example: add a new `mood` enum type with three possible values*
575+_Example: add a new `mood` enum type with three possible values_
518576
519577```toml
520578[[actions]]
@@ -527,7 +585,7 @@ values = ["happy", "ok", "sad"]
527585
528586The `remove_enum` action will remove an existing [enum type](https://www.postgresql.org/docs/current/datatype-enum.html). Make sure all usages of the enum has been removed before running the migration. The enum will only be removed once the migration is completed.
529587
530−*Example: remove the `mood` enum type*
588+_Example: remove the `mood` enum type_
531589
532590```toml
533591[[actions]]
@@ -545,7 +603,7 @@ There are three optional settings available which all accept SQL queries. All qu
545603- `complete`: run when a migration is completed using `reshape migration complete`
546604- `abort`: run when a migration is aborted using `reshape migration abort`
547605
548−*Example: enable PostGIS and pg_stat_statements extensions*
606+_Example: enable PostGIS and pg_stat_statements extensions_
549607
550608```toml
551609[[actions]]
@@ -562,6 +620,92 @@ abort = """
562620"""
563621```
564622
623+### Complex changes across tables
624+
625+The `up` and `down` options available when creating tables, adding columns and removing columns can also perform more complex changes that span tables.
626+
627+_Example: move `email` column from `users` to `profiles` table_
628+
629+```toml
630+[[actions]]
631+type = "add_column"
632+table = "profiles"
633+
634+ [actions.column]
635+ name = "email"
636+ type = "TEXT"
637+ nullable = false
638+
639+ # When `users` is updated in the old schema, we write the email value to `profiles`
640+ [actions.up]
641+ table = "users"
642+ value = "users.email"
643+ where = "profiles.user_id = users.id"
644+
645+[[actions]]
646+type = "remove_column"
647+table = "users"
648+column = "email"
649+
650+ # When `profiles` is changed in the new schema, we write the email address back to the removed column
651+ [actions.down]
652+ table = "profiles"
653+ value = "profiles.email"
654+ where = "users.id = profiles.user_id"
655+```
656+
657+_Example: turn a 1:N relationship between `users` and `accounts` into N:M and change the format of the associated `role`_
658+
659+```toml
660+# Add `user_account_connections` as a junction table
661+[[actions]]
662+type = "create_table"
663+name = "user_account_connections"
664+primary_key = ["account_id", "user_id"]
665+
666+ [[actions.columns]]
667+ name = "account_id"
668+ type = "INTEGER"
669+
670+ [[actions.columns]]
671+ name = "user_id"
672+ type = "INTEGER"
673+
674+ # `role` is currently stored directly on the `users` table but is part of the relationship
675+ [[actions.columns]]
676+ name = "role"
677+ type = "TEXT"
678+ nullable = false
679+
680+ # Backfill the new table from `users` and uppercase the `role`
681+ [actions.up]
682+ table = "users"
683+ values = { user_id = "id", account_id = "account_id", role = "UPPER(account_role)" }
684+ where = "user_account_connections.user_id = users.id"
685+
686+[[actions]]
687+type = "remove_column"
688+table = "users"
689+column = "account_id"
690+
691+ # When `user_account_connections` is updated, we write the `account_id` back to `users`
692+ [actions.down]
693+ table = "user_account_connections"
694+ value = "user_account_connections.account_id"
695+ where = "users.id = user_account_connections.user_id"
696+
697+[[actions]]
698+type = "remove_column"
699+table = "users"
700+column = "account_role"
701+
702+ # When `user_account_connections` is updated, we write the lowercase role back to `users`
703+ [actions.down]
704+ table = "user_account_connections"
705+ value = "LOWER(user_account_connections.role)"
706+ where = "users.id = user_account_connections.user_id"
707+```
708+
565709## Commands and options
566710
567711### `reshape migration start`
@@ -570,16 +714,16 @@ Starts a new migration, applying all migrations under `migrations/` that haven't
570714
571715#### Options
572716
573−*See also [Connection options](#connection-options)*
717+_See also [Connection options](#connection-options)_
574718
575−| Option | Default | Description |
576−| ------ | ------- | ----------- |
577−| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
578−| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
719+| Option | Default | Description |
720+| ------------------ | ------------- | --------------------------------------------------------------------------------------------------------------- |
721+| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
722+| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
579723
580724### `reshape migration complete`
581725
582−Completes migrations previously started with `reshape migration complete`.
726+Completes migrations previously started with `reshape migration complete`.
583727
584728#### Options
585729
@@ -587,7 +731,7 @@ See [Connection options](#connection-options)
587731
588732### `reshape migration abort`
589733
590−Aborts any migrations which haven't yet been completed.
734+Aborts any migrations which haven't yet been completed.
591735
592736#### Options
593737
@@ -603,8 +747,8 @@ The query should look something like `SET search_path TO migration_1_initial_mig
603747
604748#### Options
605749
606−| Option | Default | Description |
607−| ------ | ------- | ----------- |
750+| Option | Default | Description |
751+| -------- | ------------- | --------------------------------------------------------------------------------------------------------------- |
608752| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
609753
610754### Connection options
@@ -613,14 +757,14 @@ The options below can be used with all commands that communicate with Postgres.
613757
614758All options can also be set using environment variables instead of flags. If a `.env` file exists, then variables will be automatically loaded from there.
615759
616−| Option | Default | Environment variable | Description |
617−| ------ | ------- | -------------------- | ----------- |
618−| `--url` | | `DB_URL` | URL to your Postgres database |
619−| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
620−| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
621−| `--database` | `postgres` | `DB_NAME` | Database name |
622−| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
623−| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
760+| Option | Default | Environment variable | Description |
761+| ------------ | ----------- | -------------------- | ------------------------------------------- |
762+| `--url` | | `DB_URL` | URL to your Postgres database |
763+| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
764+| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
765+| `--database` | `postgres` | `DB_NAME` | Database name |
766+| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
767+| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
624768
625769## License
626770