@@ -2,54 +2,54 @@
22
33[](https://github.com/fabianlindfors/reshape/actions/workflows/test.yaml) [](https://github.com/fabianlindfors/reshape/releases)
44
5+> Also check out [ReshapeDB](https://reshapedb.com), a new database built from the ground up to make zero-downtime schema and data migrations as simple and safe as possible. If you'd like to chat about it, please [reach out](contact@reshapedb.com)!
6+
57Reshape is an easy-to-use, zero-downtime schema migration tool for Postgres. It automatically handles complex migrations that would normally require downtime or manual multi-step changes. During a migration, Reshape ensures both the old and new schema are available at the same time, allowing you to gradually roll out your application. It will also perform all changes without excessive locking, avoiding downtime caused by blocking other queries. For a more thorough introduction to Reshape, check out the [introductory blog post](https://fabianlindfors.se/blog/schema-migrations-in-postgres-using-reshape/).
68
79Designed for Postgres 12 and later.
810
9−*Note: Reshape is **experimental** and should not be used in production. It can (and probably will) break your application.*
10−
1111- [How it works](#how-it-works)
1212- [Getting started](#getting-started)
13− - [Installation](#installation)
14− - [Creating your first migration](#creating-your-first-migration)
15− - [Preparing your application](#preparing-your-application)
16− - [Running your migration](#running-your-migration)
17− - [Using during development](#using-during-development)
13+ - [Installation](#installation)
14+ - [Creating your first migration](#creating-your-first-migration)
15+ - [Preparing your application](#preparing-your-application)
16+ - [Running your migration](#running-your-migration)
17+ - [Using during development](#using-during-development)
1818- [Writing migrations](#writing-migrations)
19− - [Basics](#basics)
20− - [Tables](#tables)
21− - [Create table](#create-table)
22− - [Rename table](#rename-table)
23− - [Remove table](#remove-table)
24− - [Add foreign key](#add-foreign-key)
25− - [Remove foreign key](#remove-foreign-key)
26− - [Columns](#columns)
27− - [Add column](#add-column)
28− - [Alter column](#alter-column)
29− - [Remove column](#remove-column)
30− - [Indices](#indices)
31− - [Add index](#add-index)
32− - [Remove index](#remove-index)
33− - [Enums](#enums)
34− - [Create enum](#create-enum)
35− - [Remove enum](#remove-enum)
36− - [Custom](#custom)
19+ - [Basics](#basics)
20+ - [Tables](#tables)
21+ - [Create table](#create-table)
22+ - [Rename table](#rename-table)
23+ - [Remove table](#remove-table)
24+ - [Add foreign key](#add-foreign-key)
25+ - [Remove foreign key](#remove-foreign-key)
26+ - [Columns](#columns)
27+ - [Add column](#add-column)
28+ - [Alter column](#alter-column)
29+ - [Remove column](#remove-column)
30+ - [Indices](#indices)
31+ - [Add index](#add-index)
32+ - [Remove index](#remove-index)
33+ - [Enums](#enums)
34+ - [Create enum](#create-enum)
35+ - [Remove enum](#remove-enum)
36+ - [Custom](#custom)
37+ - [Complex changes across tables](#complex-changes-across-tables)
3738- [Commands and options](#commands-and-options)
38− - [`reshape migration start`](#reshape-migration-start)
39− - [`reshape migration complete`](#reshape-migration-complete)
40− - [`reshape migration abort`](#reshape-migration-abort)
41− - [`reshape schema-query`](#reshape-schema-query)
42− - [Connection options](#connection-options)
39+ - [`reshape migration start`](#reshape-migration-start)
40+ - [`reshape migration complete`](#reshape-migration-complete)
41+ - [`reshape migration abort`](#reshape-migration-abort)
42+ - [`reshape schema-query`](#reshape-schema-query)
43+ - [Connection options](#connection-options)
4344- [License](#license)
4445
45−
4646## How it works
4747
4848Reshape works by creating views that encapsulate the underlying tables, which your application will interact with. During a migration, Reshape will automatically create a new set of views and set up triggers to translate inserts and updates between the old and new schema. This means that every deployment is a three-phase process:
4949
50501. **Start migration** (`reshape migration start`): Sets up views and triggers to ensure both the new and old schema are usable at the same time.
51512. **Roll out application**: Your application can be gradually rolled out without downtime. The existing deployment will continue using the old schema whilst the new deployment uses the new schema.
52−3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
52+3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
5353
5454If the application deployment fails, you should run `reshape migration abort` which will roll back any changes made by `reshape migration start` without losing data.
5555
@@ -71,7 +71,7 @@ cargo install reshape
7171
7272#### Docker
7373
74−Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
74+Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
7575
7676```shell
7777docker run -v $(pwd):/usr/share/app fabianlindfors/reshape reshape migration start
@@ -107,6 +107,8 @@ Reshape relies on your application using a specific schema. When establishing th
107107
108108- [Rust](https://github.com/fabianlindfors/reshape-helper)
109109- [Ruby (and Rails)](https://github.com/fabianlindfors/reshape-ruby)
110+- [Python (and Django)](https://github.com/fabianlindfors/reshape-python)
111+- [Go](https://github.com/leourbina/reshape-helper)
110112
111113If your application is not using one of the languages with an available helper library, you can instead generate the query with the command: `reshape schema-query`. To pass it along to your application, you can for example use an environment variable in your run script: `RESHAPE_SCHEMA_QUERY=$(reshape schema-query)`. Then in your application:
112114
@@ -165,9 +167,9 @@ Every action has a `type`. The supported types are detailed below.
165167
166168#### Create table
167169
168−The `create_table` action will create a new table with the specified columns, indices and constraints.
170+The `create_table` action will create a new table with the specified columns, indices and constraints. You can optionally provide an `up` option to backfill values from an existing table.
169171
170−*Example: create a `customers` table with a few columns and a primary key*
172+_Example: create a `customers` table with a few columns and a primary key_
171173
172174```toml
173175[[actions]]
@@ -185,13 +187,13 @@ primary_key = ["id"]
185187 type = "TEXT"
186188
187189 # Columns default to nullable
188− nullable = false
190+ nullable = false
189191
190192 # default can be any valid SQL value, in this case a string literal
191− default = "'PLACEHOLDER'"
193+ default = "'PLACEHOLDER'"
192194```
193195
194−*Example: create `users` and `items` tables with a foreign key between them*
196+_Example: create `users` and `items` tables with a foreign key between them_
195197
196198```toml
197199[[actions]]
@@ -224,11 +226,34 @@ primary_key = ["id"]
224226 referenced_columns = ["id"]
225227```
226228
229+_Example: create `profiles` table based on existing `users` table_
230+
231+```toml
232+[[actions]]
233+type = "create_table"
234+name = "profiles"
235+primary_key = ["user_id"]
236+
237+ [[actions.columns]]
238+ name = "user_id"
239+ type = "INTEGER"
240+
241+ [[actions.columns]]
242+ name = "user_email"
243+ type = "TEXT"
244+
245+ # Backfill from `users` table and copy `users.email` to `user_email` column
246+ # This will perform an upsert based on the primary key to avoid duplicate rows
247+ [actions.up]
248+ table = "users"
249+ values = { user_id = "id", user_email = "email" }
250+```
251+
227252#### Rename table
228253
229254The `rename_table` action will change the name of an existing table.
230255
231−*Example: change name of `users` table to `customers`*
256+_Example: change name of `users` table to `customers`_
232257
233258```toml
234259[[actions]]
@@ -241,7 +266,7 @@ new_name = "customers"
241266
242267The `remove_table` action will remove an existing table.
243268
244−*Example: remove `users` table*
269+_Example: remove `users` table_
245270
246271```toml
247272[[actions]]
@@ -253,7 +278,7 @@ table = "users"
253278
254279The `add_foreign_key` action will add a foreign key between two existing tables. The migration will fail if the existing column values aren't valid references.
255280
256−*Example: create foreign key from `items` to `users` table*
281+_Example: create foreign key from `items` to `users` table_
257282
258283```toml
259284[[actions]]
@@ -270,7 +295,7 @@ table = "items"
270295
271296The `remove_foreign_key` action will remove an existing foreign key. The foreign key will only be removed once the migration is completed, which means that your new application must continue to adhere to the foreign key constraint.
272297
273−*Example: remove foreign key `items_user_id_fkey` from `users` table*
298+_Example: remove foreign key `items_user_id_fkey` from `users` table_
274299
275300```toml
276301[[actions]]
@@ -283,9 +308,9 @@ foreign_key = "items_user_id_fkey"
283308
284309#### Add column
285310
286−The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column.
311+The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column. `up` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)).
287312
288−*Example: add a new column `reference` to table `products`*
313+_Example: add a new column `reference` to table `products`_
289314
290315```toml
291316[[actions]]
@@ -299,7 +324,7 @@ table = "products"
299324 default = "10"
300325```
301326
302−*Example: replace an existing `name` column with two new columns, `first_name` and `last_name`*
327+_Example: replace an existing `name` column with two new columns, `first_name` and `last_name`_
303328
304329```toml
305330[[actions]]
@@ -335,7 +360,7 @@ column = "name"
335360down = "first_name || ' ' || last_name"
336361```
337362
338−*Example: extract nested value from unstructured JSON `data` column to new `name` column*
363+_Example: extract nested value from unstructured JSON `data` column to new `name` column_
339364
340365```toml
341366[[actions]]
@@ -350,6 +375,25 @@ up = "data['path']['to']['value'] #>> '{}'"
350375 type = "TEXT"
351376```
352377
378+_Example: duplicate `email` column from `users` to `profiles` table_
379+
380+```toml
381+# `profiles` has `user_id` column which maps to `users.id`
382+[[actions]]
383+type = "add_column"
384+table = "profiles"
385+
386+ [actions.column]
387+ name = "email"
388+ type = "TEXT"
389+ nullable = false
390+
391+ # When `users` is updated in the old schema, we write the email value to `profiles`
392+ [actions.up]
393+ table = "users"
394+ value = "email"
395+ where = "user_id = id"
396+```
353397
354398#### Alter column
355399
@@ -357,7 +401,7 @@ The `alter_column` action enables many different changes to an existing column,
357401
358402When performing more complex changes than a rename, `up` and `down` should be provided. These should be SQL expressions which determine how to transform between the new and old version of the column. Inside those expressions, you can reference the current column value by the column name.
359403
360−*Example: rename `last_name` column on `users` table to `family_name`*
404+_Example: rename `last_name` column on `users` table to `family_name`_
361405
362406```toml
363407[[actions]]
@@ -369,7 +413,7 @@ column = "last_name"
369413 name = "family_name"
370414```
371415
372−*Example: change the type of `reference` column from `INTEGER` to `TEXT`*
416+_Example: change the type of `reference` column from `INTEGER` to `TEXT`_
373417
374418```toml
375419[[actions]]
@@ -384,7 +428,7 @@ down = "CAST(reference AS INTEGER)" # Converts from text value to integer
384428 type = "TEXT" # Previous type was 'INTEGER'
385429```
386430
387−*Example: increment all values of an `index` column by one*
431+_Example: increment all values of an `index` column by one_
388432
389433```toml
390434[[actions]]
@@ -399,7 +443,7 @@ down = "index - 1" # Decrement to revert for old schema
399443 name = "index"
400444```
401445
402−*Example: make `name` column not nullable*
446+_Example: make `name` column not nullable_
403447
404448```toml
405449[[actions]]
@@ -414,7 +458,7 @@ up = "COALESCE(name, 'N/A')"
414458 nullable = false
415459```
416460
417−*Example: change default value of `created_at` column to current time*
461+_Example: change default value of `created_at` column to current time_
418462
419463```toml
420464[[actions]]
@@ -428,11 +472,11 @@ column = "created_at"
428472
429473#### Remove column
430474
431−The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
475+The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. `down` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)) . The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
432476
433477Any indices that cover the column will be removed.
434478
435−*Example: remove column `name` from table `users`*
479+_Example: remove column `name` from table `users`_
436480
437481```toml
438482[[actions]]
@@ -444,18 +488,34 @@ column = "name"
444488down = "'N/A'"
445489```
446490
491+_Example: remove `email` column from `users` table and use column from `profiles` table instead_
492+
493+```toml
494+[[actions]]
495+type = "remove_column"
496+table = "users"
497+column = "email"
498+
499+ # Our application will use the `profiles.email` column instead
500+ # For backwards compatibility, we will write back to the removed `email` column whenever `profiles` is changed
501+ [actions.down]
502+ table = "profiles"
503+ value = "profiles.email"
504+ where = "users.id = profiles.user_id"
505+```
506+
447507### Indices
448508
449509#### Add index
450510
451511The `add_index` action will add a new index to an existing table.
452512
453−*Example: create a `users` table with a unique index on the `name` column*
513+_Example: create a `users` table with a unique index on the `name` column_
454514
455515```toml
456516[[actions]]
457517type = "create_table"
458−table = "users"
518+name = "users"
459519primary_key = "id"
460520
461521 [[actions.columns]]
@@ -479,7 +539,7 @@ table = "users"
479539 unique = true
480540```
481541
482−*Example: add GIN index to `data` column on `products` table*
542+_Example: add GIN index to `data` column on `products` table_
483543
484544```toml
485545[[actions]]
@@ -498,7 +558,7 @@ table = "products"
498558
499559The `remove_index` action will remove an existing index. The index won't actually be removed until the migration is completed.
500560
501−*Example: remove the `name_idx` index*
561+_Example: remove the `name_idx` index_
502562
503563```toml
504564[[actions]]
@@ -512,7 +572,7 @@ index = "name_idx"
512572
513573The `create_enum` action will create a new [enum type](https://www.postgresql.org/docs/current/datatype-enum.html) with the specified values.
514574
515−*Example: add a new `mood` enum type with three possible values*
575+_Example: add a new `mood` enum type with three possible values_
516576
517577```toml
518578[[actions]]
@@ -525,7 +585,7 @@ values = ["happy", "ok", "sad"]
525585
526586The `remove_enum` action will remove an existing [enum type](https://www.postgresql.org/docs/current/datatype-enum.html). Make sure all usages of the enum has been removed before running the migration. The enum will only be removed once the migration is completed.
527587
528−*Example: remove the `mood` enum type*
588+_Example: remove the `mood` enum type_
529589
530590```toml
531591[[actions]]
@@ -543,7 +603,7 @@ There are three optional settings available which all accept SQL queries. All qu
543603- `complete`: run when a migration is completed using `reshape migration complete`
544604- `abort`: run when a migration is aborted using `reshape migration abort`
545605
546−*Example: enable PostGIS and pg_stat_statements extensions*
606+_Example: enable PostGIS and pg_stat_statements extensions_
547607
548608```toml
549609[[actions]]
@@ -555,11 +615,97 @@ start = """
555615"""
556616
557617abort = """
558− DROP EXTENSION IF NOT EXISTS postgis;
559− DROP EXTENSION IF NOT EXISTS pg_stat_statements;
618+ DROP EXTENSION IF EXISTS postgis;
619+ DROP EXTENSION IF EXISTS pg_stat_statements;
560620"""
561621```
562622
623+### Complex changes across tables
624+
625+The `up` and `down` options available when creating tables, adding columns and removing columns can also perform more complex changes that span tables.
626+
627+_Example: move `email` column from `users` to `profiles` table_
628+
629+```toml
630+[[actions]]
631+type = "add_column"
632+table = "profiles"
633+
634+ [actions.column]
635+ name = "email"
636+ type = "TEXT"
637+ nullable = false
638+
639+ # When `users` is updated in the old schema, we write the email value to `profiles`
640+ [actions.up]
641+ table = "users"
642+ value = "users.email"
643+ where = "profiles.user_id = users.id"
644+
645+[[actions]]
646+type = "remove_column"
647+table = "users"
648+column = "email"
649+
650+ # When `profiles` is changed in the new schema, we write the email address back to the removed column
651+ [actions.down]
652+ table = "profiles"
653+ value = "profiles.email"
654+ where = "users.id = profiles.user_id"
655+```
656+
657+_Example: turn a 1:N relationship between `users` and `accounts` into N:M and change the format of the associated `role`_
658+
659+```toml
660+# Add `user_account_connections` as a junction table
661+[[actions]]
662+type = "create_table"
663+name = "user_account_connections"
664+primary_key = ["account_id", "user_id"]
665+
666+ [[actions.columns]]
667+ name = "account_id"
668+ type = "INTEGER"
669+
670+ [[actions.columns]]
671+ name = "user_id"
672+ type = "INTEGER"
673+
674+ # `role` is currently stored directly on the `users` table but is part of the relationship
675+ [[actions.columns]]
676+ name = "role"
677+ type = "TEXT"
678+ nullable = false
679+
680+ # Backfill the new table from `users` and uppercase the `role`
681+ [actions.up]
682+ table = "users"
683+ values = { user_id = "id", account_id = "account_id", role = "UPPER(account_role)" }
684+ where = "user_account_connections.user_id = users.id"
685+
686+[[actions]]
687+type = "remove_column"
688+table = "users"
689+column = "account_id"
690+
691+ # When `user_account_connections` is updated, we write the `account_id` back to `users`
692+ [actions.down]
693+ table = "user_account_connections"
694+ value = "user_account_connections.account_id"
695+ where = "users.id = user_account_connections.user_id"
696+
697+[[actions]]
698+type = "remove_column"
699+table = "users"
700+column = "account_role"
701+
702+ # When `user_account_connections` is updated, we write the lowercase role back to `users`
703+ [actions.down]
704+ table = "user_account_connections"
705+ value = "LOWER(user_account_connections.role)"
706+ where = "users.id = user_account_connections.user_id"
707+```
708+
563709## Commands and options
564710
565711### `reshape migration start`
@@ -568,16 +714,16 @@ Starts a new migration, applying all migrations under `migrations/` that haven't
568714
569715#### Options
570716
571−*See also [Connection options](#connection-options)*
717+_See also [Connection options](#connection-options)_
572718
573−| Option | Default | Description |
574−| ------ | ------- | ----------- |
575−| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
576−| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
719+| Option | Default | Description |
720+| ------------------ | ------------- | --------------------------------------------------------------------------------------------------------------- |
721+| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
722+| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
577723
578724### `reshape migration complete`
579725
580−Completes migrations previously started with `reshape migration complete`.
726+Completes migrations previously started with `reshape migration complete`.
581727
582728#### Options
583729
@@ -585,7 +731,7 @@ See [Connection options](#connection-options)
585731
586732### `reshape migration abort`
587733
588−Aborts any migrations which haven't yet been completed.
734+Aborts any migrations which haven't yet been completed.
589735
590736#### Options
591737
@@ -601,8 +747,8 @@ The query should look something like `SET search_path TO migration_1_initial_mig
601747
602748#### Options
603749
604−| Option | Default | Description |
605−| ------ | ------- | ----------- |
750+| Option | Default | Description |
751+| -------- | ------------- | --------------------------------------------------------------------------------------------------------------- |
606752| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
607753
608754### Connection options
@@ -611,14 +757,14 @@ The options below can be used with all commands that communicate with Postgres.
611757
612758All options can also be set using environment variables instead of flags. If a `.env` file exists, then variables will be automatically loaded from there.
613759
614−| Option | Default | Environment variable | Description |
615−| ------ | ------- | -------------------- | ----------- |
616−| `--url` | | `DB_URL` | URL to your Postgres database |
617−| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
618−| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
619−| `--database` | `postgres` | `DB_NAME` | Database name |
620−| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
621−| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
760+| Option | Default | Environment variable | Description |
761+| ------------ | ----------- | -------------------- | ------------------------------------------- |
762+| `--url` | | `DB_URL` | URL to your Postgres database |
763+| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
764+| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
765+| `--database` | `postgres` | `DB_NAME` | Database name |
766+| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
767+| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
622768
623769## License
624770