@@ -2,54 +2,54 @@
22
33[](https://github.com/fabianlindfors/reshape/actions/workflows/test.yaml) [](https://github.com/fabianlindfors/reshape/releases)
44
5+> Also check out [ReshapeDB](https://reshapedb.com), a new database built from the ground up to make zero-downtime schema and data migrations as simple and safe as possible. If you'd like to chat about it, please [reach out](contact@reshapedb.com)!
6+
57Reshape is an easy-to-use, zero-downtime schema migration tool for Postgres. It automatically handles complex migrations that would normally require downtime or manual multi-step changes. During a migration, Reshape ensures both the old and new schema are available at the same time, allowing you to gradually roll out your application. It will also perform all changes without excessive locking, avoiding downtime caused by blocking other queries. For a more thorough introduction to Reshape, check out the [introductory blog post](https://fabianlindfors.se/blog/schema-migrations-in-postgres-using-reshape/).
68
79Designed for Postgres 12 and later.
810
9−*Note: Reshape is **experimental** and should not be used in production. It can (and probably will) break your application.*
10−
1111- [How it works](#how-it-works)
1212- [Getting started](#getting-started)
13− - [Installation](#installation)
14− - [Creating your first migration](#creating-your-first-migration)
15− - [Preparing your application](#preparing-your-application)
16− - [Running your migration](#running-your-migration)
17− - [Using during development](#using-during-development)
13+ - [Installation](#installation)
14+ - [Creating your first migration](#creating-your-first-migration)
15+ - [Preparing your application](#preparing-your-application)
16+ - [Running your migration](#running-your-migration)
17+ - [Using during development](#using-during-development)
1818- [Writing migrations](#writing-migrations)
19− - [Basics](#basics)
20− - [Tables](#tables)
21− - [Create table](#create-table)
22− - [Rename table](#rename-table)
23− - [Remove table](#remove-table)
24− - [Add foreign key](#add-foreign-key)
25− - [Remove foreign key](#remove-foreign-key)
26− - [Columns](#columns)
27− - [Add column](#add-column)
28− - [Alter column](#alter-column)
29− - [Remove column](#remove-column)
30− - [Indices](#indices)
31− - [Add index](#add-index)
32− - [Remove index](#remove-index)
33− - [Enums](#enums)
34− - [Create enum](#create-enum)
35− - [Remove enum](#remove-enum)
36− - [Custom](#custom)
19+ - [Basics](#basics)
20+ - [Tables](#tables)
21+ - [Create table](#create-table)
22+ - [Rename table](#rename-table)
23+ - [Remove table](#remove-table)
24+ - [Add foreign key](#add-foreign-key)
25+ - [Remove foreign key](#remove-foreign-key)
26+ - [Columns](#columns)
27+ - [Add column](#add-column)
28+ - [Alter column](#alter-column)
29+ - [Remove column](#remove-column)
30+ - [Indices](#indices)
31+ - [Add index](#add-index)
32+ - [Remove index](#remove-index)
33+ - [Enums](#enums)
34+ - [Create enum](#create-enum)
35+ - [Remove enum](#remove-enum)
36+ - [Custom](#custom)
37+ - [Complex changes across tables](#complex-changes-across-tables)
3738- [Commands and options](#commands-and-options)
38− - [`reshape migration start`](#reshape-migration-start)
39− - [`reshape migration complete`](#reshape-migration-complete)
40− - [`reshape migration abort`](#reshape-migration-abort)
41− - [`reshape schema-query`](#reshape-schema-query)
42− - [Connection options](#connection-options)
39+ - [`reshape migration start`](#reshape-migration-start)
40+ - [`reshape migration complete`](#reshape-migration-complete)
41+ - [`reshape migration abort`](#reshape-migration-abort)
42+ - [`reshape schema-query`](#reshape-schema-query)
43+ - [Connection options](#connection-options)
4344- [License](#license)
4445
45−
4646## How it works
4747
4848Reshape works by creating views that encapsulate the underlying tables, which your application will interact with. During a migration, Reshape will automatically create a new set of views and set up triggers to translate inserts and updates between the old and new schema. This means that every deployment is a three-phase process:
4949
50501. **Start migration** (`reshape migration start`): Sets up views and triggers to ensure both the new and old schema are usable at the same time.
51512. **Roll out application**: Your application can be gradually rolled out without downtime. The existing deployment will continue using the old schema whilst the new deployment uses the new schema.
52−3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
52+3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
5353
5454If the application deployment fails, you should run `reshape migration abort` which will roll back any changes made by `reshape migration start` without losing data.
5555
@@ -71,7 +71,7 @@ cargo install reshape
7171
7272#### Docker
7373
74−Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
74+Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
7575
7676```shell
7777docker run -v $(pwd):/usr/share/app fabianlindfors/reshape reshape migration start
@@ -108,6 +108,7 @@ Reshape relies on your application using a specific schema. When establishing th
108108- [Rust](https://github.com/fabianlindfors/reshape-helper)
109109- [Ruby (and Rails)](https://github.com/fabianlindfors/reshape-ruby)
110110- [Python (and Django)](https://github.com/fabianlindfors/reshape-python)
111+- [Go](https://github.com/leourbina/reshape-helper)
111112
112113If your application is not using one of the languages with an available helper library, you can instead generate the query with the command: `reshape schema-query`. To pass it along to your application, you can for example use an environment variable in your run script: `RESHAPE_SCHEMA_QUERY=$(reshape schema-query)`. Then in your application:
113114
@@ -166,9 +167,9 @@ Every action has a `type`. The supported types are detailed below.
166167
167168#### Create table
168169
169−The `create_table` action will create a new table with the specified columns, indices and constraints.
170+The `create_table` action will create a new table with the specified columns, indices and constraints. You can optionally provide an `up` option to backfill values from an existing table.
170171
171−*Example: create a `customers` table with a few columns and a primary key*
172+_Example: create a `customers` table with a few columns and a primary key_
172173
173174```toml
174175[[actions]]
@@ -186,13 +187,13 @@ primary_key = ["id"]
186187 type = "TEXT"
187188
188189 # Columns default to nullable
189− nullable = false
190+ nullable = false
190191
191192 # default can be any valid SQL value, in this case a string literal
192− default = "'PLACEHOLDER'"
193+ default = "'PLACEHOLDER'"
193194```
194195
195−*Example: create `users` and `items` tables with a foreign key between them*
196+_Example: create `users` and `items` tables with a foreign key between them_
196197
197198```toml
198199[[actions]]
@@ -225,11 +226,34 @@ primary_key = ["id"]
225226 referenced_columns = ["id"]
226227```
227228
229+_Example: create `profiles` table based on existing `users` table_
230+
231+```toml
232+[[actions]]
233+type = "create_table"
234+name = "profiles"
235+primary_key = ["user_id"]
236+
237+ [[actions.columns]]
238+ name = "user_id"
239+ type = "INTEGER"
240+
241+ [[actions.columns]]
242+ name = "user_email"
243+ type = "TEXT"
244+
245+ # Backfill from `users` table and copy `users.email` to `user_email` column
246+ # This will perform an upsert based on the primary key to avoid duplicate rows
247+ [actions.up]
248+ table = "users"
249+ values = { user_id = "id", user_email = "email" }
250+```
251+
228252#### Rename table
229253
230254The `rename_table` action will change the name of an existing table.
231255
232−*Example: change name of `users` table to `customers`*
256+_Example: change name of `users` table to `customers`_
233257
234258```toml
235259[[actions]]
@@ -242,7 +266,7 @@ new_name = "customers"
242266
243267The `remove_table` action will remove an existing table.
244268
245−*Example: remove `users` table*
269+_Example: remove `users` table_
246270
247271```toml
248272[[actions]]
@@ -254,7 +278,7 @@ table = "users"
254278
255279The `add_foreign_key` action will add a foreign key between two existing tables. The migration will fail if the existing column values aren't valid references.
256280
257−*Example: create foreign key from `items` to `users` table*
281+_Example: create foreign key from `items` to `users` table_
258282
259283```toml
260284[[actions]]
@@ -271,7 +295,7 @@ table = "items"
271295
272296The `remove_foreign_key` action will remove an existing foreign key. The foreign key will only be removed once the migration is completed, which means that your new application must continue to adhere to the foreign key constraint.
273297
274−*Example: remove foreign key `items_user_id_fkey` from `users` table*
298+_Example: remove foreign key `items_user_id_fkey` from `users` table_
275299
276300```toml
277301[[actions]]
@@ -284,9 +308,9 @@ foreign_key = "items_user_id_fkey"
284308
285309#### Add column
286310
287−The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column.
311+The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column. `up` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)).
288312
289−*Example: add a new column `reference` to table `products`*
313+_Example: add a new column `reference` to table `products`_
290314
291315```toml
292316[[actions]]
@@ -300,7 +324,7 @@ table = "products"
300324 default = "10"
301325```
302326
303−*Example: replace an existing `name` column with two new columns, `first_name` and `last_name`*
327+_Example: replace an existing `name` column with two new columns, `first_name` and `last_name`_
304328
305329```toml
306330[[actions]]
@@ -336,7 +360,7 @@ column = "name"
336360down = "first_name || ' ' || last_name"
337361```
338362
339−*Example: extract nested value from unstructured JSON `data` column to new `name` column*
363+_Example: extract nested value from unstructured JSON `data` column to new `name` column_
340364
341365```toml
342366[[actions]]
@@ -351,6 +375,25 @@ up = "data['path']['to']['value'] #>> '{}'"
351375 type = "TEXT"
352376```
353377
378+_Example: duplicate `email` column from `users` to `profiles` table_
379+
380+```toml
381+# `profiles` has `user_id` column which maps to `users.id`
382+[[actions]]
383+type = "add_column"
384+table = "profiles"
385+
386+ [actions.column]
387+ name = "email"
388+ type = "TEXT"
389+ nullable = false
390+
391+ # When `users` is updated in the old schema, we write the email value to `profiles`
392+ [actions.up]
393+ table = "users"
394+ value = "email"
395+ where = "user_id = id"
396+```
354397
355398#### Alter column
356399
@@ -358,7 +401,7 @@ The `alter_column` action enables many different changes to an existing column,
358401
359402When performing more complex changes than a rename, `up` and `down` should be provided. These should be SQL expressions which determine how to transform between the new and old version of the column. Inside those expressions, you can reference the current column value by the column name.
360403
361−*Example: rename `last_name` column on `users` table to `family_name`*
404+_Example: rename `last_name` column on `users` table to `family_name`_
362405
363406```toml
364407[[actions]]
@@ -370,7 +413,7 @@ column = "last_name"
370413 name = "family_name"
371414```
372415
373−*Example: change the type of `reference` column from `INTEGER` to `TEXT`*
416+_Example: change the type of `reference` column from `INTEGER` to `TEXT`_
374417
375418```toml
376419[[actions]]
@@ -385,7 +428,7 @@ down = "CAST(reference AS INTEGER)" # Converts from text value to integer
385428 type = "TEXT" # Previous type was 'INTEGER'
386429```
387430
388−*Example: increment all values of an `index` column by one*
431+_Example: increment all values of an `index` column by one_
389432
390433```toml
391434[[actions]]
@@ -400,7 +443,7 @@ down = "index - 1" # Decrement to revert for old schema
400443 name = "index"
401444```
402445
403−*Example: make `name` column not nullable*
446+_Example: make `name` column not nullable_
404447
405448```toml
406449[[actions]]
@@ -415,7 +458,7 @@ up = "COALESCE(name, 'N/A')"
415458 nullable = false
416459```
417460
418−*Example: change default value of `created_at` column to current time*
461+_Example: change default value of `created_at` column to current time_
419462
420463```toml
421464[[actions]]
@@ -429,11 +472,11 @@ column = "created_at"
429472
430473#### Remove column
431474
432−The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
475+The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. `down` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)) . The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
433476
434477Any indices that cover the column will be removed.
435478
436−*Example: remove column `name` from table `users`*
479+_Example: remove column `name` from table `users`_
437480
438481```toml
439482[[actions]]
@@ -445,13 +488,29 @@ column = "name"
445488down = "'N/A'"
446489```
447490
491+_Example: remove `email` column from `users` table and use column from `profiles` table instead_
492+
493+```toml
494+[[actions]]
495+type = "remove_column"
496+table = "users"
497+column = "email"
498+
499+ # Our application will use the `profiles.email` column instead
500+ # For backwards compatibility, we will write back to the removed `email` column whenever `profiles` is changed
501+ [actions.down]
502+ table = "profiles"
503+ value = "profiles.email"
504+ where = "users.id = profiles.user_id"
505+```
506+
448507### Indices
449508
450509#### Add index
451510
452511The `add_index` action will add a new index to an existing table.
453512
454−*Example: create a `users` table with a unique index on the `name` column*
513+_Example: create a `users` table with a unique index on the `name` column_
455514
456515```toml
457516[[actions]]
@@ -480,7 +539,7 @@ table = "users"
480539 unique = true
481540```
482541
483−*Example: add GIN index to `data` column on `products` table*
542+_Example: add GIN index to `data` column on `products` table_
484543
485544```toml
486545[[actions]]
@@ -499,7 +558,7 @@ table = "products"
499558
500559The `remove_index` action will remove an existing index. The index won't actually be removed until the migration is completed.
501560
502−*Example: remove the `name_idx` index*
561+_Example: remove the `name_idx` index_
503562
504563```toml
505564[[actions]]
@@ -513,7 +572,7 @@ index = "name_idx"
513572
514573The `create_enum` action will create a new [enum type](https://www.postgresql.org/docs/current/datatype-enum.html) with the specified values.
515574
516−*Example: add a new `mood` enum type with three possible values*
575+_Example: add a new `mood` enum type with three possible values_
517576
518577```toml
519578[[actions]]
@@ -526,7 +585,7 @@ values = ["happy", "ok", "sad"]
526585
527586The `remove_enum` action will remove an existing [enum type](https://www.postgresql.org/docs/current/datatype-enum.html). Make sure all usages of the enum has been removed before running the migration. The enum will only be removed once the migration is completed.
528587
529−*Example: remove the `mood` enum type*
588+_Example: remove the `mood` enum type_
530589
531590```toml
532591[[actions]]
@@ -544,7 +603,7 @@ There are three optional settings available which all accept SQL queries. All qu
544603- `complete`: run when a migration is completed using `reshape migration complete`
545604- `abort`: run when a migration is aborted using `reshape migration abort`
546605
547−*Example: enable PostGIS and pg_stat_statements extensions*
606+_Example: enable PostGIS and pg_stat_statements extensions_
548607
549608```toml
550609[[actions]]
@@ -561,6 +620,92 @@ abort = """
561620"""
562621```
563622
623+### Complex changes across tables
624+
625+The `up` and `down` options available when creating tables, adding columns and removing columns can also perform more complex changes that span tables.
626+
627+_Example: move `email` column from `users` to `profiles` table_
628+
629+```toml
630+[[actions]]
631+type = "add_column"
632+table = "profiles"
633+
634+ [actions.column]
635+ name = "email"
636+ type = "TEXT"
637+ nullable = false
638+
639+ # When `users` is updated in the old schema, we write the email value to `profiles`
640+ [actions.up]
641+ table = "users"
642+ value = "users.email"
643+ where = "profiles.user_id = users.id"
644+
645+[[actions]]
646+type = "remove_column"
647+table = "users"
648+column = "email"
649+
650+ # When `profiles` is changed in the new schema, we write the email address back to the removed column
651+ [actions.down]
652+ table = "profiles"
653+ value = "profiles.email"
654+ where = "users.id = profiles.user_id"
655+```
656+
657+_Example: turn a 1:N relationship between `users` and `accounts` into N:M and change the format of the associated `role`_
658+
659+```toml
660+# Add `user_account_connections` as a junction table
661+[[actions]]
662+type = "create_table"
663+name = "user_account_connections"
664+primary_key = ["account_id", "user_id"]
665+
666+ [[actions.columns]]
667+ name = "account_id"
668+ type = "INTEGER"
669+
670+ [[actions.columns]]
671+ name = "user_id"
672+ type = "INTEGER"
673+
674+ # `role` is currently stored directly on the `users` table but is part of the relationship
675+ [[actions.columns]]
676+ name = "role"
677+ type = "TEXT"
678+ nullable = false
679+
680+ # Backfill the new table from `users` and uppercase the `role`
681+ [actions.up]
682+ table = "users"
683+ values = { user_id = "id", account_id = "account_id", role = "UPPER(account_role)" }
684+ where = "user_account_connections.user_id = users.id"
685+
686+[[actions]]
687+type = "remove_column"
688+table = "users"
689+column = "account_id"
690+
691+ # When `user_account_connections` is updated, we write the `account_id` back to `users`
692+ [actions.down]
693+ table = "user_account_connections"
694+ value = "user_account_connections.account_id"
695+ where = "users.id = user_account_connections.user_id"
696+
697+[[actions]]
698+type = "remove_column"
699+table = "users"
700+column = "account_role"
701+
702+ # When `user_account_connections` is updated, we write the lowercase role back to `users`
703+ [actions.down]
704+ table = "user_account_connections"
705+ value = "LOWER(user_account_connections.role)"
706+ where = "users.id = user_account_connections.user_id"
707+```
708+
564709## Commands and options
565710
566711### `reshape migration start`
@@ -569,16 +714,16 @@ Starts a new migration, applying all migrations under `migrations/` that haven't
569714
570715#### Options
571716
572−*See also [Connection options](#connection-options)*
717+_See also [Connection options](#connection-options)_
573718
574−| Option | Default | Description |
575−| ------ | ------- | ----------- |
576−| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
577−| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
719+| Option | Default | Description |
720+| ------------------ | ------------- | --------------------------------------------------------------------------------------------------------------- |
721+| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
722+| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
578723
579724### `reshape migration complete`
580725
581−Completes migrations previously started with `reshape migration complete`.
726+Completes migrations previously started with `reshape migration complete`.
582727
583728#### Options
584729
@@ -586,7 +731,7 @@ See [Connection options](#connection-options)
586731
587732### `reshape migration abort`
588733
589−Aborts any migrations which haven't yet been completed.
734+Aborts any migrations which haven't yet been completed.
590735
591736#### Options
592737
@@ -602,8 +747,8 @@ The query should look something like `SET search_path TO migration_1_initial_mig
602747
603748#### Options
604749
605−| Option | Default | Description |
606−| ------ | ------- | ----------- |
750+| Option | Default | Description |
751+| -------- | ------------- | --------------------------------------------------------------------------------------------------------------- |
607752| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
608753
609754### Connection options
@@ -612,14 +757,14 @@ The options below can be used with all commands that communicate with Postgres.
612757
613758All options can also be set using environment variables instead of flags. If a `.env` file exists, then variables will be automatically loaded from there.
614759
615−| Option | Default | Environment variable | Description |
616−| ------ | ------- | -------------------- | ----------- |
617−| `--url` | | `DB_URL` | URL to your Postgres database |
618−| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
619−| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
620−| `--database` | `postgres` | `DB_NAME` | Database name |
621−| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
622−| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
760+| Option | Default | Environment variable | Description |
761+| ------------ | ----------- | -------------------- | ------------------------------------------- |
762+| `--url` | | `DB_URL` | URL to your Postgres database |
763+| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
764+| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
765+| `--database` | `postgres` | `DB_NAME` | Database name |
766+| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
767+| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
623768
624769## License
625770