@@ -2,54 +2,54 @@
22
33[](https://github.com/fabianlindfors/reshape/actions/workflows/test.yaml) [](https://github.com/fabianlindfors/reshape/releases)
44
5+> Also check out [ReshapeDB](https://reshapedb.com), a new database built from the ground up to make zero-downtime schema and data migrations as simple and safe as possible. If you'd like to chat about it, please [reach out](contact@reshapedb.com)!
6+
57Reshape is an easy-to-use, zero-downtime schema migration tool for Postgres. It automatically handles complex migrations that would normally require downtime or manual multi-step changes. During a migration, Reshape ensures both the old and new schema are available at the same time, allowing you to gradually roll out your application. It will also perform all changes without excessive locking, avoiding downtime caused by blocking other queries. For a more thorough introduction to Reshape, check out the [introductory blog post](https://fabianlindfors.se/blog/schema-migrations-in-postgres-using-reshape/).
68
79Designed for Postgres 12 and later.
810
9−*Note: Reshape is **experimental** and should not be used in production. It can (and probably will) break your application.*
10−
1111- [How it works](#how-it-works)
1212- [Getting started](#getting-started)
13− - [Installation](#installation)
14− - [Creating your first migration](#creating-your-first-migration)
15− - [Preparing your application](#preparing-your-application)
16− - [Running your migration](#running-your-migration)
17− - [Using during development](#using-during-development)
13+ - [Installation](#installation)
14+ - [Creating your first migration](#creating-your-first-migration)
15+ - [Preparing your application](#preparing-your-application)
16+ - [Running your migration](#running-your-migration)
17+ - [Using during development](#using-during-development)
1818- [Writing migrations](#writing-migrations)
19− - [Basics](#basics)
20− - [Tables](#tables)
21− - [Create table](#create-table)
22− - [Rename table](#rename-table)
23− - [Remove table](#remove-table)
24− - [Add foreign key](#add-foreign-key)
25− - [Remove foreign key](#remove-foreign-key)
26− - [Columns](#columns)
27− - [Add column](#add-column)
28− - [Alter column](#alter-column)
29− - [Remove column](#remove-column)
30− - [Indices](#indices)
31− - [Add index](#add-index)
32− - [Remove index](#remove-index)
33− - [Enums](#enums)
34− - [Create enum](#create-enum)
35− - [Remove enum](#remove-enum)
36− - [Custom](#custom)
19+ - [Basics](#basics)
20+ - [Tables](#tables)
21+ - [Create table](#create-table)
22+ - [Rename table](#rename-table)
23+ - [Remove table](#remove-table)
24+ - [Add foreign key](#add-foreign-key)
25+ - [Remove foreign key](#remove-foreign-key)
26+ - [Columns](#columns)
27+ - [Add column](#add-column)
28+ - [Alter column](#alter-column)
29+ - [Remove column](#remove-column)
30+ - [Indices](#indices)
31+ - [Add index](#add-index)
32+ - [Remove index](#remove-index)
33+ - [Enums](#enums)
34+ - [Create enum](#create-enum)
35+ - [Remove enum](#remove-enum)
36+ - [Custom](#custom)
37+ - [Complex changes across tables](#complex-changes-across-tables)
3738- [Commands and options](#commands-and-options)
38− - [`reshape migration start`](#reshape-migration-start)
39− - [`reshape migration complete`](#reshape-migration-complete)
40− - [`reshape migration abort`](#reshape-migration-abort)
41− - [`reshape schema-query`](#reshape-schema-query)
42− - [Connection options](#connection-options)
39+ - [`reshape migration start`](#reshape-migration-start)
40+ - [`reshape migration complete`](#reshape-migration-complete)
41+ - [`reshape migration abort`](#reshape-migration-abort)
42+ - [`reshape schema-query`](#reshape-schema-query)
43+ - [Connection options](#connection-options)
4344- [License](#license)
4445
45−
4646## How it works
4747
4848Reshape works by creating views that encapsulate the underlying tables, which your application will interact with. During a migration, Reshape will automatically create a new set of views and set up triggers to translate inserts and updates between the old and new schema. This means that every deployment is a three-phase process:
4949
50501. **Start migration** (`reshape migration start`): Sets up views and triggers to ensure both the new and old schema are usable at the same time.
51512. **Roll out application**: Your application can be gradually rolled out without downtime. The existing deployment will continue using the old schema whilst the new deployment uses the new schema.
52−3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
52+3. **Complete migration** (`reshape migration complete`): Removes the old schema and any intermediate data and triggers.
5353
5454If the application deployment fails, you should run `reshape migration abort` which will roll back any changes made by `reshape migration start` without losing data.
5555
@@ -71,7 +71,7 @@ cargo install reshape
7171
7272#### Docker
7373
74−Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
74+Reshape is available as a Docker image on [Docker Hub](https://hub.docker.com/repository/docker/fabianlindfors/reshape).
7575
7676```shell
7777docker run -v $(pwd):/usr/share/app fabianlindfors/reshape reshape migration start
@@ -103,9 +103,14 @@ This is the equivalent of running `CREATE TABLE users (id INTEGER GENERATED ALWA
103103
104104### Preparing your application
105105
106−Reshape relies on your application using a specific schema. When establishing the connection to Postgres in your application, you need to run a query to select the most recent schema. This query can be generated using: `reshape schema-query`.
106+Reshape relies on your application using a specific schema. When establishing the connection to Postgres in your application, you need to run a query to select the most recent schema. The simplest way to do this is to use one of the helper libraries:
107107
108−To pass it along to your application, you can for example use an environment variable in your run script: `RESHAPE_SCHEMA_QUERY=$(reshape schema-query)`. Then in your application:
108+- [Rust](https://github.com/fabianlindfors/reshape-helper)
109+- [Ruby (and Rails)](https://github.com/fabianlindfors/reshape-ruby)
110+- [Python (and Django)](https://github.com/fabianlindfors/reshape-python)
111+- [Go](https://github.com/leourbina/reshape-helper)
112+
113+If your application is not using one of the languages with an available helper library, you can instead generate the query with the command: `reshape schema-query`. To pass it along to your application, you can for example use an environment variable in your run script: `RESHAPE_SCHEMA_QUERY=$(reshape schema-query)`. Then in your application:
109114
110115```python
111116# Example for Python
@@ -113,8 +118,6 @@ reshape_schema_query = os.getenv("RESHAPE_SCHEMA_QUERY")
113118db.execute(reshape_schema_query)
114119```
115120
116−If your application is written in Rust, you might prefer the [Rust helper library](https://github.com/fabianlindfors/reshape-helper) which embeds the query directly in your application with a macro.
117−
118121### Running your migration
119122
120123To create your new `users` table, run:
@@ -164,9 +167,9 @@ Every action has a `type`. The supported types are detailed below.
164167
165168#### Create table
166169
167−The `create_table` action will create a new table with the specified columns, indices and constraints.
170+The `create_table` action will create a new table with the specified columns, indices and constraints. You can optionally provide an `up` option to backfill values from an existing table.
168171
169−*Example: create a `customers` table with a few columns and a primary key*
172+_Example: create a `customers` table with a few columns and a primary key_
170173
171174```toml
172175[[actions]]
@@ -184,13 +187,13 @@ primary_key = ["id"]
184187 type = "TEXT"
185188
186189 # Columns default to nullable
187− nullable = false
190+ nullable = false
188191
189192 # default can be any valid SQL value, in this case a string literal
190− default = "'PLACEHOLDER'"
193+ default = "'PLACEHOLDER'"
191194```
192195
193−*Example: create `users` and `items` tables with a foreign key between them*
196+_Example: create `users` and `items` tables with a foreign key between them_
194197
195198```toml
196199[[actions]]
@@ -223,11 +226,34 @@ primary_key = ["id"]
223226 referenced_columns = ["id"]
224227```
225228
229+_Example: create `profiles` table based on existing `users` table_
230+
231+```toml
232+[[actions]]
233+type = "create_table"
234+name = "profiles"
235+primary_key = ["user_id"]
236+
237+ [[actions.columns]]
238+ name = "user_id"
239+ type = "INTEGER"
240+
241+ [[actions.columns]]
242+ name = "user_email"
243+ type = "TEXT"
244+
245+ # Backfill from `users` table and copy `users.email` to `user_email` column
246+ # This will perform an upsert based on the primary key to avoid duplicate rows
247+ [actions.up]
248+ table = "users"
249+ values = { user_id = "id", user_email = "email" }
250+```
251+
226252#### Rename table
227253
228254The `rename_table` action will change the name of an existing table.
229255
230−*Example: change name of `users` table to `customers`*
256+_Example: change name of `users` table to `customers`_
231257
232258```toml
233259[[actions]]
@@ -240,7 +266,7 @@ new_name = "customers"
240266
241267The `remove_table` action will remove an existing table.
242268
243−*Example: remove `users` table*
269+_Example: remove `users` table_
244270
245271```toml
246272[[actions]]
@@ -252,7 +278,7 @@ table = "users"
252278
253279The `add_foreign_key` action will add a foreign key between two existing tables. The migration will fail if the existing column values aren't valid references.
254280
255−*Example: create foreign key from `items` to `users` table*
281+_Example: create foreign key from `items` to `users` table_
256282
257283```toml
258284[[actions]]
@@ -269,7 +295,7 @@ table = "items"
269295
270296The `remove_foreign_key` action will remove an existing foreign key. The foreign key will only be removed once the migration is completed, which means that your new application must continue to adhere to the foreign key constraint.
271297
272−*Example: remove foreign key `items_user_id_fkey` from `users` table*
298+_Example: remove foreign key `items_user_id_fkey` from `users` table_
273299
274300```toml
275301[[actions]]
@@ -282,9 +308,9 @@ foreign_key = "items_user_id_fkey"
282308
283309#### Add column
284310
285−The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column.
311+The `add_column` action will add a new column to an existing table. You can optionally provide an `up` setting. This should be an SQL expression which will be run for all existing rows to backfill the new column. `up` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)).
286312
287−*Example: add a new column `reference` to table `products`*
313+_Example: add a new column `reference` to table `products`_
288314
289315```toml
290316[[actions]]
@@ -298,7 +324,7 @@ table = "products"
298324 default = "10"
299325```
300326
301−*Example: replace an existing `name` column with two new columns, `first_name` and `last_name`*
327+_Example: replace an existing `name` column with two new columns, `first_name` and `last_name`_
302328
303329```toml
304330[[actions]]
@@ -334,7 +360,7 @@ column = "name"
334360down = "first_name || ' ' || last_name"
335361```
336362
337−*Example: extract nested value from unstructured JSON `data` column to new `name` column*
363+_Example: extract nested value from unstructured JSON `data` column to new `name` column_
338364
339365```toml
340366[[actions]]
@@ -349,6 +375,25 @@ up = "data['path']['to']['value'] #>> '{}'"
349375 type = "TEXT"
350376```
351377
378+_Example: duplicate `email` column from `users` to `profiles` table_
379+
380+```toml
381+# `profiles` has `user_id` column which maps to `users.id`
382+[[actions]]
383+type = "add_column"
384+table = "profiles"
385+
386+ [actions.column]
387+ name = "email"
388+ type = "TEXT"
389+ nullable = false
390+
391+ # When `users` is updated in the old schema, we write the email value to `profiles`
392+ [actions.up]
393+ table = "users"
394+ value = "email"
395+ where = "user_id = id"
396+```
352397
353398#### Alter column
354399
@@ -356,7 +401,7 @@ The `alter_column` action enables many different changes to an existing column,
356401
357402When performing more complex changes than a rename, `up` and `down` should be provided. These should be SQL expressions which determine how to transform between the new and old version of the column. Inside those expressions, you can reference the current column value by the column name.
358403
359−*Example: rename `last_name` column on `users` table to `family_name`*
404+_Example: rename `last_name` column on `users` table to `family_name`_
360405
361406```toml
362407[[actions]]
@@ -368,7 +413,7 @@ column = "last_name"
368413 name = "family_name"
369414```
370415
371−*Example: change the type of `reference` column from `INTEGER` to `TEXT`*
416+_Example: change the type of `reference` column from `INTEGER` to `TEXT`_
372417
373418```toml
374419[[actions]]
@@ -383,7 +428,7 @@ down = "CAST(reference AS INTEGER)" # Converts from text value to integer
383428 type = "TEXT" # Previous type was 'INTEGER'
384429```
385430
386−*Example: increment all values of an `index` column by one*
431+_Example: increment all values of an `index` column by one_
387432
388433```toml
389434[[actions]]
@@ -398,7 +443,7 @@ down = "index - 1" # Decrement to revert for old schema
398443 name = "index"
399444```
400445
401−*Example: make `name` column not nullable*
446+_Example: make `name` column not nullable_
402447
403448```toml
404449[[actions]]
@@ -413,7 +458,7 @@ up = "COALESCE(name, 'N/A')"
413458 nullable = false
414459```
415460
416−*Example: change default value of `created_at` column to current time*
461+_Example: change default value of `created_at` column to current time_
417462
418463```toml
419464[[actions]]
@@ -427,11 +472,11 @@ column = "created_at"
427472
428473#### Remove column
429474
430−The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
475+The `remove_column` action will remove an existing column from a table. You can optionally provide a `down` setting. This should be an SQL expression which will be used to determine values for the old schema when inserting or updating rows using the new schema. `down` may also reference another table to perform cross-table migrations (see ["Complex changes across tables"](#complex-changes-across-tables)) . The `down` setting must be provided when the removed column is `NOT NULL` or doesn't have a default value.
431476
432477Any indices that cover the column will be removed.
433478
434−*Example: remove column `name` from table `users`*
479+_Example: remove column `name` from table `users`_
435480
436481```toml
437482[[actions]]
@@ -443,18 +488,34 @@ column = "name"
443488down = "'N/A'"
444489```
445490
491+_Example: remove `email` column from `users` table and use column from `profiles` table instead_
492+
493+```toml
494+[[actions]]
495+type = "remove_column"
496+table = "users"
497+column = "email"
498+
499+ # Our application will use the `profiles.email` column instead
500+ # For backwards compatibility, we will write back to the removed `email` column whenever `profiles` is changed
501+ [actions.down]
502+ table = "profiles"
503+ value = "profiles.email"
504+ where = "users.id = profiles.user_id"
505+```
506+
446507### Indices
447508
448509#### Add index
449510
450511The `add_index` action will add a new index to an existing table.
451512
452−*Example: create a `users` table with a unique index on the `name` column*
513+_Example: create a `users` table with a unique index on the `name` column_
453514
454515```toml
455516[[actions]]
456517type = "create_table"
457−table = "users"
518+name = "users"
458519primary_key = "id"
459520
460521 [[actions.columns]]
@@ -478,7 +539,7 @@ table = "users"
478539 unique = true
479540```
480541
481−*Example: add GIN index to `data` column on `products` table*
542+_Example: add GIN index to `data` column on `products` table_
482543
483544```toml
484545[[actions]]
@@ -497,7 +558,7 @@ table = "products"
497558
498559The `remove_index` action will remove an existing index. The index won't actually be removed until the migration is completed.
499560
500−*Example: remove the `name_idx` index*
561+_Example: remove the `name_idx` index_
501562
502563```toml
503564[[actions]]
@@ -511,7 +572,7 @@ index = "name_idx"
511572
512573The `create_enum` action will create a new [enum type](https://www.postgresql.org/docs/current/datatype-enum.html) with the specified values.
513574
514−*Example: add a new `mood` enum type with three possible values*
575+_Example: add a new `mood` enum type with three possible values_
515576
516577```toml
517578[[actions]]
@@ -524,7 +585,7 @@ values = ["happy", "ok", "sad"]
524585
525586The `remove_enum` action will remove an existing [enum type](https://www.postgresql.org/docs/current/datatype-enum.html). Make sure all usages of the enum has been removed before running the migration. The enum will only be removed once the migration is completed.
526587
527−*Example: remove the `mood` enum type*
588+_Example: remove the `mood` enum type_
528589
529590```toml
530591[[actions]]
@@ -542,7 +603,7 @@ There are three optional settings available which all accept SQL queries. All qu
542603- `complete`: run when a migration is completed using `reshape migration complete`
543604- `abort`: run when a migration is aborted using `reshape migration abort`
544605
545−*Example: enable PostGIS and pg_stat_statements extensions*
606+_Example: enable PostGIS and pg_stat_statements extensions_
546607
547608```toml
548609[[actions]]
@@ -554,11 +615,97 @@ start = """
554615"""
555616
556617abort = """
557− DROP EXTENSION IF NOT EXISTS postgis;
558− DROP EXTENSION IF NOT EXISTS pg_stat_statements;
618+ DROP EXTENSION IF EXISTS postgis;
619+ DROP EXTENSION IF EXISTS pg_stat_statements;
559620"""
560621```
561622
623+### Complex changes across tables
624+
625+The `up` and `down` options available when creating tables, adding columns and removing columns can also perform more complex changes that span tables.
626+
627+_Example: move `email` column from `users` to `profiles` table_
628+
629+```toml
630+[[actions]]
631+type = "add_column"
632+table = "profiles"
633+
634+ [actions.column]
635+ name = "email"
636+ type = "TEXT"
637+ nullable = false
638+
639+ # When `users` is updated in the old schema, we write the email value to `profiles`
640+ [actions.up]
641+ table = "users"
642+ value = "users.email"
643+ where = "profiles.user_id = users.id"
644+
645+[[actions]]
646+type = "remove_column"
647+table = "users"
648+column = "email"
649+
650+ # When `profiles` is changed in the new schema, we write the email address back to the removed column
651+ [actions.down]
652+ table = "profiles"
653+ value = "profiles.email"
654+ where = "users.id = profiles.user_id"
655+```
656+
657+_Example: turn a 1:N relationship between `users` and `accounts` into N:M and change the format of the associated `role`_
658+
659+```toml
660+# Add `user_account_connections` as a junction table
661+[[actions]]
662+type = "create_table"
663+name = "user_account_connections"
664+primary_key = ["account_id", "user_id"]
665+
666+ [[actions.columns]]
667+ name = "account_id"
668+ type = "INTEGER"
669+
670+ [[actions.columns]]
671+ name = "user_id"
672+ type = "INTEGER"
673+
674+ # `role` is currently stored directly on the `users` table but is part of the relationship
675+ [[actions.columns]]
676+ name = "role"
677+ type = "TEXT"
678+ nullable = false
679+
680+ # Backfill the new table from `users` and uppercase the `role`
681+ [actions.up]
682+ table = "users"
683+ values = { user_id = "id", account_id = "account_id", role = "UPPER(account_role)" }
684+ where = "user_account_connections.user_id = users.id"
685+
686+[[actions]]
687+type = "remove_column"
688+table = "users"
689+column = "account_id"
690+
691+ # When `user_account_connections` is updated, we write the `account_id` back to `users`
692+ [actions.down]
693+ table = "user_account_connections"
694+ value = "user_account_connections.account_id"
695+ where = "users.id = user_account_connections.user_id"
696+
697+[[actions]]
698+type = "remove_column"
699+table = "users"
700+column = "account_role"
701+
702+ # When `user_account_connections` is updated, we write the lowercase role back to `users`
703+ [actions.down]
704+ table = "user_account_connections"
705+ value = "LOWER(user_account_connections.role)"
706+ where = "users.id = user_account_connections.user_id"
707+```
708+
562709## Commands and options
563710
564711### `reshape migration start`
@@ -567,16 +714,16 @@ Starts a new migration, applying all migrations under `migrations/` that haven't
567714
568715#### Options
569716
570−*See also [Connection options](#connection-options)*
717+_See also [Connection options](#connection-options)_
571718
572−| Option | Default | Description |
573−| ------ | ------- | ----------- |
574−| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
575−| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
719+| Option | Default | Description |
720+| ------------------ | ------------- | --------------------------------------------------------------------------------------------------------------- |
721+| `--complete`, `-c` | `false` | Automatically complete migration after applying it. |
722+| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
576723
577724### `reshape migration complete`
578725
579−Completes migrations previously started with `reshape migration complete`.
726+Completes migrations previously started with `reshape migration complete`.
580727
581728#### Options
582729
@@ -584,7 +731,7 @@ See [Connection options](#connection-options)
584731
585732### `reshape migration abort`
586733
587−Aborts any migrations which haven't yet been completed.
734+Aborts any migrations which haven't yet been completed.
588735
589736#### Options
590737
@@ -600,8 +747,8 @@ The query should look something like `SET search_path TO migration_1_initial_mig
600747
601748#### Options
602749
603−| Option | Default | Description |
604−| ------ | ------- | ----------- |
750+| Option | Default | Description |
751+| -------- | ------------- | --------------------------------------------------------------------------------------------------------------- |
605752| `--dirs` | `migrations/` | Directories to search for migration files. Multiple directories can be specified using `--dirs dir1 dir2 dir3`. |
606753
607754### Connection options
@@ -610,14 +757,14 @@ The options below can be used with all commands that communicate with Postgres.
610757
611758All options can also be set using environment variables instead of flags. If a `.env` file exists, then variables will be automatically loaded from there.
612759
613−| Option | Default | Environment variable | Description |
614−| ------ | ------- | -------------------- | ----------- |
615−| `--url` | | `DB_URL` | URL to your Postgres database |
616−| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
617−| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
618−| `--database` | `postgres` | `DB_NAME` | Database name |
619−| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
620−| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
760+| Option | Default | Environment variable | Description |
761+| ------------ | ----------- | -------------------- | ------------------------------------------- |
762+| `--url` | | `DB_URL` | URL to your Postgres database |
763+| `--host` | `localhost` | `DB_HOST` | Hostname to use when connecting to Postgres |
764+| `--port` | `5432` | `DB_PORT` | Port which Postgres is listening on |
765+| `--database` | `postgres` | `DB_NAME` | Database name |
766+| `--username` | `postgres` | `DB_USERNAME` | Postgres username |
767+| `--password` | `postgres` | `DB_PASSWORD` | Postgres password |
621768
622769## License
623770