6.8 KiB
Reshape
Reshape is an easy-to-use, zero-downtime schema migration tool for Postgres. It automatically handles complex migrations that would normally require downtime or manual multi-step changes. During a migration, Reshape ensures both the old and new schema are available at the same time, allowing you to gradually roll out your application.
Note: Reshape is experimental and should not be used in production. It can (and probably will) destroy your data and break your application.
Getting started
Installation
On macOS:
brew install reshape
On Debian:
apt-get install reshape
Creating your first migration
Each migration should be stored as a separate file under migrations/
. The files can be in either JSON or TOML format. The name of the file will become the name of your migration and they will be sorted by file name. We recommend prefixing every migration with an incrementing number.
Let's create a simple migration to set up a new table users
with two fields, id
and name
. We'll create a file called migration/1_create_users_table.toml
:
[[actions]]
type = "create_table"
table = "users"
[[actions.columns]]
name = "id"
type = "SERIAL"
[[actions.columns]]
name = "name"
type = "TEXT"
This is the equivalent of running CREATE TABLE users (id SERIAL, name TEXT)
.
Preparing your application
Reshape relies on your application using a specific schema. When establishing the connection to Postgres in your application, you need to run a query to select the most recent schema. This query can be generated using: reshape generate-schema-query
.
To pass it along to your application, you could use an environment variable in your build script: RESHAPE_SCHEMA_QUERY=$(reshape generate-schema-query)
. Then in your application:
# Example for Python
reshape_schema_query = os.getenv("RESHAPE_SCHEMA_QUERY")
db.execute(reshape_schema_query)
Running your migration
To create your new users
table, run:
reshape migrate
As this is the first migration, Reshape will automatically complete it. For subsequent migrations, you will need to first run reshape migrate
, roll out your application and then complete the migration using reshape complete
.
Writing migrations
Basics
Every migration consists of one or more actions. The actions will be run sequentially. Here's an example of a migration with two actions to create two tables, customers
and products
:
[[actions]]
type = "create_table"
table = "customers"
[[actions.columns]]
name = "id"
type = "SERIAL"
[[actions]]
type = "create_table"
table = "products"
[[actions.columns]]
name = "sku"
type = "TEXT"
Every action has a type
. The supported types are detailed below.
Create table
The create_table
action will create a new table with the specified columns and indices.
Example: creating a customers
table with a few columns and a primary key
[[actions]]
type = "create_table"
table = "customers"
primary_key = "id"
[[actions.columns]]
name = "id"
type = "SERIAL"
[[actions.columns]]
name = "name"
type = "SERIAL"
# Columns default to nullable
nullable = false
# default can be any valid SQL value, in this case a string literal
default = "'PLACEHOLDER'"
Add column
The add_column
action will add a new column to an existing table. You can optionally provide an up
setting. This should be an SQL expression which will be run for all existing rows to backfill the new column.
Example: add a new column reference
to table products
[[actions]]
type = "add_column"
table = "products"
[actions.column]
name = "reference"
type = "INTEGER"
nullable = false
default = "10"
Example: replace an existing name
column with two new columns, first_name
and last_name
[[actions]]
type = "add_column"
table = "users"
# Extract the first name from the existing name column
up = "(STRING_TO_ARRAY(name, ' '))[1]"
[actions.column]
name = "first_name"
type = "TEXT"
[[actions]]
type = "add_column"
table = "users"
# Extract the last name from the existing name column
up = "(STRING_TO_ARRAY(name, ' '))[2]"
[actions.column]
name = "last_name"
type = "TEXT"
[[actions]]
type = "remove_column"
table = "users"
column = "name"
# Reconstruct name column by concatenating first and last name
down = "first_name || ' ' || last_name"
Alter column
The alter_column
action enables many different changes to an existing column, for example renaming, changing type and changing existing values.
When performing more complex changes than a rename, up
and down
must be provided. These should be set to SQL expressions which determine how to transform between the new and old version of the column. Inside those expressions, you can reference the current column value by the column name.
Example: rename last_name
column on users
table to family_name
[[actions]]
type = "alter_column"
table = "users"
column = "last_name"
[actions.changes]
name = "family_name"
Example: change the type of reference
column from INTEGER
to TEXT
[[actions]]
type = "alter_column"
table = "users"
column = "reference"
up = "CAST(reference AS TEXT)" # Converts from integer value to text
down = "CAST(reference AS INTEGER)" # Converts from text value to integer
[actions.changes]
type = "TEXT" # Previous type was 'INTEGER'
Example: increment all values of a index
column by one
[[actions]]
type = "alter_column"
table = "users"
up = "index + 1" # Increment for new schema
down = "index - 1" # Decrement to revert for old schema
[actions.changes]
name = "index"
How it works
Reshape works by creating views that encapsulate the underlying tables, which your application will interact with. During a migration, Reshape will automatically create a new set of views and set up triggers to translate inserts and updates between the old and new schema. This means that every migration is a two phase process:
- Start migration (
reshape migrate
): Create new views to ensure both the new and old schema are usable at the same time.- After phase one is complete, you can start the roll out of your application. Once the roll out is complete, the second phase can be run.
- Complete migration (
reshape complete
): Removes the old schema and any intermediate data.