This change updates the `sql` operation to: * Allow for a new `onComplete` flag, that will make it run on the Complete phase, rather than doing it during Start (default behavior). * Allows for `sql` operations next to others in the same migration. We added this limitation to ensure this operation doesn't affect others, especially around schema state management. Having `sql` next to other operations has proven convenient sometimes, by adding `onComplete` flag, we can allow for these migrations to run and rely on views recreation based on the final state. --------- Co-authored-by: Andrew Farries <andyrb@gmail.com> |
||
---|---|---|
.github | ||
.vscode | ||
brand-kit | ||
cmd | ||
docs | ||
examples | ||
pkg | ||
.gitignore | ||
.golangci.yml | ||
.goreleaser.yaml | ||
docker-compose.yml | ||
Dockerfile | ||
go.mod | ||
go.sum | ||
LICENSE | ||
main.go | ||
Makefile | ||
README.md | ||
schema.json |
pgroll - Zero-downtime, reversible, schema migrations for Postgres
pgroll
is an open source command-line tool that offers safe and reversible schema migrations for PostgreSQL by serving multiple schema versions simultaneously. It takes care of the complex migration operations to ensure that client applications continue working while the database schema is being updated. This includes ensuring changes are applied without locking the database, and that both old and new schema versions work simultaneously (even when breaking changes are being made!). This removes risks related to schema migrations, and greatly simplifies client application rollout, also allowing for instant rollbacks.
See the introductory blog post for more about the problems solved by pgroll
.
Features
- Zero-downtime migrations (no database locking, no breaking changes).
- Keep old and new schema versions working simultaneously.
- Automatic columns backfilling when needed.
- Instant rollback in case of issues during migration.
- Works against existing schemas, no need to start from scratch.
- Works with Postgres 14.0 or later.
- Works with any Postgres service (including RDS and Aurora).
- Written in Go, cross-platform single binary with no external dependencies.
How pgroll works
pgroll
works by creating virtual schemas by using views on top of the physical tables. This allows for performing all the necessary changes needed for a migration without affecting the existing clients.
pgroll
follows a expand/contract workflow. On migration start, it will perform all the additive changes (create tables, add columns, etc) in the physical schema, without breaking it.
When a breaking change is required on a column, it will create a new column in the physical schema, and backfill it from the old column. Also, configure triggers to make sure all writes to the old/new column get propagated to its counterpart during the whole active migration period. The new column will be then exposed in the new version of the schema.
Once the start phase is complete, the new schema version is ready, mapping all the views to the proper tables & columns. Client applications can then access the new schema version, while the old one is still available. This is the moment to start rolling out the new version of the client application.
When no more client applications are using the old schema version, the migration can be completed. This will remove the old schema, and the new one will be the only one available. No longer needed tables & columns will be removed (no client is using this at this point), and the new ones will be renamed to their final names. Client applications still work during this phase, as the views are still mapping to the proper tables & columns.
Table of Contents
Installation
Binaries
Binaries are available for Linux, macOS & Windows, check our Releases.
From source
To install pgroll
from the source, run the following command:
go install github.com/xataio/pgroll@latest
Note: requires Go 1.21 or later.
From package manager - Homebrew
To install pgroll
with homebrew, run the following command:
# macOS or Linux
brew tap xataio/pgroll
brew install pgroll
Usage
Follow these steps to perform your first schema migration using pgroll
:
Prepare the database
pgroll
needs to store some internal state in the database. A table is created to track the current schema version and store version history. To prepare the database, run the following command:
pgroll init --postgres-url postgres://user:password@host:port/dbname
Start a migration
Create a migration file. You can check the examples folder for some examples. For instance, use this migration file to create a new customers
table:
initial_migration.json
{
"name": "initial_migration",
"operations": [
{
"create_table": {
"name": "customers",
"columns": [
{
"name": "id",
"type": "integer",
"pk": true
},
{
"name": "name",
"type": "varchar(255)",
"unique": true
},
{
"name": "bio",
"type": "text",
"nullable": true
}
]
}
}
]
}
Then run the following command to start the migration:
pgroll --postgres-url postgres://user:password@host:port/dbname start initial_migration.json
This will create a new schema version in the database, and apply the migration operations (create a table). After this command finishes, both the old version of the schema (with no customers table) and the new one (with the customers table) will be accessible simultaneously.
Configure client applications
After starting a migration, client applications can start using the new schema version. In order to do so, they need to be configured to access it. This can be done by setting the search_path
to the new schema version name (provided by pgroll start
output), for instance:
SET search_path TO 'public_initial_migration';
Complete the migration
Once there are no more client applications using the old schema version, the migration can be completed. This will remove the old schema. To complete the migration, run the following command:
pgroll --postgres-url postgres://user:password@host:port/dbname complete
Rolling back a migration
At any point during a migration, it can be rolled back to the previous version. This will remove the new schema and leave the old one as it was before the migration started. To rollback a migration, run the following command:
pgroll --postgres-url postgres://user:password@host:port/dbname rollback
Documentation
For more advanced usage, a tutorial, and detailed options refer to the full Documentation.
Contributing
We welcome contributions from the community! If you'd like to contribute to pgroll
, please follow these guidelines:
- Create an issue for any questions, bug reports, or feature requests.
- Check the documentation and existing issues before opening a new issue.
Contributing Code
- Fork the repository.
- Create a new branch for your feature or bug fix.
- Make your changes and write tests if applicable.
- Ensure your code passes linting and tests.
- Submit a pull request.
For this project, we pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
References
This is a list of projects and articles that helped as inspiration, or otherwise are similar to pgroll
:
- Reshape by Fabian Lindfors
- PgHaMigrations
- PostgreSQL at Scale: Database Schema Changes Without Downtime
- Zero downtime schema migrations in highly available databases
- Expand and contract pattern
License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Support
If you have any questions, encounter issues, or need assistance, open an issue in this repository our join our Discord, and our community will be happy to help.
Made with ❤️ by Xata 🦋