c4e4ee3c22
https://github.com/xataio/pgroll/pull/365 changed the definition of `previous_version` and introduced a performance regression. Update the function with extra `WHERE` clauses so that the recursive CTE can take better advantage of the table PK index. On a large (~1M record) `pgroll.migrations` table the query analysis for the recursive CTE is: **Before**: ``` +----------------------------------------------------------------------------------------------------------------------------------------------------------+ | QUERY PLAN | |----------------------------------------------------------------------------------------------------------------------------------------------------------| | CTE Scan on ancestors (cost=1819977.60..1819978.62 rows=51 width=214) (actual time=1163.608..6463.980 rows=1 loops=1) | | CTE ancestors | | -> Recursive Union (cost=0.00..1819977.60 rows=51 width=114) (actual time=1163.606..6463.977 rows=1 loops=1) | | -> Seq Scan on migrations (cost=0.00..442329.71 rows=1 width=114) (actual time=1163.604..6385.489 rows=1 loops=1) | | Filter: (name = latest_version('bb_00bo06t68d5ot7i5k5m8um70kg_bb19qc'::name)) | | Rows Removed by Filter: 1193140 | | -> Nested Loop (cost=0.55..137764.74 rows=5 width=114) (actual time=78.485..78.486 rows=0 loops=1) | | -> WorkTable Scan on ancestors a (cost=0.00..0.20 rows=10 width=36) (actual time=0.001..0.002 rows=1 loops=1) | | -> Index Scan using migrations_pkey on migrations m (cost=0.55..13776.44 rows=1 width=110) (actual time=78.459..78.459 rows=0 loops=1) | | Index Cond: (name = a.parent) | | Filter: ((migration_type)::text = 'inferred'::text) | | Rows Removed by Filter: 1 | | Planning Time: 0.764 ms | | JIT: | | Functions: 11 | | Options: Inlining true, Optimization true, Expressions true, Deforming true | | Timing: Generation 2.222 ms, Inlining 29.422 ms, Optimization 51.028 ms, Emission 24.552 ms, Total 107.225 ms | | Execution Time: 6466.347 ms | +----------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` **After**: ``` +----------------------------------------------------------------------------------------------------------------------------------------------------+ | QUERY PLAN | |----------------------------------------------------------------------------------------------------------------------------------------------------| | CTE Scan on ancestors (cost=987.99..988.21 rows=11 width=214) (actual time=0.277..0.379 rows=1 loops=1) | | CTE ancestors | | -> Recursive Union (cost=0.55..987.99 rows=11 width=114) (actual time=0.275..0.377 rows=1 loops=1) | | -> Index Scan using history_is_linear on migrations (cost=0.55..128.36 rows=1 width=114) (actual time=0.274..0.354 rows=1 loops=1) | | Index Cond: (schema = 'bb_00bo06t68d5ot7i5k5m8um70kg_bb19qc'::name) | | Filter: (name = latest_version('bb_00bo06t68d5ot7i5k5m8um70kg_bb19qc'::name)) | | Rows Removed by Filter: 5 | | -> Nested Loop (cost=0.55..85.95 rows=1 width=114) (actual time=0.020..0.020 rows=0 loops=1) | | -> WorkTable Scan on ancestors a (cost=0.00..0.20 rows=10 width=100) (actual time=0.001..0.001 rows=1 loops=1) | | -> Index Scan using migrations_pkey on migrations m (cost=0.55..8.57 rows=1 width=110) (actual time=0.017..0.017 rows=0 loops=1) | | Index Cond: ((schema = a.schema) AND (name = a.parent)) | | Filter: ((migration_type)::text = 'inferred'::text) | | Rows Removed by Filter: 1 | | Planning Time: 0.744 ms | | Execution Time: 0.459 ms | +----------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ie the recursive portion of the table is able to take advantage of the table's PK (defined on `(schema, name)`). |
||
|---|---|---|
| .github | ||
| .vscode | ||
| brand-kit | ||
| cmd | ||
| docs | ||
| examples | ||
| pkg | ||
| .gitignore | ||
| .golangci.yml | ||
| .goreleaser.yaml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| main.go | ||
| Makefile | ||
| README.md | ||
| schema.json | ||
pgroll - Zero-downtime, reversible, schema migrations for Postgres
pgroll is an open source command-line tool that offers safe and reversible schema migrations for PostgreSQL by serving multiple schema versions simultaneously. It takes care of the complex migration operations to ensure that client applications continue working while the database schema is being updated. This includes ensuring changes are applied without locking the database, and that both old and new schema versions work simultaneously (even when breaking changes are being made!). This removes risks related to schema migrations, and greatly simplifies client application rollout, also allowing for instant rollbacks.
See the introductory blog post for more about the problems solved by pgroll.
Features
- Zero-downtime migrations (no database locking, no breaking changes).
- Keep old and new schema versions working simultaneously.
- Automatic columns backfilling when needed.
- Instant rollback in case of issues during migration.
- Works against existing schemas, no need to start from scratch.
- Works with Postgres 14.0 or later.
- Works with any Postgres service (including RDS and Aurora).
- Written in Go, cross-platform single binary with no external dependencies.
How pgroll works
pgroll works by creating virtual schemas by using views on top of the physical tables. This allows for performing all the necessary changes needed for a migration without affecting the existing clients.
pgroll follows a expand/contract workflow. On migration start, it will perform all the additive changes (create tables, add columns, etc) in the physical schema, without breaking it.
When a breaking change is required on a column, it will create a new column in the physical schema, and backfill it from the old column. Also, configure triggers to make sure all writes to the old/new column get propagated to its counterpart during the whole active migration period. The new column will be then exposed in the new version of the schema.
Once the start phase is complete, the new schema version is ready, mapping all the views to the proper tables & columns. Client applications can then access the new schema version, while the old one is still available. This is the moment to start rolling out the new version of the client application.
When no more client applications are using the old schema version, the migration can be completed. This will remove the old schema, and the new one will be the only one available. No longer needed tables & columns will be removed (no client is using this at this point), and the new ones will be renamed to their final names. Client applications still work during this phase, as the views are still mapping to the proper tables & columns.
Table of Contents
Installation
Binaries
Binaries are available for Linux, macOS & Windows, check our Releases.
From source
To install pgroll from the source, run the following command:
go install github.com/xataio/pgroll@latest
Note: requires Go 1.21 or later.
From package manager - Homebrew
To install pgroll with homebrew, run the following command:
# macOS or Linux
brew tap xataio/pgroll
brew install pgroll
Usage
Follow these steps to perform your first schema migration using pgroll:
Prepare the database
pgroll needs to store some internal state in the database. A table is created to track the current schema version and store version history. To prepare the database, run the following command:
pgroll init --postgres-url postgres://user:password@host:port/dbname
Start a migration
Create a migration file. You can check the examples folder for some examples. For instance, use this migration file to create a new customers table:
initial_migration.json
{
"name": "initial_migration",
"operations": [
{
"create_table": {
"name": "customers",
"columns": [
{
"name": "id",
"type": "integer",
"pk": true
},
{
"name": "name",
"type": "varchar(255)",
"unique": true
},
{
"name": "bio",
"type": "text",
"nullable": true
}
]
}
}
]
}
Then run the following command to start the migration:
pgroll --postgres-url postgres://user:password@host:port/dbname start initial_migration.json
This will create a new schema version in the database, and apply the migration operations (create a table). After this command finishes, both the old version of the schema (with no customers table) and the new one (with the customers table) will be accessible simultaneously.
Configure client applications
After starting a migration, client applications can start using the new schema version. In order to do so, they need to be configured to access it. This can be done by setting the search_path to the new schema version name (provided by pgroll start output), for instance:
SET search_path TO 'public_initial_migration';
Complete the migration
Once there are no more client applications using the old schema version, the migration can be completed. This will remove the old schema. To complete the migration, run the following command:
pgroll --postgres-url postgres://user:password@host:port/dbname complete
Rolling back a migration
At any point during a migration, it can be rolled back to the previous version. This will remove the new schema and leave the old one as it was before the migration started. To rollback a migration, run the following command:
pgroll --postgres-url postgres://user:password@host:port/dbname rollback
Documentation
For more advanced usage, a tutorial, and detailed options refer to the full Documentation.
Contributing
We welcome contributions from the community! If you'd like to contribute to pgroll, please follow these guidelines:
- Create an issue for any questions, bug reports, or feature requests.
- Check the documentation and existing issues before opening a new issue.
Contributing Code
- Fork the repository.
- Create a new branch for your feature or bug fix.
- Make your changes and write tests if applicable.
- Ensure your code passes linting and tests.
- Submit a pull request.
For this project, we pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
References
This is a list of projects and articles that helped as inspiration, or otherwise are similar to pgroll:
- Reshape by Fabian Lindfors
- PgHaMigrations
- PostgreSQL at Scale: Database Schema Changes Without Downtime
- Zero downtime schema migrations in highly available databases
- Expand and contract pattern
License
This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
Support
If you have any questions, encounter issues, or need assistance, open an issue in this repository our join our Discord, and our community will be happy to help.
Made with ❤️ by Xata 🦋