The state query changes from #358 are causing performance issues in an
upstream integration of `pgroll`.
Revert the PR until the cause of the regression is found and fixed.
This reverts commit f994a42192.
Export `OpAlterColumn.IsRenameOnly` method and add a test for it.
Applications using `pgroll` as a module may need to test if an alter
column operation is only a column rename operation without having to
duplicate the logic of checking the fields of the `OpAlterColumn`
struct.
Fixes https://github.com/xataio/pgroll/issues/355
Postgres stores index names with uppercase characters in the `pg_index`
catalog using the quoted version of the name. For example:
```
"idx_USERS_name"
```
whereas a lowercase index name would be stored as:
```
idx_users_name
```
This is different to how other object types are stored in their
respective catalogs. For example, table names are stored in
the`pg_class` catalog without quotes, regardless of whether they contain
uppercase characters.
This makes it necessary to strip quotes from index names when retrieving
them from the `pg_index` catalog when building the internal schema
representation.
Ensure that 'alter column' operations that rename a column and also
specify `down` SQL (such as those that alter some other column attribute
at the time of the rename) must use the new name of the column in the
`down` SQL.
Without this change, the `down` SQL would require the use of the old
column name.
Fixes#350
When a column is duplicated with a new type, check constraints and
defaults defined on the column *may* be incompatible with the new type.
In this case column duplication should not fail, but rather the
incompatible constraints and defaults should be ignored and not be
duplicated to the new column.
This PR changes column duplication to ignore errors on `DEFAULT` and
`CHECK` constraint duplication that look as though they are caused by a
change of column type.
Fixes https://github.com/xataio/pgroll/issues/348
Retry statements and transactions that fail due to `lock_timeout`
errors.
DDL operations and backfills are run in a session in which `SET
lock_timout TO xms'` has been set (`x` defaults to `500` but can be
specified with the `--lock-timeout` parameter). This ensures that a long
running query can't cause other queries to queue up behind a DDL
operation as it waits to acquire its lock.
The current behaviour if a DDL operation or backfill batch times out
when requesting a lock is to fail, forcing the user to retry the
migration operation (start, rollback, or complete).
This PR retries individual statements (like the DDL operations run by
migration operations) and transactions (used by backfills) if they fail
due to a `lock_timeout` error. The retry uses an exponential backoff
with jitter.
Fixes#171
Allow 'alter column' operations to set the comment on a column:
```json
{
"name": "02_change_comment",
"operations": [
{
"alter_column": {
"table": "events",
"column": "name",
"comment": "The full name of the event"
}
}
]
}
```
This is a versioned migration so the column to which the comment is
applied is duplicated and backfilled according to the `up` and `down`
SQL supplied with the 'alter column' operation. `up` and `down` default
to a simple copy of the field between new and old columns.
The intention is that this can be combined with a 'change type'
sub-operation if there is some column metadata that should be updated as
part of the type change:
```json
{
"name": "35_alter_column_multiple",
"operations": [
{
"alter_column": {
"table": "events",
"column": "name",
"name": "event_name",
"type": "text",
"comment": "{type: some-metadata}",
"up": "name",
"down": "name"
}
}
]
}
```
Fixes#328
Allow 'alter column' operations to set the default value on a column:
```json
{
"name": "02_change_default",
"operations": [
{
"alter_column": {
"table": "events",
"column": "name",
"default": "'new default value'"
}
}
]
}
```
This is a versioned migration so the column to which the default is
applied is duplicated and backfilled according to the `up` and `down`
SQL supplied with the 'alter column' operation. `up` and `down` default
to a simple copy of the field between new and old columns.
Fixes#327
Allow rename-only 'alter column' operations on unbackfillable columns.
Rename operations don't require backfills so there is no reason to
impose such a restriction.
Fixes#340
Allow 'alter column' operations to include multiple sub-operations. This
means migrations like this one are now possible:
```json
{
"name": "35_alter_column_multiple",
"operations": [
{
"alter_column": {
"table": "events",
"column": "name",
"name": "event_name",
"type": "text",
"nullable": false,
"unique": {
"name": "events_event_name_unique"
},
"check": {
"name": "event_name_length",
"constraint": "length(name) > 3"
},
"up": "(SELECT CASE WHEN name IS NULL THEN 'placeholder' ELSE name END)",
"down": "name"
}
}
]
}
```
This 'alter column' operation:
* Renames a column
* Changes its type
* Sets it `NOT NULL`
* Adds a unique constraint
* Adds a check constraint
Previously, this would have required 5 different operations.
Builds on https://github.com/xataio/pgroll/pull/337. Fixes
https://github.com/xataio/pgroll/issues/336
Restrict the search path of the connection used by the state package to
only the state schema (the schema that contains the `migrations` table).
This ensures that any column types that might reside in other schema are
always qualified with the schema name in the internal schema
representation. Without the restriction of the search path, these type
names would be unqualified if they lived in a schema that was on the
search path used by the state connection.
The representation (qualified/unqualified) of type names in the internal
schema is ultimately due to how the
[format-type](https://www.postgresql.org/docs/9.5/functions-info.html)
function behaves; types in the `search_path` of the caller are
unqualified, otherwise they are qualified.
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.20.0 to
0.23.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="c48da13158"><code>c48da13</code></a>
http2: fix TestServerContinuationFlood flakes</li>
<li><a
href="762b58d1cf"><code>762b58d</code></a>
http2: fix tipos in comment</li>
<li><a
href="ba872109ef"><code>ba87210</code></a>
http2: close connections when receiving too many headers</li>
<li><a
href="ebc8168ac8"><code>ebc8168</code></a>
all: fix some typos</li>
<li><a
href="3678185f8a"><code>3678185</code></a>
http2: make TestCanonicalHeaderCacheGrowth faster</li>
<li><a
href="448c44f928"><code>448c44f</code></a>
http2: remove clientTester</li>
<li><a
href="c7877ac421"><code>c7877ac</code></a>
http2: convert the remaining clientTester tests to testClientConn</li>
<li><a
href="d8870b0bf2"><code>d8870b0</code></a>
http2: use synthetic time in TestIdleConnTimeout</li>
<li><a
href="d73acffdc9"><code>d73acff</code></a>
http2: only set up deadline when Server.IdleTimeout is positive</li>
<li><a
href="89f602b7bb"><code>89f602b</code></a>
http2: validate client/outgoing trailers</li>
<li>Additional commits viewable in <a
href="https://github.com/golang/net/compare/v0.20.0...v0.23.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts page](https://github.com/xataio/pgroll/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Remove duplication between 'alter column' sub-operations. Pull:
* column duplication and trigger creation on migration start
* column rename and trigger removal on complete
* trigger and column drop on rollback
up to the parent 'alter column' operation. This removes a lot of
duplicated code from the sub-operations and will make it easier to
support multiple sub-operations in one 'alter column' operation.
Part of #336
Extend `MigrationHooks` with two new hooks:
* `BeforeCompleteDDL` - run before any DDL operations on migration
completion.
* `AfterCompleteDDL` - guaranteed to run after DDL operations on
migration complete are finished.
These hooks can be used, for example, to temporarily switch off schema
replication until all DDL has completed.
Use the SQL transformer added in #329 to rewrite or reject column
`DEFAULT` values.
Column `DEFAULT` values are user-supplied SQL expressions that may need
to be restricted or rewritten in some environments.
Improve the existing tests around SQL transformation by adding a
`MockSQLTransformer` type and using it in the existing SQL
transformation tests.
Also add several new testcases for transformation of raw SQL operations
to ensure that when transformation fails the operation fails.
---------
Co-authored-by: Alexis Rico <me@sferadev.com>
Revert #315 as there is no longer a demonstrated need for this option.
The `WithSQLTransformer` option was added (#329) and can be used to
transform raw SQL operations in #330, removing the need for the
`WithRawSQLURL` option.
Use the SQL transformer to transform the `up` and `down` fields of a raw
SQL migration.
Builds on https://github.com/xataio/pgroll/pull/329 which added a
`SQLTransformer` option to rewrite user-supplied SQL.
Add a new `WithSQLTransformer` option to rewrite the user-defined SQL
used to define `up` and `down` triggers.
The intention is that the transformer be used to sanitize user-input
SQL.
Transformers implement the following interface:
```go
type SQLTransformer interface {
Transform(sql string) (string, error)
}
```
and are used by the `createTrigger` function to rewrite the `up` or
`down` SQL before using it in the trigger function definition.
Later PRs will use the same transformer to rewrite the `up` and `down`
values used in raw SQL migrations and column `DEFAULT` expressions.
Improve consistency between operation types by updating operation types
to ensure that all operations that use `up` or `down` SQL default these
fields to `""`.
There is no distinction between an empty string and `nil` for these
fields.
Bumps google.golang.org/protobuf from 1.31.0 to 1.33.0.
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts page](https://github.com/xataio/pgroll/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This setting comes handy when you need to use a different connection
string for raw SQL operations, as they may require some more security
checks.
---------
Co-authored-by: Andrew Farries <andyrb@gmail.com>
Move the validation of the examples' JSON schema from a step in the
`example` job into its own job.
In its previous location the examples were having their schema validated
in each job in the matrix (pg version x public/non_public schema),
making each job in the matrix take ~15s longer.
---------
Co-authored-by: Carlos Pérez-Aradros Herce <carlos@xata.io>
Ensure that migrations that fail during backfill are rolled back.
Add testcases to ensure rollback occurs when an error occurs:
* during the DDL phase of migration start
* during the backfill phase of migration start
Fixes#316
Remove the `BeforeBackfill` hook.
The `BeforeBackfill` hook was added along with the `BeforeStartDDL` and
`AfterStartDDL` hooks in #290, but is of limited use as a hook that runs
only after successful execution of start phase DDL. Prefer a simpler API
with fewer hooks until there is a demonstrated need for it.
Add the `ON DELETE` setting of a foreign key to the information stored
about the key in the internal schema representation.
The schema representation for a foreign key now looks like:
```json
{
"some_table": {
...
"foreignKeys": {
"fk_users_id": {
"name": "fk_users_id",
"columns": [
"user_id"
],
"onDelete": "NO ACTION",
"referencedTable": "users",
"referencedColumns": [
"id"
]
}
}
}
}
```
Fixes https://github.com/xataio/pgroll/issues/309
#297 added the ability to specify the `ON DELETE` behavior of a foreign
key when altering an existing column.
Make it so that the `ON DELETE` property of the the FK is respected for
FKs on new columns too (ie those created by add column and create table
operations).
Fixes#306
Ensure that only one `inferred` migration is created in the
`pgroll.migrations` table when a column is dropped outside of a
migration.
From the Postgres
[docs](https://www.postgresql.org/docs/current/event-trigger-definition.html):
> The sql_drop event occurs just before the ddl_command_end event
trigger for any operation that drops database objects
This means that when the `raw_migration` function is run in response to
`sql_drop` and `ddl_command_end`, duplicate entries will be created in
`pgroll.migrations`; once as the function is run for `sql_drop` and
again when it's run for `ddl_command_end`.
Change the definition of the `pg_roll_handle_drop` event trigger to only
run on those kinds of drops that won't result in duplicates when the
`pg_roll_handle_ddl` trigger runs for the same change. `DROP TABLE` and
`DROP VIEW` won't result in duplicate migrations because their schema
can't be inferred by the `ddl_command_event` trigger because the object
has already been dropped when the trigger runs.
Update the inferred migration tests with two new testcases covering
dropping tables and columns.
Fixes#304
* Add an extra step to the validation for the `rename_constraint`
operation to ensure that the referenced table exists.
* Add validation tests for the operation.
Fixes#302
Add support for setting the `ON DELETE` behaviour of a foreign key
constraint.
An example migration that uses the behaviour is:
```json
{
"name": "21_add_foreign_key_constraint",
"operations": [
{
"alter_column": {
"table": "posts",
"column": "user_id",
"references": {
"name": "fk_users_id",
"table": "users",
"column": "id",
"on_delete": "CASCADE"
},
"up": "(SELECT CASE WHEN EXISTS (SELECT 1 FROM users WHERE users.id = user_id) THEN user_id ELSE NULL END)",
"down": "user_id"
}
}
]
}
```
The valid options for `on_delete` are `CASCADE`, `SET NULL`, `RESTRICT`,
or `NO ACTION`. If the field is omitted, the default is `NO ACTION`,
Fixes#221
Add a test to ensure that new columns can't be used as the identity for
a backfill, even if they satisfy the conditions for an identity column.
A newly added `NOT NULL` and `UNIQUE` column could in theory be used as
the identity column for a backfill. However, it shouldn't be used as the
identity column fora backfill because it's a newly added column whose
temporary column will be full of NULLs for any existing rows in the
table.
Currently the column won't be selected as an identity for backfills
because the nullability and uniqueness for new column are not populated
when adding it to the virtual schema in the add_column operation:
c08ef7065c/pkg/migrations/op_add_column.go (L61-L63)
Following on from the discussion here:
https://github.com/xataio/pgroll/pull/289#discussion_r1505832826
Add two new migration options:
* `WithSettingsOnMigrationStart`: defines a map of Postgres
setting/value pairs to be set for the duration of the DDL phase of
migration start. Settings will be restored to their previous values once
the DDL phase is complete.
* `WithKickstartReplication`: defines an option that when set will make
a no-op schema change in between completing the DDL operations for
migration start and performing backfills. This can be used to ensure
that schema replication is up-to-date before starting backfills.
Neither of these options are exposed via the CLI; they are intended for
use by `pgroll` integrators, ie modules using `pgroll` as a Go
dependency.
Separate the DDL operations required during migration start from the DML
operations (backfills). Complete all DDL steps before starting DML.
Each `Start` operation returns the name of the table that requires
backfill, if any. After all operations have started, backfills are run
on each table that requires one.
Separating DDL and DML during migration start like this will allow for
setting Postgres options that should apply only for the DDL phase of
migration start (as in #290).
UNIQUE NOT NULL columns should also work in order to perform backfills.
This change relaxes the check on backfill requirements to use those
columns if a primary key is not available.
Validation will still error out if no suitable column is found
Note: this also fixes the `unique` retrieval from schema, where we were
failing to understand composed unique indices, resulting in columns
flagged as unique where they weren't really unique.
Primary keys are only required by pgroll when backfilling tables, relax
checks in the `add_column` op so we only require a present PK if
backfill will be needed (up function defined).
Bump the `golangci-lint` action to `v4` and stop using the cache from
`setup-go` as `golangci-lint` manages its own cache.
This removes all warnings from the summary page from workflow runs.
