The intention was to make this two cases, using a top-level YAML list.
The result was one test with duplicate keys (effectively only running
the second test). This is an error that's now flagged by newer ruamel.
Both tests needed to be "corrected" to pass and need review.
Tested on python 3.5 and 3.7
We make light use of pyenv to set an appropriate python version if
installed. We could easily install a correct version too if we wanted
but that seemed invasive.
The newer ruamel was an annoying upgrade but also offers some
improvements that exposed some test suite issues (fixed later).
We add a new pytest flag `--accept` that will automatically write back
yaml files with updated responses. This makes it much easier and less
error-prone to update test cases when we expect output to change, or
when authoring new tests.
Second we make sure to test that we actually preserve the order of the
selection set when returning results. This is a "SHOULD" part of the
spec but seems pretty important and something that users will rely on.
To support both of the above we use ruamel.yaml which preserves a
certain amount of formatting and comments (so that --accept can work in
a failry ergonomic way), as well as ordering (so that when we write yaml
the order of keys has meaning that's preserved during parsing).
Use ruamel.yaml everywhere for consistency (since both libraries have
different quirks).
Quirks of ruamel.yaml:
- trailing whitespace in multiline strings in yaml files isn't written
back out as we'd like: https://bitbucket.org/ruamel/yaml/issues/47/multiline-strings-being-changed-if-they
- formatting is only sort of preserved; ruamel e.g. normalizes
indentation. Normally the diff is pretty clean though, and you can
always just check in portions of your test file after --accept
fixup
### Description
This PR introduces three new features:
- Support for a new migrations folder structure.
- Add `squash` command in preview.
- ~List of migrations on the Console and ability to squash them from console.~
#### New migrations folder structure
Starting with this commit, Hasura CLI supports a new directory structure for migrations folder and defaults to that for all new migrations created.
Each migration will get a new directory with the name format `timestamp_name` and inside the directory, there will be four files:
```bash
└── migrations
├── 1572237730898_squashed
│ ├── up.sql
│ ├── up.yaml
│ ├── down.yaml
│ └── down.sql
```
Existing files old migration format `timestamp_name.up|down.yaml|sql` will continue to work alongside new migration files.
#### Squash command
Lots of users have expressed their interest in squashing migrations (see #2724 and #2254) and some even built [their own tools](https://github.com/domasx2/hasura-squasher) to do squash. In this PR, we take a systematic approach to squash migrations.
A new command called `migrate squash` is introduced. Note that this command is in **PREVIEW** and the correctness of squashed migration is not guaranteed (especially for down migrations). From our tests, **it works for most use cases**, but we have found some issues with squashing all the down migrations, partly because the console doesn't generate down migrations for all actions.
Hence, until we add an extensive test suite for squashing, we'll keep the command in preview. We recommend you to confirm the correctness yourself by diffing the SQL and Metadata before and after applying the squashed migrations (we're also thinking about embedding some checks into the command itself).
```bash
$ hasura migrate squash --help
(PREVIEW) Squash multiple migrations leading upto the latest one into a single migration file
Usage:
hasura migrate squash [flags]
Examples:
# NOTE: This command is in PREVIEW, correctness is not guaranteed and the usage may change.
# squash all migrations from version 1572238297262 to the latest one:
hasura migrate squash --from 1572238297262
Flags:
--from uint start squashing form this version
--name string name for the new squashed migration (default "squashed")
--delete-source delete the source files after squashing without any confirmation
```
### Affected components
<!-- Remove non-affected components from the list -->
- CLI
### Related Issues
<!-- Please make sure you have an issue associated with this Pull Request -->
<!-- And then add `(close #<issue-no>)` to the pull request title -->
<!-- Add the issue number below (e.g. #234) -->
Close#2724, Close#2254,
### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->
For the squash command, a state machine is implemented to track changes to Hasura metadata. After applying each action on the metadata state, a list of incremental changes is created.
### Steps to test and verify
1. Open console via cli and create some migrations.
2. Run `hasura migrate squash --from <version>`
### Limitations, known bugs & workarounds
<!-- Limitations of the PR, known bugs and suggested workarounds -->
<!-- Feel free to delete these comment lines -->
- The `squash` command is in preview
- Support for squashing from the console is WIP
- Support for squashing migrations that are not committed yet is planned.
- Un-tracking or dropping a table will cause inconsistent squashed down migration since console doesn't generate correct down migration.
- If cascade setting is set to `true` on any of the metadata action, generated migration may be wrong
### Description
Adds a `metadata diff` command to show comparisons between two different sets of Hasura metadata.
```
# Show changes between server metadata and the exported metadata file:
hasura metadata diff
# Show changes between server metadata and that in local_metadata.yaml:
hasura metadata diff local_metadata.yaml
# Show changes between metadata from metadata.yaml and metadata_old.yaml:
hasura metadata diff metadata.yaml metadata_old.yaml
```
Also adds a `--dry-run` flag to `metadata apply` command which will print the diff and exit rather than actually applying the metadata.
### Affected components
- CLI
- Docs
### Related Issues
Close#3126, Close#3127
### Solution and Design
- Added `metadata_diff.go` and `metadata_diff_test.go`
### Steps to test and verify
```
hasura metadata export
# Make changes to migrations/metadata.yaml
hasura metadata diff
```
### Limitations, known bugs & workarounds
This is just a general-purpose diff.
A more contextual diff with the understanding of metadata can be added once #3072 is merged.
