mirror of
https://github.com/djrobstep/migra.git
synced 2024-08-16 01:01:00 +03:00
docs update
This commit is contained in:
Robert Lechte
parent
b2602eb1af
commit
ee0f7d6a55
11
docs/comparison.md
Normal file
11
docs/comparison.md
Normal file
@ -0,0 +1,11 @@
|
||||
<div class="scroll-on-overflow full-width " markdown="1">
|
||||
| Feature | migra | alembic | django |
|
||||
| - | - | - | - |
|
||||
| Migration files required for DB changes during dev? | No, autosyncs | Yes, required | Yes, required |
|
||||
| Requires schema version number tracking | No version numbers involved | Yes, relies on version numbers | Yes, relies on version numbers |
|
||||
| Must store entire chain of migration files? | No, only need one file with pending changes | Yes, required | Yes, required |
|
||||
| ORMs supported | Supports any ORM or no ORM | SQLAlchemy ORM only | Django ORM only |
|
||||
| Testability | Explicitly tests for matching schema | Doesn't test | Doesn't test |
|
||||
| Needs copy of access to (copy of) current schema to generate migration scripts | Yes | No | No |
|
||||
| Databases supported | PostgreSQL only | Postgres, mysql, various others | Postgres, mysql, various others |
|
||||
</div>
|
||||
@ -1,3 +1,5 @@
|
||||
# Deploy Usage
|
||||
|
||||
With migra you can dispense with schema version numbers and multiple migration files.
|
||||
|
||||
## Different deployment styles
|
||||
@ -1,3 +1,5 @@
|
||||
# Development Usage
|
||||
|
||||
Migra is handy for speeding up database-related development and testing tasks.
|
||||
|
||||
### Auto-syncing dev database to application code
|
||||
@ -1,14 +0,0 @@
|
||||
# `migra` is a schema comparison tool for PostgreSQL.
|
||||
|
||||
It's a command line tool, and Python library. Find differences in database schemas as easily as running a diff on two text files.
|
||||
|
||||
Migra makes schema changes almost automatic. Management of database migration deployments becomes much easier, faster, and more reliable.
|
||||
|
||||
Using `migra` is as simple as this:
|
||||
|
||||
$ migra postgresql:///a postgresql:///b
|
||||
alter table "public"."book" add column "author" character varying not null;
|
||||
|
||||
alter table "public"."book" alter column "name" set not null;
|
||||
|
||||
To get started, hit the [Quickstart guide](/quickstart).
|
||||
87
docs/index.md
Normal file
87
docs/index.md
Normal file
@ -0,0 +1,87 @@
|
||||
# `migra` is a schema comparison tool for PostgreSQL.
|
||||
|
||||
It's a command line tool, and Python library. Find differences in database schemas as easily as running a diff on two text files.
|
||||
|
||||
`migra` makes schema changes almost automatic. Management of database migration deployments becomes much easier, faster, and more reliable.
|
||||
|
||||
Using `migra` is as simple as this:
|
||||
|
||||
$ migra postgresql:///a postgresql:///b
|
||||
alter table "public"."book" add column "author" character varying not null;
|
||||
|
||||
alter table "public"."book" alter column "name" set not null;
|
||||
|
||||
To get started, hit the [Quickstart guide](/docs/migra/quickstart).
|
||||
|
||||
Migra is 100% open source software. The code is available on [github](https://github.com/djrobstep/migra).
|
||||
|
||||
## Features and Limitations
|
||||
|
||||
The following features of postgres are supported:
|
||||
|
||||
<div markdown="block" class="full-width scroll-on-overflow">
|
||||
|
||||
Feature | Supported | Notes/limitations
|
||||
--- | --- | ---
|
||||
tables | ✔ |
|
||||
partitioned tables | ✔ |
|
||||
constraints | ✔ |
|
||||
views | ✔ |
|
||||
functions | ✔ | All languages except C/INTERNAL
|
||||
indexes | ✔ |
|
||||
sequences | ✔ | Does not track sequence numbers
|
||||
schemas | ✔ |
|
||||
extensions | ✔ |
|
||||
enums | ✔ |
|
||||
privileges | ✔ | Not exhaustive. Requires --with-privileges flag
|
||||
row-level security | ✔ | NEW! Doesn't include role management
|
||||
triggers | ✔ |
|
||||
identity columns | ✔ |
|
||||
generated columns | ✔ |
|
||||
custom types/domains | ✔ |Basic support (drop-and-create only, no alter) |
|
||||
|
||||
</div>
|
||||
|
||||
`migra` plays nicely with extensions. Schema contents belonging to extensions will be ignored and left to the extension to manage.
|
||||
|
||||
`migra` plays nicely with view/function dependencies, and will drop/create them in the correct order.
|
||||
|
||||
## Endorsements
|
||||
|
||||
`migra` was [used to manage the schema that powers PyPI](https://twitter.com/dstufft/status/988410901459034113):
|
||||
|
||||
> *Migra is cool as hell though, when I was trying to reconcile PyPI with the alembic initial schema I had to download some weird java app I found on some sketchy website to do that :P*
|
||||
|
||||
>- [Donald Stufft](https://twitter.com/dstufft), PyPI maintainer
|
||||
|
||||
It's [good for local development](https://news.ycombinator.com/item?id=16676481):
|
||||
|
||||
> *I can definitely see Migra is more productive when switching around between schemas in development.*
|
||||
|
||||
>- [Mike Bayer](https://twitter.com/zzzeek), SQLAlchemy author
|
||||
|
||||
<div markdown="block" class="hilighted">
|
||||
|
||||
## CI for Databases?
|
||||
|
||||
If you like migra, you might also be interested in [CI for databases](/ci-for-databases), a new service I'm building to solve problems with database deployments. [Register your interest](/ci-for-databases).
|
||||
|
||||
</div>
|
||||
|
||||
## Development
|
||||
|
||||
Migra is developed on [github](https://github.com/djrobstep/migra). Contributions are welcome, get involved!
|
||||
|
||||
## Philosophy
|
||||
|
||||
There was [a good comment on Hacker News](https://news.ycombinator.com/item?id=16679665) discussing how `migra` differs from traditional migration tools.
|
||||
|
||||
> *This is awesome!*
|
||||
|
||||
> *A lot of people point to migrations as the best way to track changes to a database schema. But there are a lot of problems with them. For example, they involve adding version control on top of another version control system, which can cause a ton of problems. They also don't maintain themselves well. If you leave it alone, running migrations will just take longer and longer over time, even though you don't really get more utility.*
|
||||
|
||||
> *I think we need more support from databases themselves to solve this problem.*
|
||||
|
||||
> *In the meantime, this is a really good stopgap, because it can theoretically allow you to just have a file with your "ideal schema" for each commit. No need to maintain a separate database migration history too. You can even generate a series of migrations by looking at your git history!"*
|
||||
|
||||
> - `blaisio` on Hacker News
|
||||
58
docs/installing_and_connecting.md
Normal file
58
docs/installing_and_connecting.md
Normal file
@ -0,0 +1,58 @@
|
||||
# Installing and connecting
|
||||
|
||||
`migra` is written in Python and thus requires a recent version of Python to be installed. Make sure you use at least Python 3.6: `migra` will very soon stop supporting earlier versions (Python 2 will soon reach end-of-life anyway).
|
||||
|
||||
## Installing migra
|
||||
|
||||
Install migra and necessary dependencies with the usual `pip install`:
|
||||
|
||||
:::bash
|
||||
pip install migra
|
||||
|
||||
This doesn't include the PostgreSQL driver, but you can install that separately, or at the same with:
|
||||
|
||||
:::bash
|
||||
pip install migra[pg]
|
||||
|
||||
If you're on `zsh`, you'll need quote marks, as in: `'migra[pg]'`.
|
||||
|
||||
|
||||
## Connecting
|
||||
|
||||
`migra` uses database URLs to specify database connections. If you're unfamiliar with these, they work very much like a URL, incorporating all the details necessary for making a database connection into one concise string.
|
||||
|
||||
The general form is as follows:
|
||||
|
||||
:::bash
|
||||
connectiontype://username:password@databasehostname/databasename?extraparam1=a&extraparam2=b
|
||||
|
||||
These can be left out if not necessary. For instance, if connecting locally, using your default system username and passwordless "trust" login, only a connection type and database name is required:
|
||||
|
||||
:::bash
|
||||
postgresql:///exampledatabase
|
||||
|
||||
Note that all the slashes are still required, so there are three slashes in a row.
|
||||
|
||||
Alternate drivers can be specified with a `+` modifier to the connection type:
|
||||
|
||||
:::bash
|
||||
postgresql+pg9000:///example
|
||||
|
||||
For further details, consult the `sqlalchemy` docs: This is what migra uses under the hood to do connections (via a wrapper module called `sqlbag`).
|
||||
|
||||
## Background: Installing python and pip
|
||||
|
||||
### Python
|
||||
|
||||
Your operating system may include Python, but possibly not the latest version. `migra` will always work best with the latest stable version of Python, so ideally install the latest stable release (as specified on [python.org](https://python.org/)).
|
||||
|
||||
On Windows/MacOS, you can use the installer from [python.org](https://python.org/). On linux, as always, this situation is more complicated: Your default OS repos often have version 2 installed by default, and if they have a separate version of Python 3 it won't be the latest one. You can add various external repos or build your own version from source. I wrote a tool called [autovenv](/docs/autovenv) that manages multiple python versions (and virtual environments). Feel free to try it (I use it on both MacOS and linux), but it may be buggy (it isn't yet production software).
|
||||
|
||||
### pip
|
||||
|
||||
Once you have python installed you need to install `pip`. This too can complicated - recent python versions automatically install pip too, but often OS-managed versions do not. Check that when you run `pip -V` that it runs the desired Python version you have installed.
|
||||
|
||||
## Aside: Virtual environments
|
||||
|
||||
`python` has the notion of `virtual environments` (often called `virtualenv`s or `venv`s) which create an isolated place to install external python packages separate from what might have already be installed by the operating system. It's generally good practice to use these when working with Python - but if you're not otherwise using Python and just need `migra` from the command line, these may not be worth bothering with.
|
||||
|
||||
@ -1,17 +0,0 @@
|
||||
site_name: migra.djrobstep.com
|
||||
pages:
|
||||
- migra: index.md
|
||||
- Quickstart: quickstart.md
|
||||
- Development usage: dev_usage.md
|
||||
- Deployment usage: deploy_usage.md
|
||||
|
||||
theme: 'material'
|
||||
|
||||
repo_url: https://github.com/djrobstep/migra
|
||||
google_analytics: ['UA-94078517-1', 'migra.djrobstep.com']
|
||||
|
||||
markdown_extensions:
|
||||
- fenced_code
|
||||
- meta
|
||||
- toc
|
||||
- codehilite
|
||||
30
docs/options.md
Normal file
30
docs/options.md
Normal file
@ -0,0 +1,30 @@
|
||||
# Options
|
||||
|
||||
The command line version of migra includes a number of options for getting the diff output you desire. If using the API, similarly named options are available on the Migration object.
|
||||
|
||||
## `--unsafe`
|
||||
|
||||
Migra will throw an exception if `drop...` statements are generated, as a precaution. Adding this flag will disable the safety feature and happily generate the drop statements. Remember, always review generated statements before running them!
|
||||
|
||||
## `--schema [SCHEMA_NAME]`
|
||||
|
||||
Specify a single schema to diff.
|
||||
|
||||
## `--create-extensions-only`
|
||||
|
||||
Only output create extension statements, nothing else. This is useful when you have extensions as part of a setup script for a single schema: Those extensions need to be installed, but extensions are usually not installed in a custom schema.
|
||||
|
||||
You'd generate a setup script in two steps:
|
||||
|
||||
- Generate the necessary create extension if not exists... statements.
|
||||
- Generate the schema changes with --schema-only.
|
||||
|
||||
Then combine the output of 1 and 2 into a single database sync script.
|
||||
|
||||
## `--with-privileges`
|
||||
|
||||
This tells migra to spit out permission-related change statements (grant/revoke). This is False by default: Often one is comparing databases from different environments, where the users and permissions are completely different and not something one would want to sync.
|
||||
|
||||
## `--force-utf8`
|
||||
|
||||
Some folks have reported unicode character output issues on windows command lines. This flag often fixes it!
|
||||
@ -1,4 +1,6 @@
|
||||
### Installation
|
||||
# Quickstart
|
||||
|
||||
## Installation
|
||||
|
||||
`migra` is written in Python so you need to install it with `pip`, the Python Package Manager (don't worry, you don't need to know or use any Python to use the `migra` command).
|
||||
|
||||
@ -6,23 +8,17 @@
|
||||
|
||||
2. Run:
|
||||
|
||||
:::bash
|
||||
$ pip install migra[pg]
|
||||
pip install migra[pg]
|
||||
|
||||
This will install the latest version of migra from PyPI (the global Python Package Index), along with psycopg2, the python PostgreSQL driver.
|
||||
|
||||
Alternatively, if you don't want to install Python, you can run it from a
|
||||
self-contained Docker image by first running:
|
||||
Alternatively, if you don't want to install Python, you can run it from a self-contained Docker image by first running:
|
||||
|
||||
```shell script
|
||||
docker pull djrobstep/migra
|
||||
```
|
||||
docker pull djrobstep/migra
|
||||
|
||||
then creating a short alias to it with:
|
||||
then creating a short alias to it with:
|
||||
|
||||
```shell script
|
||||
alias migra="docker run djrobstep/migra migra"
|
||||
```
|
||||
alias migra="docker run djrobstep/migra migra"
|
||||
|
||||
3. Confirm migra is installed by running `migra --help`. The output should begin like this:
|
||||
|
||||
@ -32,11 +28,12 @@ alias migra="docker run djrobstep/migra migra"
|
||||
|
||||
To compare two database schemas:
|
||||
|
||||
$ migra <url_of_database_A> <url_of_database_B>
|
||||
migra [url_of_database_A] [url_of_database_B]
|
||||
|
||||
For example, we have two databases, named "alpha" and "beta". We can compare them using this command:
|
||||
|
||||
$ migra postgresql:///alpha postgresql:///beta
|
||||
:::shell
|
||||
migra postgresql:///alpha postgresql:///beta
|
||||
|
||||
Migra will then generate whatever SQL is required to change the schema of database `alpha` to match database `beta`.
|
||||
|
||||
@ -48,6 +45,7 @@ Don't blindly copy-and-paste the output of the `migra` command.
|
||||
|
||||
`migra` features a safeguard against generation of dangerous statements. If the command generates a drop statement, `migra` will exit with an error. If you're sure you want the drop statement(s), you can turn off this safeguard behaviour with the `--unsafe` flag:
|
||||
|
||||
:::shell
|
||||
migra --unsafe postgresql:///alpha postgresql:///beta
|
||||
|
||||
## Making changes to database schemas
|
||||
@ -58,19 +56,20 @@ If you're making changes to a serious production database, use a copy of it for
|
||||
|
||||
You can make a schema-only dump of your PostgreSQL database with the following command:
|
||||
|
||||
pg_dump --no-owner --no-privileges --no-acl --schema-only name_of_database -f schema.dump.sql
|
||||
:::shell
|
||||
pg_dump --no-owner --no-privileges --schema-only name_of_database -f schema.dump.sql
|
||||
|
||||
### Steps
|
||||
|
||||
1. Get the connection string of the database you want to make changes to. `migra` needs to connect to this database so it can analyse the database's schema.
|
||||
#### Connect
|
||||
Get the connection string of the database you want to make changes to. `migra` needs to connect to this database so it can analyse the database's schema.
|
||||
|
||||
2. Prepare a second PostgreSQL database. This database needs to have the new/desired/target schema. You might create a temporary database and set it up for this purpose.
|
||||
|
||||
3. Generate a migration script using the following command (substituting your own connection strings):
|
||||
|
||||
```
|
||||
$ migra --unsafe postgresql:///existing postgresql:///database_with_new_schema > migration_script.sql
|
||||
```
|
||||
:::shell
|
||||
migra --unsafe postgresql:///existing postgresql:///database_with_new_schema > migration_script.sql
|
||||
|
||||
4. Carefully review the migration script in `migration_script.sql`
|
||||
|
||||
@ -78,9 +77,9 @@ You can make a schema-only dump of your PostgreSQL database with the following c
|
||||
|
||||
- The generated script may result in data loss from your database when you apply this script. Consider if you intend for this to happen or if you need to add statements to copy data out of the relevant tables/columns before you drop them forever.
|
||||
|
||||
- Some migration operations can take a long time and cause interruptions and downtime, particularly when involving tables containing large amounts of data. [More on this here](/link).
|
||||
- Some migration operations can take a long time and cause interruptions and downtime, particularly when involving tables containing large amounts of data..
|
||||
|
||||
5. Apply `migration_script.sql` to your production database with a command similar to the following (again substituting your own connection string).
|
||||
|
||||
```psql postgresql://production -1 -f migration_script.sql
|
||||
```
|
||||
:::shell
|
||||
psql postgresql://production -1 -f migration_script.sql
|
||||
109
docs/with_django.md
Normal file
109
docs/with_django.md
Normal file
@ -0,0 +1,109 @@
|
||||
# Using `migra` with `django`
|
||||
|
||||
Incorporating non-default parts with django can be a little awkward. However, with some config tweaks you can use migra alongside django's default migration setup.
|
||||
|
||||
Below is a sample script that implements a hybrid sync command: It'll run any defined django migrations as normal, but allows you to specify a list of installed django "`apps`" that you can manage with `migra` instead.
|
||||
|
||||
## Deploying
|
||||
|
||||
The general advice elsewhere in the docs regarding the creation of migration scripts for your production environment still applies as normal.
|
||||
|
||||
A script to generate production scripts would like quite similar the provided local-sync script below, except that the comparison would be `production -> target` instead of `current local -> target`.
|
||||
|
||||
## Testing
|
||||
|
||||
Initialize your database for tests as follows:
|
||||
|
||||
- `migrate` any django migrations in use
|
||||
- sync your prepared script of changes
|
||||
|
||||
## A sample local syncing script
|
||||
`migra`'s built-in apps come with some migrations, so you don't want to disable the built-in migrations entirely.
|
||||
|
||||
:::python
|
||||
import manage
|
||||
import os
|
||||
import sys
|
||||
from mysite import settings
|
||||
from django.core import management
|
||||
import django
|
||||
from sqlbag import temporary_database, S
|
||||
from migra import Migration
|
||||
from contextlib import contextmanager
|
||||
|
||||
# Point to your settings.
|
||||
os.environ["DJANGO_SETTINGS_MODULE"] = "mysite.settings"
|
||||
|
||||
# The "polls" app (as per the django official tutorial)
|
||||
# is set here to be managed with migra instead
|
||||
MANAGE_WITH_MIGRA = ["polls"]
|
||||
|
||||
# To disable migrations on a django app, you have to
|
||||
# set the MIGRATION_MODULES config with None for each
|
||||
# disabled "app"
|
||||
class DisableMigrations(object):
|
||||
def __contains__(self, item):
|
||||
return item in MANAGE_WITH_MIGRA
|
||||
|
||||
def __getitem__(self, item):
|
||||
return None
|
||||
|
||||
# Compare two schemas, prompt to run a sync if necessary
|
||||
def _sync_with_prompt(db_url_current, db_url_target):
|
||||
with S(db_url_current) as s0, S(db_url_target) as s1:
|
||||
m = Migration(s0, s1)
|
||||
m.set_safety(False)
|
||||
m.add_all_changes()
|
||||
|
||||
if m.statements:
|
||||
print("THE FOLLOWING CHANGES ARE PENDING:", end="\n\n")
|
||||
print(m.sql)
|
||||
print()
|
||||
if input('Type "yes" to apply these changes: ') == "yes":
|
||||
print("Applying...")
|
||||
m.apply()
|
||||
else:
|
||||
print("Not applying.")
|
||||
else:
|
||||
print("Already synced.")
|
||||
|
||||
|
||||
# Create a temporary database for loading our
|
||||
# "target" schema into
|
||||
@contextmanager
|
||||
def tmptarget():
|
||||
with temporary_database() as tdb:
|
||||
settings.DATABASES["tmp_target"] = {
|
||||
"ENGINE": "django.db.backends.postgresql",
|
||||
"NAME": tdb.split("/")[-1],
|
||||
}
|
||||
django.setup()
|
||||
yield tdb
|
||||
|
||||
|
||||
def syncdb():
|
||||
# Disable django migrations if we're using migra instead
|
||||
settings.MIGRATION_MODULES = DisableMigrations()
|
||||
|
||||
|
||||
with tmptarget() as tdb:
|
||||
management.call_command("migrate")
|
||||
|
||||
management.call_command(
|
||||
"migrate",
|
||||
"--run-syncdb",
|
||||
"--database=tmp_target",
|
||||
verbosity=0,
|
||||
interactive=False,
|
||||
)
|
||||
real_db_name = settings.DATABASES["default"]["NAME"]
|
||||
_sync_with_prompt(f"postgresql:///{real_db_name}", tdb)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
syncdb()
|
||||
|
||||
|
||||
# Can it be done better?
|
||||
|
||||
How well does `migra` work for you with django? Let us know via email, github issues, twitter, whatever.
|
||||
Loading…
Reference in New Issue
Block a user