* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.
## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content
There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).
## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
* Rename `TileCopierOptions` -> `TileCopier`
* remove a few un-needed sqlite open to detect mbtiles type
* move `open_and_detect_type` to `MBTiles`
* add `attach_to` to `MBTiles`
* move various table creation fn to mbtiles_queries file
* a few sql format
* Do not open the same mbtiles file more than once at the same time:
reuse the (unsafe) sqlite handle to create a `RusqliteConnection`
instead.
* The copying should set `agg_tiles_hash` in all cases because now it
uses the always available `tiles` table/view.
* a few minor cleanups and renames related to that
* Rename `global_hash` to `agg_tiles_hash`
This is still a big sticking point: what should be the name for the
metadata key for this value? The value represents the hash of all
`z,x,y,tile` over all rows of the `tiles` table (or view). Should it
include `md5` in its name, or should the hash be auto-detected by its
length? (details in #856)
* Generate it based on `tiles` table/view
* validate or generate, but not both (it will always fail otherwise)
* break up logic for per-tile, total, and integrity checks
* delete unused sqlx prep file
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)
---------
Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
* rename configuration `auto_publish.tables.id_format` and
`auto_publish.functions.id_format` fields from `id_format` to
`source_id_format` fields. The `id_format` will continue to be supported
(read) from the configuration, but it will be auto-converted to the new
name on save. It is an error to have both in the same config file.
* The rename was discussed in #682
* internal refactorings: consolidate PG-related utilities, rename a few
vars, move PG errors to their own file.
This is partially made due to #790 (thanks @Binabh!) - and should be
merged before that to make that PR easier.
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
This is mostly a noop, just updating to the new way of storing cached
queries and a few other changes per
https://github.com/launchbadge/sqlx/blob/main/CHANGELOG.md#070---2023-06-30
Note that this uses some of the code from #729 (thanks @upsicleclown !)
A minor unrelated change - a reformat of `martin/release.toml`
I'd like to add optional rustls support for SQLx. This PR implements the
required features, but tests using `--all-features` fail, because
runtime selection features are not additive. I didn't find a solution
apart from avoiding `--all-features`.
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
Add `--force-simple` flag to `mbtiles copy` tool.
If supplied, ensures the destination file has a `tiles` _table_ (as
opposed to a `tiles` _view_ made up of `images` and `map` tables).
Copy an existing `.mbtiles` file to a new file.
```shell
mbtiles copy <src_file.mbtiles> <dst_file.mbtiles>
```
Optionally filters by zooms. Supports de-duplicated and simple mbtiles
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
* moved `IdResolver` to a separate file
* added `just fmt2` build target to format code using nightly mode
* moved all Actix `Data<AppState>` into individual state objects, e.g.
`Data<Sources>` (allows other source types, separate from `Sources`
type)
* move all Source-related code to a new `Sources` struct (a simple
wrapper over a HashMap)
Feature for `Mbtiles` struct to detect which format a `.mbtiles` file is
in according to the [MBTiles
specfication](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md#database).
The function `detect_type` identifies whether the `.mbtiles` file
contains a `tiles` _table_ OR if it contains `map` and `images` table
(which provide the data for a `tiles` _view_).
See also #667
* Broke up martin-mbtiles into multiple files
* Made all mbtiles functions take a `SqliteExecutor` -- this way they
can be used with any SQLX connection structs - either a pool connection
or an individual non-pooled connection.
* Simplified mbtiles bin a bit - I realized there is really no need to
pretty print the output for the single value retrieval. Easier to just
dump it to console as is.
* Bump martin-mbtiles to v0.2.0
* Minor fixes in tools docs, cargo.toml, and justfile
* MBTiles tool Integration tests and release publishing
Major thanks to the
[stackoverflow](https://stackoverflow.com/questions/76394665/how-to-pass-sqlx-connection-a-mut-trait-as-a-fn-parameter-in-rust/76395111)
quick reply by @cafce25 on how to use generic sql executor!
* Add functionality to retrieve a metadata value in an mbtiles file by
key; simple implementation for one of the items in #667
* Also, disable TTY in docker-up `just` target
