* Rename `TileCopierOptions` -> `TileCopier`
* remove a few un-needed sqlite open to detect mbtiles type
* move `open_and_detect_type` to `MBTiles`
* add `attach_to` to `MBTiles`
* move various table creation fn to mbtiles_queries file
* a few sql format
* Do not open the same mbtiles file more than once at the same time:
reuse the (unsafe) sqlite handle to create a `RusqliteConnection`
instead.
* The copying should set `agg_tiles_hash` in all cases because now it
uses the always available `tiles` table/view.
* a few minor cleanups and renames related to that
* Rename `global_hash` to `agg_tiles_hash`
This is still a big sticking point: what should be the name for the
metadata key for this value? The value represents the hash of all
`z,x,y,tile` over all rows of the `tiles` table (or view). Should it
include `md5` in its name, or should the hash be auto-detected by its
length? (details in #856)
* Generate it based on `tiles` table/view
* validate or generate, but not both (it will always fail otherwise)
* break up logic for per-tile, total, and integrity checks
* delete unused sqlx prep file
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)
---------
Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
* rename configuration `auto_publish.tables.id_format` and
`auto_publish.functions.id_format` fields from `id_format` to
`source_id_format` fields. The `id_format` will continue to be supported
(read) from the configuration, but it will be auto-converted to the new
name on save. It is an error to have both in the same config file.
* The rename was discussed in #682
* internal refactorings: consolidate PG-related utilities, rename a few
vars, move PG errors to their own file.
This is partially made due to #790 (thanks @Binabh!) - and should be
merged before that to make that PR easier.
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
This is mostly a noop, just updating to the new way of storing cached
queries and a few other changes per
https://github.com/launchbadge/sqlx/blob/main/CHANGELOG.md#070---2023-06-30
Note that this uses some of the code from #729 (thanks @upsicleclown !)
A minor unrelated change - a reformat of `martin/release.toml`
I'd like to add optional rustls support for SQLx. This PR implements the
required features, but tests using `--all-features` fail, because
runtime selection features are not additive. I didn't find a solution
apart from avoiding `--all-features`.
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
Add `--force-simple` flag to `mbtiles copy` tool.
If supplied, ensures the destination file has a `tiles` _table_ (as
opposed to a `tiles` _view_ made up of `images` and `map` tables).
Copy an existing `.mbtiles` file to a new file.
```shell
mbtiles copy <src_file.mbtiles> <dst_file.mbtiles>
```
Optionally filters by zooms. Supports de-duplicated and simple mbtiles
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
* moved `IdResolver` to a separate file
* added `just fmt2` build target to format code using nightly mode
* moved all Actix `Data<AppState>` into individual state objects, e.g.
`Data<Sources>` (allows other source types, separate from `Sources`
type)
* move all Source-related code to a new `Sources` struct (a simple
wrapper over a HashMap)
Feature for `Mbtiles` struct to detect which format a `.mbtiles` file is
in according to the [MBTiles
specfication](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md#database).
The function `detect_type` identifies whether the `.mbtiles` file
contains a `tiles` _table_ OR if it contains `map` and `images` table
(which provide the data for a `tiles` _view_).
See also #667
* Broke up martin-mbtiles into multiple files
* Made all mbtiles functions take a `SqliteExecutor` -- this way they
can be used with any SQLX connection structs - either a pool connection
or an individual non-pooled connection.
* Simplified mbtiles bin a bit - I realized there is really no need to
pretty print the output for the single value retrieval. Easier to just
dump it to console as is.
* Bump martin-mbtiles to v0.2.0
* Minor fixes in tools docs, cargo.toml, and justfile
* MBTiles tool Integration tests and release publishing
Major thanks to the
[stackoverflow](https://stackoverflow.com/questions/76394665/how-to-pass-sqlx-connection-a-mut-trait-as-a-fn-parameter-in-rust/76395111)
quick reply by @cafce25 on how to use generic sql executor!
* Add functionality to retrieve a metadata value in an mbtiles file by
key; simple implementation for one of the items in #667
* Also, disable TTY in docker-up `just` target
Compression middleware turned out to be hard to use for image cases - it
simply looks at the content-encoding, and if not set, tries to compress
if accepted by the client.
Instead, now individual routes are configured with either that
middleware, or for tiles, I decompress and optionally recompress if
applicable.
Now encoding is tracked separately from the tile content, making it
cleaner too. Plus lots of tests for mbtiles & pmtiles.
Fixes#577
Adds a new [.mbtiles](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md)
backend, without the grid support. Uses extensive tile content
detection, i.e. if the content is gzipped, png, jpeg, gif, webp.
From CLI, can be as easy as adding a path to a directory that contains a
.mbtiles file (works just like pmtiles support)
```bash
# All *.mbtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```
From configuration file, the path can be specified in a number of ways
(same as pmtiles)
```yaml
mbtiles:
paths:
# scan this whole dir, matching all *.mbtiles files
- /dir-path
# specific mbtiles file will be published as mbtiles2 source
- /path/to/mbtiles2.mbtiles
sources:
# named source matching source name to a single file
pm-src1: /tmp/mbtiles.mbtiles
# named source, where the filename is explicitly set. This way we will be able to add more options later
pm-src2:
path: /tmp/mbtiles.mbtiles
```
Fixes#494
