When applying raw (no gzip) patches, `mbtiles` was trying to un-gzip
them first. Now handles it properly. Also adds a number of tests to catch these cases.
* `mbtiles diff` now has an additional `--patch-type` param with
`whole`, `bin-diff-raw` and `bin-diff-gz` values:
* `whole` stores different tiles as before - as whole tiles in the
`tiles` table
* `bin-diff-raw` computes binary difference between tiles, and stores
them as brotli-encoded value in a `bsdiffraw` table, together with a
`xxh3_64` hash of the tile as it will be stored after patching
* `bin-diff-gz` same as `bin-diff-raw`, but assumes the tiles are
gzip-compressed, so it uncompresses them before comparing. The `xxh3_64`
stores the hash of the uncompressed tile. The data will be stored in the
`bsdiffrawgz` table (identical structure with above)
* `mbtiles copy --apply-patch` will automatically detect if
`bsdiffrawgz` or `bsdiffraw` tables exist, and will use binary patching.
* `mbtiles apply-patch` does not support binary patching yet
* `mbtiles copy --diff-with-file ... --patch-type ...` is an alias to
`mbtiles diff --patch-type ...`
* Install `nasm` when running CI on Windows (required by rustls new
faster crypto-provider)
* In integration testing, wrap connection in mutex due to `rstest` now
requiring `once` fixtures to be `Sync`
* Upgrade to Brotli v4, and disable FFI due to compilation conflict
This is just a documentation change - the actual value stored in the
mbtiles was always `agg_tiles_hash_after_apply`, but accidentally the
constant in the code was not renamed, nor were the docs updated.
If a diff between two mbtiles is empty, `agg_tiles_hash_after_apply`
must still be generated. It was not before because two identical
tilesets would have the same `agg_tiles_hash`, thus skipping the
generation of `agg_tiles_hash_after_apply`
* Some small test cleanups
* A bump to dependencies
* Disable sprite generation test - PNGs tests are too flaky
* Add an mbtiles copy test for empty diffs
Implement `mbtiles diff a.mbtiles b.mbtiles diff.mbtiles` command
This should behave exactly the same as `mbtiles copy a.mbtiles
--diff-with-file b.mbtiles diff.mbtiles`.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
Require SQLite v3.44+ ORDER BY clause inside aggregate function instead
of the windowing one - might solve out of memory issues reported by
users - see #1154
Add a top level config parameter -- the size of cache memory (in MB) to
use for caching tiles and PMT directories, defaulting to 512, and 0 to
disable. This also removes the `pmtiles.dir_cache_size_mb` parameter (it
will be ignored, but will give a warning)
```
cache_size_mb: 512
```
The new cache will contain all tiles as provided by the source. So if
PostgreSQL returns a non-compressed tile, the cache will contain the
uncompressed variant, and will be compressed for each response. This
will be fixed in the later releases.
Note that fonts and sprites are not cached at this time, and are still a
TODO.
* Remove `CA_ROOT_FILE` env var support (has been deprecated for a
while)
* Remove `WATCH_MODE` env var warning - has not been in use since a long
time ago
* minor lints/clippy suggestions, lock update
* Use `Option<&T>` instead of `&Option<T>` in function arguments.
* Cleaner var names
* Slight optimization of `get_tile` with query params
* Split up srv/server.rs into fonts, sprites, tiles, and tiles_info
files
* better error reporting in tests
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
* **postgres** - enable PostgreSQL/PostGIS tile sources
* **pmtiles** - enable PMTile tile sources
* **mbtiles** - enable MBTile tile sources
* **fonts** - enable font sources
* **sprites** - enable sprite sources
* Use justfile in the CI
Fixes#1093
* `mbtiles update` now allows different types of zoom updates - reset to
content, grow only, or skip (dry run)
* `martin-cp` will now update (grow-only) metadata zooms
Addresses a few concerns in the #1081
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.
PMTiles config now also allows `http` and `https` protocol.
```
# Publish PMTiles files
pmtiles:
paths:
# specific pmtiles file will be published as mypmtiles source
# (use last portion of the URL without extension)
- http://example.org/path/to/mypmtiles.pmtiles
sources:
# named source matching source name to a single file
pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
```
fixes#884
---------
Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
* BREAKING: `martin-cp` will now set `format=pbf` instead of `mvt`. This
is what QGIS and possibly others expect, and this is what tools like
tilelive generates.
* `martin-cp` sets `minzoom` and `maxzoom` metadata values based on the
zoom parameters
* Add `mbtiles meta-update` command to refresh zoom levels based on the
present tiles.
Partially addresses items in #1081
Figure out and fix the bug with `tile-grid` crate or remove the crate
and implement it ourself.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
When using `martin-cp` or `mbtiles copy` into an existing file, require
the user to set `--on-duplicate (override|ignore|abort)`. This prevents
accidental writing to an existing file.
Make sure all values in the `tiles` table or view are correct:
* zoom_level is between 0 and 30 (max allowed zoom)
* the x,y values are within the limits for the corresponding zoom level
* the column type of z,x,y are all integers
* the `tile_data` is a NULL or a BLOB
Mathematics should be consolidated in the utils crate. Eventually, I
hope we will move it to [tile-grid](https://crates.io/crates/tile-grid)
or [utiles](https://crates.io/crates/utiles) crates and consolidate our
efforts there.
Also, modify earth circumference by a few centimeters (to match various
other implementations): from `.7` to `.685_578_5`. This modified a few
unit tests.
* Now by default mbtiles will not print internal warnings (e.g. that
something is taking too long to execute)
* remove mbtiles --help test - not really needed, but adds complexity
because it has to be updated every time a version changes.
* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.
`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files
# Other
* Always sort JSON-serialized keys for consistency
* this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
