Commit Graph

6 Commits

Author SHA1 Message Date
Yuri Astrakhan
b71c846abc
fix for mbtiles apply-patch with raw bindiff (#1384)
When applying raw (no gzip) patches, `mbtiles` was trying to un-gzip
them first. Now handles it properly.  Also adds a number of tests to catch these cases.
2024-06-26 20:37:20 +00:00
Yuri Astrakhan
ccd550a2cf
Add binary diff for mbtiles (#1358)
* `mbtiles diff` now has an additional `--patch-type` param with
`whole`, `bin-diff-raw` and `bin-diff-gz` values:
* `whole` stores different tiles as before - as whole tiles in the
`tiles` table
* `bin-diff-raw` computes binary difference between tiles, and stores
them as brotli-encoded value in a `bsdiffraw` table, together with a
`xxh3_64` hash of the tile as it will be stored after patching
* `bin-diff-gz` same as `bin-diff-raw`, but assumes the tiles are
gzip-compressed, so it uncompresses them before comparing. The `xxh3_64`
stores the hash of the uncompressed tile. The data will be stored in the
`bsdiffrawgz` table (identical structure with above)

* `mbtiles copy --apply-patch` will automatically detect if
`bsdiffrawgz` or `bsdiffraw` tables exist, and will use binary patching.
* `mbtiles apply-patch` does not support binary patching yet
* `mbtiles copy --diff-with-file ... --patch-type ...` is an alias to
`mbtiles diff --patch-type ...`
2024-06-25 19:34:14 +00:00
Yuri Astrakhan
140ed25679
mbtiles improvements, minor changes (#1013)
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files

# Other
* Always sort JSON-serialized keys for consistency
  * this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
2023-11-19 10:06:37 +00:00
Yuri Astrakhan
8b34cd374c
Add metadat copy/apply-diff, new testing framework (#921)
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.

## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content

There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).

## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
2023-10-10 11:10:17 -04:00
Yuri Astrakhan
6f08aa9465
Fix mbtiles validation, CI, and logging (#903) 2023-09-29 14:37:18 -04:00
rstanciu
e004908722
Add diff-with-file option to copy command (#729)
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-07-05 16:55:39 +00:00