Commit Graph

24 Commits

Author SHA1 Message Date
Yuri Astrakhan
8ec040665f
Optimize produced release binaries (#1385)
Minor optimization to create a slightly more compact code (should be the
same performance, or possibly even faster)
2024-06-26 20:31:41 -04:00
Yuri Astrakhan
b71c846abc
fix for mbtiles apply-patch with raw bindiff (#1384)
When applying raw (no gzip) patches, `mbtiles` was trying to un-gzip
them first. Now handles it properly.  Also adds a number of tests to catch these cases.
2024-06-26 20:37:20 +00:00
Yuri Astrakhan
ccd550a2cf
Add binary diff for mbtiles (#1358)
* `mbtiles diff` now has an additional `--patch-type` param with
`whole`, `bin-diff-raw` and `bin-diff-gz` values:
* `whole` stores different tiles as before - as whole tiles in the
`tiles` table
* `bin-diff-raw` computes binary difference between tiles, and stores
them as brotli-encoded value in a `bsdiffraw` table, together with a
`xxh3_64` hash of the tile as it will be stored after patching
* `bin-diff-gz` same as `bin-diff-raw`, but assumes the tiles are
gzip-compressed, so it uncompresses them before comparing. The `xxh3_64`
stores the hash of the uncompressed tile. The data will be stored in the
`bsdiffrawgz` table (identical structure with above)

* `mbtiles copy --apply-patch` will automatically detect if
`bsdiffrawgz` or `bsdiffraw` tables exist, and will use binary patching.
* `mbtiles apply-patch` does not support binary patching yet
* `mbtiles copy --diff-with-file ... --patch-type ...` is an alias to
`mbtiles diff --patch-type ...`
2024-06-25 19:34:14 +00:00
Yuri Astrakhan
3dc54d7f9e
Implement tile caching (#1105)
Add a top level config parameter -- the size of cache memory (in MB) to
use for caching tiles and PMT directories, defaulting to 512, and 0 to
disable. This also removes the `pmtiles.dir_cache_size_mb` parameter (it
will be ignored, but will give a warning)

```
cache_size_mb: 512
```

The new cache will contain all tiles as provided by the source. So if
PostgreSQL returns a non-compressed tile, the cache will contain the
uncompressed variant, and will be compressed for each response. This
will be fixed in the later releases.

Note that fonts and sprites are not cached at this time, and are still a
TODO.
2023-12-30 22:48:23 -05:00
Yuri Astrakhan
2def6288f1
PMTiles cache, refactor file configs, modularize (#1094)
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
    * **postgres** - enable PostgreSQL/PostGIS tile sources
    * **pmtiles** - enable PMTile tile sources
    * **mbtiles** - enable MBTile tile sources
    * **fonts** - enable font sources
    * **sprites** - enable sprite sources
* Use justfile in the CI

Fixes #1093
2023-12-25 05:52:04 +00:00
Yuri Astrakhan
24819bdb45
Multiple mbtiles and martin-cp fixes (#1083)
* BREAKING: `martin-cp` will now set `format=pbf` instead of `mvt`. This
is what QGIS and possibly others expect, and this is what tools like
tilelive generates.
* `martin-cp` sets `minzoom` and `maxzoom` metadata values based on the
zoom parameters
* Add `mbtiles meta-update` command to refresh zoom levels based on the
present tiles.

Partially addresses items in #1081
2023-12-18 23:19:36 -05:00
Yuri Astrakhan
fc15e0da5e Preparing release, version bump 2023-12-18 20:24:43 -05:00
Yuri Astrakhan
bb802c5688
Feature: mbtiles copy --bbox ... to copy tiles within a bbox only (#1060)
Allow users to copy tiles between mbtiles files that are only within a
bounding box. Multiple `--bbox` params are allowed.
2023-12-15 23:13:57 -05:00
Yuri Astrakhan
280dbe162f
Move math utils to martin-tile-utils (#1056)
Mathematics should be consolidated in the utils crate. Eventually, I
hope we will move it to [tile-grid](https://crates.io/crates/tile-grid)
or [utiles](https://crates.io/crates/utiles) crates and consolidate our
efforts there.

Also, modify earth circumference by a few centimeters (to match various
other implementations): from `.7` to `.685_578_5`. This modified a few
unit tests.
2023-12-09 04:38:15 +00:00
Yuri Astrakhan
7ab3f5e21a
Clean up mbtiles tool output (#1048)
* Now by default mbtiles will not print internal warnings (e.g. that
something is taking too long to execute)
* remove mbtiles --help test - not really needed, but adds complexity
because it has to be updated every time a version changes.
2023-12-08 08:15:31 +08:00
Yuri Astrakhan
a33f980d12 update versions 2023-12-05 23:48:59 -05:00
Yuri Astrakhan
d6219a6526
Add compression and query to martin-cp (#1019)
* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
2023-11-22 05:26:53 +00:00
Yuri Astrakhan
999179029b Prepare for release 2023-11-20 01:51:08 -05:00
Lucas
a3b2bfd705
Minor cleanup (#1008)
- Add separators to long literal.
- Use first() instead of get(0)
- Move some SQL-making code to their own functions
- Minor mathematics and rounding cleanup

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-11-17 18:55:05 +00:00
Yuri Astrakhan
53ebd589c8 relax mbtiles pedantic requirements 2023-11-17 00:15:29 -05:00
Yuri Astrakhan
7e25f4c306
Minor refactoring and cleanup (#1007)
* remove benchmarks - not used for now, will need to be rewritten
* move some CLI settings to a separate struct
* move config saving to a shared function
* simplify zoom level CLI parsing
* rename main.rs to be the same name as generated binary (we can have
multiple bins later)
2023-11-16 19:59:43 -05:00
Yuri Astrakhan
d8e386fb0c
Use Rust's 1.74 cargo lint section (#1006)
https://blog.rust-lang.org/2023/11/16/Rust-1.74.0.html
2023-11-17 08:29:25 +08:00
Yuri Astrakhan
44313e3c6a
Fix mbtiles copy --dst-type arg name, better err msg (#1003)
* Fix `mbtiles copy --dst-type` to use a `-` instead of `_`
* Better error message for `agg_tiles_hash`
2023-11-14 02:29:00 +00:00
Yuri Astrakhan
8fa8f31336 bump cargo.lock, mbtiles ver bump 2023-11-13 16:58:26 -05:00
Lucas
0398336114
Add mbtiles statistics command (#986)
- [x] Add tile statistics to mbtiles tools
- [x] Add test
- [x] Use 4326 instead of 3857 for tile bounds
- [x] Add document
- [x] Use size-format to prettify output 
- [x]  Statistics  struct Refactor
- [x] Cleanup and reformat

Closes #964 

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-11-13 06:03:40 +00:00
Yuri Astrakhan
fad89d742f
simplify serde optional handling (#998) 2023-11-12 22:22:04 -05:00
Yuri Astrakhan
bfbe52d032 Add crate categories 2023-10-30 18:52:47 -04:00
Yuri Astrakhan
f1241e264c fix CI status badges 2023-10-28 16:29:38 -04:00
Yuri Astrakhan
47b3106ac6
Rename martin-mbtiles crate to mbtiles (#976)
This simplifies usage as both a crate and as a tool
2023-10-28 08:50:49 +00:00