When applying raw (no gzip) patches, `mbtiles` was trying to un-gzip
them first. Now handles it properly. Also adds a number of tests to catch these cases.
* `mbtiles diff` now has an additional `--patch-type` param with
`whole`, `bin-diff-raw` and `bin-diff-gz` values:
* `whole` stores different tiles as before - as whole tiles in the
`tiles` table
* `bin-diff-raw` computes binary difference between tiles, and stores
them as brotli-encoded value in a `bsdiffraw` table, together with a
`xxh3_64` hash of the tile as it will be stored after patching
* `bin-diff-gz` same as `bin-diff-raw`, but assumes the tiles are
gzip-compressed, so it uncompresses them before comparing. The `xxh3_64`
stores the hash of the uncompressed tile. The data will be stored in the
`bsdiffrawgz` table (identical structure with above)
* `mbtiles copy --apply-patch` will automatically detect if
`bsdiffrawgz` or `bsdiffraw` tables exist, and will use binary patching.
* `mbtiles apply-patch` does not support binary patching yet
* `mbtiles copy --diff-with-file ... --patch-type ...` is an alias to
`mbtiles diff --patch-type ...`
* Some small test cleanups
* A bump to dependencies
* Disable sprite generation test - PNGs tests are too flaky
* Add an mbtiles copy test for empty diffs
This just adds a test for the unusual case of a table with quotes,
spaces, and dots in their identifiers. Another similar test should be
added for functions.
Implement `mbtiles diff a.mbtiles b.mbtiles diff.mbtiles` command
This should behave exactly the same as `mbtiles copy a.mbtiles
--diff-with-file b.mbtiles diff.mbtiles`.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
This adds the lambda-web crate to adapt the actix App to speak to Lambda
by way of the lambda_runtime crate.
AWS Lambda has native support for scripting languages to
execute a function directly; compiled languages must embed a runtime to
fetch incoming events from Lambda and post the responses. This detects
the environment variables to start up in Lambda mode instead of the
normal HTTP server, and is added as an optional feature.
Lambda has five (!) distinct ways of routing HTTP requests to a
function; this supports some of them. (Specifically, the most obvious
way to do this is with a Function URL, which is newest and simplest, and
perhaps with CloudFront, which speaks to the Function URL and not Lambda
directly.)
The error handling could probably be refined, I was just trying to get
this to compile.
(Supported: API Gateway HTTP API with payload format version 2.0; API
Gateway REST API; Lambda function URLs / Not supported: API Gateway HTTP
API with payload format version 1.0; Application Load Balancer)
Necessary for #1102 to be able to run the released packages directly,
and only having to configure the appropriate environment.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
Move a few functions around for consistency, renamed files. No actual
functionality changes. Some logging output would produce slightly
different results because of the module name changes.
Add a top level config parameter -- the size of cache memory (in MB) to
use for caching tiles and PMT directories, defaulting to 512, and 0 to
disable. This also removes the `pmtiles.dir_cache_size_mb` parameter (it
will be ignored, but will give a warning)
```
cache_size_mb: 512
```
The new cache will contain all tiles as provided by the source. So if
PostgreSQL returns a non-compressed tile, the cache will contain the
uncompressed variant, and will be compressed for each response. This
will be fixed in the later releases.
Note that fonts and sprites are not cached at this time, and are still a
TODO.
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
* **postgres** - enable PostgreSQL/PostGIS tile sources
* **pmtiles** - enable PMTile tile sources
* **mbtiles** - enable MBTile tile sources
* **fonts** - enable font sources
* **sprites** - enable sprite sources
* Use justfile in the CI
Fixes#1093
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.
PMTiles config now also allows `http` and `https` protocol.
```
# Publish PMTiles files
pmtiles:
paths:
# specific pmtiles file will be published as mypmtiles source
# (use last portion of the URL without extension)
- http://example.org/path/to/mypmtiles.pmtiles
sources:
# named source matching source name to a single file
pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
```
fixes#884
---------
Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
* BREAKING: `martin-cp` will now set `format=pbf` instead of `mvt`. This
is what QGIS and possibly others expect, and this is what tools like
tilelive generates.
* `martin-cp` sets `minzoom` and `maxzoom` metadata values based on the
zoom parameters
* Add `mbtiles meta-update` command to refresh zoom levels based on the
present tiles.
Partially addresses items in #1081
Figure out and fix the bug with `tile-grid` crate or remove the crate
and implement it ourself.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
Make sure all values in the `tiles` table or view are correct:
* zoom_level is between 0 and 30 (max allowed zoom)
* the x,y values are within the limits for the corresponding zoom level
* the column type of z,x,y are all integers
* the `tile_data` is a NULL or a BLOB
Mathematics should be consolidated in the utils crate. Eventually, I
hope we will move it to [tile-grid](https://crates.io/crates/tile-grid)
or [utiles](https://crates.io/crates/utiles) crates and consolidate our
efforts there.
Also, modify earth circumference by a few centimeters (to match various
other implementations): from `.7` to `.685_578_5`. This modified a few
unit tests.
* Now by default mbtiles will not print internal warnings (e.g. that
something is taking too long to execute)
* remove mbtiles --help test - not really needed, but adds complexity
because it has to be updated every time a version changes.
When a SQL comment is set on a table or a function to customize
tilejson, and that tbl/func is pre-configured as part of the config
file, the comment was silently ignored. Now both table and function
cases are handled correctly.
Also, update docs to not include function parameters - makes SQL example
a bit simpler.
Thanks @jjcfrancisco for reporting!
Fixes: #1044
* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.
`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files
# Other
* Always sort JSON-serialized keys for consistency
* this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
Partial fix of #1002
* [x] Move all summary code from `mbtiles/src/mbtiles.rs` to
`mbtiles/src/summary.rs`
* [x] Move Metadata and Validation function to separate files
* [x] Remove `filename` String from the summary stats - not much point
because it is accessible from the mbtiles struct itself
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
After some thinking, it seems `mbtiles summary` (aliased as `mbtiles
info`) would be a bit better than stats. I renamed and adjusted
documentation, consolidating it in one doc page.
Other changes:
* use file system's file size, reporting 'unknown' if needed
* report page count
* moved bbox computation into a separate function
* inlined a number of things for readability
This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles
This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.
### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
* convert them to SDFs and package them into a protobuf
---------
Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
* Remove `--disable-bounds` flag and `disable_bounds` config parameters.
* Add `--auto-bounds` / `-b` CLI parameter and `auto_bounds` config
value:
* `quick`: Compute table geometry bounds, but abort if it takes longer
than 5 seconds (default)
* `calc`: Compute table geometry bounds. The startup time may be
significant. Make sure all GEO columns have indexes
* `skip`: Skip bounds calculation. The bounds will be set to the whole
world
* `-b` is now mapped to `--auto-bounds` param, but it will fail if used
by itself because it now requires a value.
Fixes#955
