Implement `mbtiles diff a.mbtiles b.mbtiles diff.mbtiles` command
This should behave exactly the same as `mbtiles copy a.mbtiles
--diff-with-file b.mbtiles diff.mbtiles`.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
This adds the lambda-web crate to adapt the actix App to speak to Lambda
by way of the lambda_runtime crate.
AWS Lambda has native support for scripting languages to
execute a function directly; compiled languages must embed a runtime to
fetch incoming events from Lambda and post the responses. This detects
the environment variables to start up in Lambda mode instead of the
normal HTTP server, and is added as an optional feature.
Lambda has five (!) distinct ways of routing HTTP requests to a
function; this supports some of them. (Specifically, the most obvious
way to do this is with a Function URL, which is newest and simplest, and
perhaps with CloudFront, which speaks to the Function URL and not Lambda
directly.)
The error handling could probably be refined, I was just trying to get
this to compile.
(Supported: API Gateway HTTP API with payload format version 2.0; API
Gateway REST API; Lambda function URLs / Not supported: API Gateway HTTP
API with payload format version 1.0; Application Load Balancer)
Necessary for #1102 to be able to run the released packages directly,
and only having to configure the appropriate environment.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
Add a top level config parameter -- the size of cache memory (in MB) to
use for caching tiles and PMT directories, defaulting to 512, and 0 to
disable. This also removes the `pmtiles.dir_cache_size_mb` parameter (it
will be ignored, but will give a warning)
```
cache_size_mb: 512
```
The new cache will contain all tiles as provided by the source. So if
PostgreSQL returns a non-compressed tile, the cache will contain the
uncompressed variant, and will be compressed for each response. This
will be fixed in the later releases.
Note that fonts and sprites are not cached at this time, and are still a
TODO.
* Remove `CA_ROOT_FILE` env var support (has been deprecated for a
while)
* Remove `WATCH_MODE` env var warning - has not been in use since a long
time ago
* minor lints/clippy suggestions, lock update
* Use `Option<&T>` instead of `&Option<T>` in function arguments.
* Cleaner var names
* Slight optimization of `get_tile` with query params
* Split up srv/server.rs into fonts, sprites, tiles, and tiles_info
files
* better error reporting in tests
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
* **postgres** - enable PostgreSQL/PostGIS tile sources
* **pmtiles** - enable PMTile tile sources
* **mbtiles** - enable MBTile tile sources
* **fonts** - enable font sources
* **sprites** - enable sprite sources
* Use justfile in the CI
Fixes#1093
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.
PMTiles config now also allows `http` and `https` protocol.
```
# Publish PMTiles files
pmtiles:
paths:
# specific pmtiles file will be published as mypmtiles source
# (use last portion of the URL without extension)
- http://example.org/path/to/mypmtiles.pmtiles
sources:
# named source matching source name to a single file
pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
```
fixes#884
---------
Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
When a SQL comment is set on a table or a function to customize
tilejson, and that tbl/func is pre-configured as part of the config
file, the comment was silently ignored. Now both table and function
cases are handled correctly.
Also, update docs to not include function parameters - makes SQL example
a bit simpler.
Thanks @jjcfrancisco for reporting!
Fixes: #1044
* Fix docker file builds to use debian
* add a few more things to `.dockerignore`
* remove unused `martin` from docker-compose.yml
* delete unused arm64.Dockerfile
This should also fix#1021
* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.
`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files
# Other
* Always sort JSON-serialized keys for consistency
* this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
After some thinking, it seems `mbtiles summary` (aliased as `mbtiles
info`) would be a bit better than stats. I renamed and adjusted
documentation, consolidating it in one doc page.
Other changes:
* use file system's file size, reporting 'unknown' if needed
* report page count
* moved bbox computation into a separate function
* inlined a number of things for readability
This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles
This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.
### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
* convert them to SDFs and package them into a protobuf
---------
Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
* Remove `--disable-bounds` flag and `disable_bounds` config parameters.
* Add `--auto-bounds` / `-b` CLI parameter and `auto_bounds` config
value:
* `quick`: Compute table geometry bounds, but abort if it takes longer
than 5 seconds (default)
* `calc`: Compute table geometry bounds. The startup time may be
significant. Make sure all GEO columns have indexes
* `skip`: Skip bounds calculation. The bounds will be set to the whole
world
* `-b` is now mapped to `--auto-bounds` param, but it will fail if used
by itself because it now requires a value.
Fixes#955
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.
## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content
There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).
## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
* Do not open the same mbtiles file more than once at the same time:
reuse the (unsafe) sqlite handle to create a `RusqliteConnection`
instead.
* The copying should set `agg_tiles_hash` in all cases because now it
uses the always available `tiles` table/view.
* a few minor cleanups and renames related to that
`postgres.auto_publish.tables` now lets users customize `clip_geom`, `buffer`, and `extent` instead of using their default values.
See also #872
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
Modify `/catalog` endpoint to return an object instead of a list. This
allows future expansion of the catalog schema, e.g. adding new types of
data.
The new schema:
```yaml
{
"tiles" {
"function_zxy_query": {
"name": "public.function_zxy_query",
"content_type": "application/x-protobuf"
},
"points1": {
"name": "public.points1.geom",
"content_type": "image/webp"
},
...
},
}
```
