Add an optional web UI interface for Martin, including docker-based cross-compilation support. The UI itself is a placeholder with a logo, but will grow in subsequent PRs.
This was branched off of https://github.com/maplibre/martin/pull/1142 to
address the PR feedback from @nyurik .
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
Co-authored-by: Tomer Ronen <tomer207@gmail.com>
Co-authored-by: tomeronen <45331634+tomeronen@users.noreply.github.com>
* `mbtiles diff` now has an additional `--patch-type` param with
`whole`, `bin-diff-raw` and `bin-diff-gz` values:
* `whole` stores different tiles as before - as whole tiles in the
`tiles` table
* `bin-diff-raw` computes binary difference between tiles, and stores
them as brotli-encoded value in a `bsdiffraw` table, together with a
`xxh3_64` hash of the tile as it will be stored after patching
* `bin-diff-gz` same as `bin-diff-raw`, but assumes the tiles are
gzip-compressed, so it uncompresses them before comparing. The `xxh3_64`
stores the hash of the uncompressed tile. The data will be stored in the
`bsdiffrawgz` table (identical structure with above)
* `mbtiles copy --apply-patch` will automatically detect if
`bsdiffrawgz` or `bsdiffraw` tables exist, and will use binary patching.
* `mbtiles apply-patch` does not support binary patching yet
* `mbtiles copy --diff-with-file ... --patch-type ...` is an alias to
`mbtiles diff --patch-type ...`
This adds the lambda-web crate to adapt the actix App to speak to Lambda
by way of the lambda_runtime crate.
AWS Lambda has native support for scripting languages to
execute a function directly; compiled languages must embed a runtime to
fetch incoming events from Lambda and post the responses. This detects
the environment variables to start up in Lambda mode instead of the
normal HTTP server, and is added as an optional feature.
Lambda has five (!) distinct ways of routing HTTP requests to a
function; this supports some of them. (Specifically, the most obvious
way to do this is with a Function URL, which is newest and simplest, and
perhaps with CloudFront, which speaks to the Function URL and not Lambda
directly.)
The error handling could probably be refined, I was just trying to get
this to compile.
(Supported: API Gateway HTTP API with payload format version 2.0; API
Gateway REST API; Lambda function URLs / Not supported: API Gateway HTTP
API with payload format version 1.0; Application Load Balancer)
Necessary for #1102 to be able to run the released packages directly,
and only having to configure the appropriate environment.
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
This should solve #1149. See also [my comment there for the
approach](https://github.com/maplibre/martin/issues/11). In short
through a variable that is substituted. Note that this may not work if
an older Docker version is installed without the Python `docker-compose`
script...As both variants will not be present. Maybe add an extra
test+warning msg for that case.
```
dockercompose := `if docker-compose --version &> /dev/null; then echo "docker-compose"; else echo "docker compose"; fi`
.
.
up:
{{ dockercompose }} up
up-build:
{{ dockercompose }} up --build
.
.
```
---------
Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
* Use `Option<&T>` instead of `&Option<T>` in function arguments.
* Cleaner var names
* Slight optimization of `get_tile` with query params
* Split up srv/server.rs into fonts, sprites, tiles, and tiles_info
files
* better error reporting in tests
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
* **postgres** - enable PostgreSQL/PostGIS tile sources
* **pmtiles** - enable PMTile tile sources
* **mbtiles** - enable MBTile tile sources
* **fonts** - enable font sources
* **sprites** - enable sprite sources
* Use justfile in the CI
Fixes#1093
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.
PMTiles config now also allows `http` and `https` protocol.
```
# Publish PMTiles files
pmtiles:
paths:
# specific pmtiles file will be published as mypmtiles source
# (use last portion of the URL without extension)
- http://example.org/path/to/mypmtiles.pmtiles
sources:
# named source matching source name to a single file
pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
```
fixes#884
---------
Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
When a SQL comment is set on a table or a function to customize
tilejson, and that tbl/func is pre-configured as part of the config
file, the comment was silently ignored. Now both table and function
cases are handled correctly.
Also, update docs to not include function parameters - makes SQL example
a bit simpler.
Thanks @jjcfrancisco for reporting!
Fixes: #1044
* Fix docker file builds to use debian
* add a few more things to `.dockerignore`
* remove unused `martin` from docker-compose.yml
* delete unused arm64.Dockerfile
This should also fix#1021
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.
`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.
## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content
There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).
## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
Resolves#682
- [x] Get id_column string from config.yaml and use for id column
- [x] Support for list of strings
- [x] Add info/warnings if column is not there or is of wrong type
- [x] if column for the feature ID is found, remove it from properties
(see inline comment)
- [x] cleanup logging messages
- [x] need more tests to catch other edge cases
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
* on `--save-config`, only save configured `auto_publish` settings
* alias `from_schemas` as `from_schema`
* add integration testing for `auto_publish`
* if integration test DB preloading fails, try to clean up the test DB
* A few more info traces
This change should benefit testing of the #790 cc: @Binabh
Add deb package by [cargo-deb](https://crates.io/crates/cargo-deb)
- [ ] add deb package metadata
- [ ] more research on systemd integration
To inspect the package:
```shell
# install cargo-deb dependices
sudo apt install dpkg
sudo apt install dpkg-dev
sudo apt install liblzma-dev
cargo install cargo-deb
cd martin
cargo deb
The Debian package will be created in `target/debian/martin_0.8.7_amd64.deb` .
```
To decompress this package:
```shell
dpkg -x martin_0.8.7_amd64.deb path_of_decompress
```
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
* Use same docker image for postgis & psql
* Improve docker book
* rename just target `mdbook` to `book` (more obvious choice)
* Cleanup justfile to not duplicate "cargo install" check & installation
This is mostly a noop, just updating to the new way of storing cached
queries and a few other changes per
https://github.com/launchbadge/sqlx/blob/main/CHANGELOG.md#070---2023-06-30
Note that this uses some of the code from #729 (thanks @upsicleclown !)
A minor unrelated change - a reformat of `martin/release.toml`
I'd like to add optional rustls support for SQLx. This PR implements the
required features, but tests using `--all-features` fail, because
runtime selection features are not additive. I didn't find a solution
apart from avoiding `--all-features`.
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
