Commit Graph

174 Commits

Author SHA1 Message Date
Frank Elsinga
25f6dd8fed
chore: snapshots on sorted test output (#1615)
Some checks are pending
Documentation / Build Docs (push) Waiting to run
CI / Lint and Unit test (push) Waiting to run
CI / Build and test docker images (push) Waiting to run
CI / Build ${{ matrix.target }} (.exe, windows-latest, x86_64-pc-windows-msvc) (push) Waiting to run
CI / Build ${{ matrix.target }} (macos-13, x86_64-apple-darwin) (push) Waiting to run
CI / Build ${{ matrix.target }} (macos-latest, aarch64-apple-darwin) (push) Waiting to run
CI / Build ${{ matrix.target }} (ubuntu-latest, debian-x86_64) (push) Waiting to run
CI / Build ${{ matrix.target }} (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Waiting to run
CI / Test AWS Lambda (push) Blocked by required conditions
CI / Test on ${{ matrix.os }} (.exe, windows-latest, x86_64-pc-windows-msvc) (push) Blocked by required conditions
CI / Test on ${{ matrix.os }} (macos-13, x86_64-apple-darwin) (push) Blocked by required conditions
CI / Test on ${{ matrix.os }} (ubuntu-latest, x86_64-unknown-linux-gnu) (push) Blocked by required conditions
CI / Test postgis:${{ matrix.img_ver }} sslmode=${{ matrix.sslmode }} (postgres -c ssl=on -c ssl_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem -c ssl_key_file=/etc/ssl/private/ssl-cert-snakeoil.key, 15-3.3, require) (push) Blocked by required conditions
CI / Test postgis:${{ matrix.img_ver }} sslmode=${{ matrix.sslmode }} (postgres, 11-3.0-alpine, disable) (push) Blocked by required conditions
CI / Test postgis:${{ matrix.img_ver }} sslmode=${{ matrix.sslmode }} (postgres, 14-3.3-alpine, disable) (push) Blocked by required conditions
CI / Package (push) Blocked by required conditions
CI / CI Finished (push) Blocked by required conditions
Demo / Demo (push) Waiting to run
Code coverage / grcov (push) Waiting to run
This PR splits out the test reproducibility fix from #1585

Currently, these fields not being sorted does not pose an issue, but
with #1585, it would at least be differnt
=> we likely should not rely upon this in our CI
2024-12-15 16:19:17 +01:00
Frank Elsinga
b134cb2bc6
Add sdf sprite support with /sdf_sprite/... endpoint (#1492)
Given a set of SVGs as a sprite source X, Martin will now generate two endpoints: `/sprite/X` and `/sdf_sprite/X`, with the second endpoint serving [signed distance field](https://en.wikipedia.org/wiki/Signed_distance_function) images.

Closes #1075
2024-10-22 12:49:36 -04:00
Lucas
6f201fcf69
Use layer id if it is provided (#1501)
Try to fix #1472

---------

Co-authored-by: zhangyijun <270361049@qq.com>
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2024-10-20 02:28:39 -04:00
Frank Elsinga
8ac8ef5f23
feat:added a postgres version check (#1427)
- [x] adds a postgres version check
- [x] added a testcase (using testcontainers) that the version is parsed
correctly

Resolves #1230 (I think at this point the PR can be considered unasigned
again, feel free to reject otherwise. Don't want to step on peoples toes
^^)
Resolves #1229 (by using the minumum supported postgres version for
postgis)

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2024-10-20 02:07:54 -04:00
Lucas
48cce661d5
Use comment on view to generate TileJson (#1507)
Try to fix #1494 
- [x] Update
[query_available_tables.sql](https://github.com/maplibre/martin/pull/1507/files#diff-449496b294ee2413fc0d5e4290688a50f008e07fece5a613da1c2417548a06bf)
- [x] Add and update test
2024-10-20 00:00:46 -04:00
Yuri Astrakhan
a3222b1da4
cleanup test.sh, fix json rounding issue (#1543)
Fixes #1532
2024-10-19 22:13:49 -04:00
Yuri Astrakhan
81153f38af
Update dependencies, tests, connection limit (#1530) 2024-10-01 00:10:06 -04:00
Yuri Astrakhan
7446f54bc1
Minor lints (#1412) 2024-07-11 10:04:58 -04:00
Yuri Astrakhan
b71c846abc
fix for mbtiles apply-patch with raw bindiff (#1384)
When applying raw (no gzip) patches, `mbtiles` was trying to un-gzip
them first. Now handles it properly.  Also adds a number of tests to catch these cases.
2024-06-26 20:37:20 +00:00
Yuri Astrakhan
ccd550a2cf
Add binary diff for mbtiles (#1358)
* `mbtiles diff` now has an additional `--patch-type` param with
`whole`, `bin-diff-raw` and `bin-diff-gz` values:
* `whole` stores different tiles as before - as whole tiles in the
`tiles` table
* `bin-diff-raw` computes binary difference between tiles, and stores
them as brotli-encoded value in a `bsdiffraw` table, together with a
`xxh3_64` hash of the tile as it will be stored after patching
* `bin-diff-gz` same as `bin-diff-raw`, but assumes the tiles are
gzip-compressed, so it uncompresses them before comparing. The `xxh3_64`
stores the hash of the uncompressed tile. The data will be stored in the
`bsdiffrawgz` table (identical structure with above)

* `mbtiles copy --apply-patch` will automatically detect if
`bsdiffrawgz` or `bsdiffraw` tables exist, and will use binary patching.
* `mbtiles apply-patch` does not support binary patching yet
* `mbtiles copy --diff-with-file ... --patch-type ...` is an alias to
`mbtiles diff --patch-type ...`
2024-06-25 19:34:14 +00:00
Yuri Astrakhan
6320e0fff3
Add agg_tiles_hash_before_apply, warnings, and validate on patch (#1266)
Implement #1244
2024-05-30 18:28:34 +00:00
Yuri Astrakhan
d77533f627
Fix CI: use new nginx and postgis actions (#1351)
I recently released a new nginx github action that should work on all OS
and in all runners. Also, postgis action has been updated.
2024-05-24 22:41:26 -04:00
Yuri Astrakhan
b28dae4466
chore: add empty pg table test (#1259) 2024-03-16 22:45:43 -04:00
Yuri Astrakhan
9505a2107e
chore: increase complexity of the SQL table test (#1256)
a few more characters that have to be properly escaped
2024-03-16 04:23:35 -04:00
Yuri Astrakhan
1c95c60ebd
Update lock, insta version deprecation, some test changes (#1240)
* Some small test cleanups
* A bump to dependencies
* Disable sprite generation test - PNGs tests are too flaky
* Add an mbtiles copy test for empty diffs
2024-03-11 01:14:27 -04:00
Lucas
656728cde1
Add test for function with special character (#1231) 2024-03-09 08:30:57 +08:00
Yuri Astrakhan
c0ec365d5c
Added a test for schema.table.column containing special chars (#1222)
This just adds a test for the unusual case of a table with quotes,
spaces, and dots in their identifiers. Another similar test should be
added for functions.
2024-03-04 02:55:21 -05:00
Yuri Astrakhan
fba9af8749
Bump locks, blessed expected tests (#1212) 2024-02-25 05:46:59 -05:00
Lucas
79a89125ed
Add mbtiles diff command (#1068)
Implement `mbtiles diff a.mbtiles b.mbtiles diff.mbtiles` command

This should behave exactly the same as `mbtiles copy a.mbtiles
--diff-with-file b.mbtiles diff.mbtiles`.

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2024-02-08 03:08:47 +00:00
Josh Lee
9f9f18163c
Add AWS Lambda support (#1127)
This adds the lambda-web crate to adapt the actix App to speak to Lambda
by way of the lambda_runtime crate.

AWS Lambda has native support for scripting languages to
execute a function directly; compiled languages must embed a runtime to
fetch incoming events from Lambda and post the responses. This detects
the environment variables to start up in Lambda mode instead of the
normal HTTP server, and is added as an optional feature.

Lambda has five (!) distinct ways of routing HTTP requests to a
function; this supports some of them. (Specifically, the most obvious
way to do this is with a Function URL, which is newest and simplest, and
perhaps with CloudFront, which speaks to the Function URL and not Lambda
directly.)

The error handling could probably be refined, I was just trying to get
this to compile.

(Supported: API Gateway HTTP API with payload format version 2.0; API
Gateway REST API; Lambda function URLs / Not supported: API Gateway HTTP
API with payload format version 1.0; Application Load Balancer)

Necessary for #1102 to be able to run the released packages directly,
and only having to configure the appropriate environment.

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2024-02-01 18:56:40 +00:00
Yuri Astrakhan
955b8da9c6
chore: some refactoring of postgres code (#1166)
Move a few functions around for consistency, renamed files. No actual
functionality changes. Some logging output would produce slightly
different results because of the module name changes.
2024-02-01 09:17:32 +08:00
Yuri Astrakhan
3dc54d7f9e
Implement tile caching (#1105)
Add a top level config parameter -- the size of cache memory (in MB) to
use for caching tiles and PMT directories, defaulting to 512, and 0 to
disable. This also removes the `pmtiles.dir_cache_size_mb` parameter (it
will be ignored, but will give a warning)

```
cache_size_mb: 512
```

The new cache will contain all tiles as provided by the source. So if
PostgreSQL returns a non-compressed tile, the cache will contain the
uncompressed variant, and will be compressed for each response. This
will be fixed in the later releases.

Note that fonts and sprites are not cached at this time, and are still a
TODO.
2023-12-30 22:48:23 -05:00
Yuri Astrakhan
2def6288f1
PMTiles cache, refactor file configs, modularize (#1094)
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
    * **postgres** - enable PostgreSQL/PostGIS tile sources
    * **pmtiles** - enable PMTile tile sources
    * **mbtiles** - enable MBTile tile sources
    * **fonts** - enable font sources
    * **sprites** - enable sprite sources
* Use justfile in the CI

Fixes #1093
2023-12-25 05:52:04 +00:00
Yuri Astrakhan
1a8e7c89a4
Implement http pmtiles (#991)
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.

PMTiles config now also allows `http` and `https` protocol.

```
# Publish PMTiles files
pmtiles:
  paths:
    # specific pmtiles file will be published as mypmtiles source
    # (use last portion of the URL without extension)
    - http://example.org/path/to/mypmtiles.pmtiles
  sources:
    # named source matching source name to a single file
    pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
 ```

fixes #884

---------

Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
2023-12-22 06:01:50 +00:00
Yuri Astrakhan
24819bdb45
Multiple mbtiles and martin-cp fixes (#1083)
* BREAKING: `martin-cp` will now set `format=pbf` instead of `mvt`. This
is what QGIS and possibly others expect, and this is what tools like
tilelive generates.
* `martin-cp` sets `minzoom` and `maxzoom` metadata values based on the
zoom parameters
* Add `mbtiles meta-update` command to refresh zoom levels based on the
present tiles.

Partially addresses items in #1081
2023-12-18 23:19:36 -05:00
Yuri Astrakhan
5006641588
Make martin-cp integration tests ignore gen version (#1082)
This avoids blessing test results after version upgrade
2023-12-18 21:20:36 -05:00
Yuri Astrakhan
f4c52d027e CI fixes 2023-12-18 20:55:08 -05:00
Lucas
f21119d548
Fix bbox_to_xyz (#1070)
Figure out and fix the bug with `tile-grid` crate or remove the crate
and implement it ourself.

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-12-18 00:02:49 -05:00
Yuri Astrakhan
886358e93f
Fix bbox to xyz roundtrip computation (#1059)
Add `tile-grid` as a dependency to use its webmercator tile math.

---------

Co-authored-by: sharkAndshark <zhangyijunmetro@hotmail.com>
2023-12-15 06:11:31 +00:00
Yuri Astrakhan
54f5a8daf0
Add tiles table/view validation to mbtiles validate (#1057)
Make sure all values in the `tiles` table or view are correct:

* zoom_level is between 0 and 30 (max allowed zoom)
* the x,y values are within the limits for the corresponding zoom level
* the column type of z,x,y are all integers
* the `tile_data` is a NULL or a BLOB
2023-12-10 10:59:14 +08:00
Yuri Astrakhan
280dbe162f
Move math utils to martin-tile-utils (#1056)
Mathematics should be consolidated in the utils crate. Eventually, I
hope we will move it to [tile-grid](https://crates.io/crates/tile-grid)
or [utiles](https://crates.io/crates/utiles) crates and consolidate our
efforts there.

Also, modify earth circumference by a few centimeters (to match various
other implementations): from `.7` to `.685_578_5`. This modified a few
unit tests.
2023-12-09 04:38:15 +00:00
Yuri Astrakhan
aeaa1e070b
Add extra CLI help string about RUST_LOG (#1053)
Improve CLI help screen
2023-12-08 16:45:37 +08:00
Yuri Astrakhan
7ab3f5e21a
Clean up mbtiles tool output (#1048)
* Now by default mbtiles will not print internal warnings (e.g. that
something is taking too long to execute)
* remove mbtiles --help test - not really needed, but adds complexity
because it has to be updated every time a version changes.
2023-12-08 08:15:31 +08:00
Yuri Astrakhan
f88db05582
Fix SQL comments when func or table is pre-configured (#1045)
When a SQL comment is set on a table or a function to customize
tilejson, and that tbl/func is pre-configured as part of the config
file, the comment was silently ignored. Now both table and function
cases are handled correctly.

Also, update docs to not include function parameters - makes SQL example
a bit simpler.

Thanks @jjcfrancisco for reporting!

Fixes: #1044
2023-12-07 07:24:51 +00:00
Yuri Astrakhan
d66e7ab7a7
feat: martin-cp can set metadata values (#1038)
Use `martin-cp --set-meta KEY=VALUE ...` to modify metadata values after
the copying is complete. Can be used multiple times.
2023-12-06 05:41:32 +00:00
Yuri Astrakhan
aeb08c8b22
add agg-tiles-hash calc to martin-cp (#1033)
* Automatically computes `agg-tiles-hash` metadata value after copying
tiles
* Add a `--skip-agg-tiles-hash` option
2023-12-02 09:12:19 +08:00
Yuri Astrakhan
713a51b74d
Upgarde to v0.10 spreet crate, cleanup SpreetResult (#984) 2023-11-29 19:09:05 -05:00
Yuri Astrakhan
e4c31384dc Minor help cleanups 2023-11-22 02:04:32 -05:00
Yuri Astrakhan
d6219a6526
Add compression and query to martin-cp (#1019)
* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
2023-11-22 05:26:53 +00:00
Yuri Astrakhan
0f2cd100cf
Rename MartinError, benchmarks, streamline get_tile (#1016)
Use MartinError, MartinResult, FileError and FileResult, and other similar enums.
2023-11-21 18:15:12 -05:00
Yuri Astrakhan
c6a7ef67c9
Rename dst-type to mbtiles-type (#1014) 2023-11-20 17:38:28 -05:00
Yuri Astrakhan
999179029b Prepare for release 2023-11-20 01:51:08 -05:00
Yuri Astrakhan
f88fd10aa8
Add martin-cp tool to bulk-copy tiles from any Martin-supported sources (#1011)
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.

`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
2023-11-20 06:27:51 +00:00
Yuri Astrakhan
140ed25679
mbtiles improvements, minor changes (#1013)
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files

# Other
* Always sort JSON-serialized keys for consistency
  * this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
2023-11-19 10:06:37 +00:00
Lucas
e72f53d9ef
Minor mbtiles summary improvements (#1004)
Partial fix of #1002 

* [x] Move all summary code from `mbtiles/src/mbtiles.rs` to
`mbtiles/src/summary.rs`
* [x] Move Metadata and Validation function to separate files
* [x] Remove `filename` String from the summary stats - not much point
because it is accessible from the mbtiles struct itself

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-11-16 18:36:40 -05:00
Yuri Astrakhan
b0e19d3787
Improve bbox formatting (#1001)
Formats bbox using the most appropriate precision for that zoom level
2023-11-13 08:53:15 +00:00
Yuri Astrakhan
021cddcccd
mbtiles summary tool cleanup (#1000)
After some thinking, it seems `mbtiles summary` (aliased as `mbtiles
info`) would be a bit better than stats. I renamed and adjusted
documentation, consolidating it in one doc page.

Other changes:
* use file system's file size, reporting 'unknown' if needed
* report page count
* moved bbox computation into a separate function
* inlined a number of things for readability
2023-11-13 07:50:10 +00:00
Lucas
0398336114
Add mbtiles statistics command (#986)
- [x] Add tile statistics to mbtiles tools
- [x] Add test
- [x] Use 4326 instead of 3857 for tile bounds
- [x] Add document
- [x] Use size-format to prettify output 
- [x]  Statistics  struct Refactor
- [x] Cleanup and reformat

Closes #964 

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-11-13 06:03:40 +00:00
Yuri Astrakhan
dec09bdabf
Do not serialize missing configs (#999)
Make sure minimum config does not save unused sub-systems.
2023-11-13 04:40:34 +00:00
Yuri Astrakhan
f7c69f8302
Minor cleanup and just reorg (#992)
Only run git push diff on linux to make development on non-linux easier,
and a few preparations for pmtiles over http serving

Fixes #855
2023-11-08 19:46:23 -05:00