Commit Graph

152 Commits

Author SHA1 Message Date
Yuri Astrakhan
2def6288f1
PMTiles cache, refactor file configs, modularize (#1094)
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
    * **postgres** - enable PostgreSQL/PostGIS tile sources
    * **pmtiles** - enable PMTile tile sources
    * **mbtiles** - enable MBTile tile sources
    * **fonts** - enable font sources
    * **sprites** - enable sprite sources
* Use justfile in the CI

Fixes #1093
2023-12-25 05:52:04 +00:00
Yuri Astrakhan
1a8e7c89a4
Implement http pmtiles (#991)
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.

PMTiles config now also allows `http` and `https` protocol.

```
# Publish PMTiles files
pmtiles:
  paths:
    # specific pmtiles file will be published as mypmtiles source
    # (use last portion of the URL without extension)
    - http://example.org/path/to/mypmtiles.pmtiles
  sources:
    # named source matching source name to a single file
    pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
 ```

fixes #884

---------

Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
2023-12-22 06:01:50 +00:00
Yuri Astrakhan
24819bdb45
Multiple mbtiles and martin-cp fixes (#1083)
* BREAKING: `martin-cp` will now set `format=pbf` instead of `mvt`. This
is what QGIS and possibly others expect, and this is what tools like
tilelive generates.
* `martin-cp` sets `minzoom` and `maxzoom` metadata values based on the
zoom parameters
* Add `mbtiles meta-update` command to refresh zoom levels based on the
present tiles.

Partially addresses items in #1081
2023-12-18 23:19:36 -05:00
Yuri Astrakhan
5006641588
Make martin-cp integration tests ignore gen version (#1082)
This avoids blessing test results after version upgrade
2023-12-18 21:20:36 -05:00
Yuri Astrakhan
f4c52d027e CI fixes 2023-12-18 20:55:08 -05:00
Lucas
f21119d548
Fix bbox_to_xyz (#1070)
Figure out and fix the bug with `tile-grid` crate or remove the crate
and implement it ourself.

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-12-18 00:02:49 -05:00
Yuri Astrakhan
886358e93f
Fix bbox to xyz roundtrip computation (#1059)
Add `tile-grid` as a dependency to use its webmercator tile math.

---------

Co-authored-by: sharkAndshark <zhangyijunmetro@hotmail.com>
2023-12-15 06:11:31 +00:00
Yuri Astrakhan
54f5a8daf0
Add tiles table/view validation to mbtiles validate (#1057)
Make sure all values in the `tiles` table or view are correct:

* zoom_level is between 0 and 30 (max allowed zoom)
* the x,y values are within the limits for the corresponding zoom level
* the column type of z,x,y are all integers
* the `tile_data` is a NULL or a BLOB
2023-12-10 10:59:14 +08:00
Yuri Astrakhan
280dbe162f
Move math utils to martin-tile-utils (#1056)
Mathematics should be consolidated in the utils crate. Eventually, I
hope we will move it to [tile-grid](https://crates.io/crates/tile-grid)
or [utiles](https://crates.io/crates/utiles) crates and consolidate our
efforts there.

Also, modify earth circumference by a few centimeters (to match various
other implementations): from `.7` to `.685_578_5`. This modified a few
unit tests.
2023-12-09 04:38:15 +00:00
Yuri Astrakhan
aeaa1e070b
Add extra CLI help string about RUST_LOG (#1053)
Improve CLI help screen
2023-12-08 16:45:37 +08:00
Yuri Astrakhan
7ab3f5e21a
Clean up mbtiles tool output (#1048)
* Now by default mbtiles will not print internal warnings (e.g. that
something is taking too long to execute)
* remove mbtiles --help test - not really needed, but adds complexity
because it has to be updated every time a version changes.
2023-12-08 08:15:31 +08:00
Yuri Astrakhan
f88db05582
Fix SQL comments when func or table is pre-configured (#1045)
When a SQL comment is set on a table or a function to customize
tilejson, and that tbl/func is pre-configured as part of the config
file, the comment was silently ignored. Now both table and function
cases are handled correctly.

Also, update docs to not include function parameters - makes SQL example
a bit simpler.

Thanks @jjcfrancisco for reporting!

Fixes: #1044
2023-12-07 07:24:51 +00:00
Yuri Astrakhan
d66e7ab7a7
feat: martin-cp can set metadata values (#1038)
Use `martin-cp --set-meta KEY=VALUE ...` to modify metadata values after
the copying is complete. Can be used multiple times.
2023-12-06 05:41:32 +00:00
Yuri Astrakhan
aeb08c8b22
add agg-tiles-hash calc to martin-cp (#1033)
* Automatically computes `agg-tiles-hash` metadata value after copying
tiles
* Add a `--skip-agg-tiles-hash` option
2023-12-02 09:12:19 +08:00
Yuri Astrakhan
713a51b74d
Upgarde to v0.10 spreet crate, cleanup SpreetResult (#984) 2023-11-29 19:09:05 -05:00
Yuri Astrakhan
e4c31384dc Minor help cleanups 2023-11-22 02:04:32 -05:00
Yuri Astrakhan
d6219a6526
Add compression and query to martin-cp (#1019)
* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
2023-11-22 05:26:53 +00:00
Yuri Astrakhan
0f2cd100cf
Rename MartinError, benchmarks, streamline get_tile (#1016)
Use MartinError, MartinResult, FileError and FileResult, and other similar enums.
2023-11-21 18:15:12 -05:00
Yuri Astrakhan
c6a7ef67c9
Rename dst-type to mbtiles-type (#1014) 2023-11-20 17:38:28 -05:00
Yuri Astrakhan
999179029b Prepare for release 2023-11-20 01:51:08 -05:00
Yuri Astrakhan
f88fd10aa8
Add martin-cp tool to bulk-copy tiles from any Martin-supported sources (#1011)
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.

`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
2023-11-20 06:27:51 +00:00
Yuri Astrakhan
140ed25679
mbtiles improvements, minor changes (#1013)
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files

# Other
* Always sort JSON-serialized keys for consistency
  * this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
2023-11-19 10:06:37 +00:00
Lucas
e72f53d9ef
Minor mbtiles summary improvements (#1004)
Partial fix of #1002 

* [x] Move all summary code from `mbtiles/src/mbtiles.rs` to
`mbtiles/src/summary.rs`
* [x] Move Metadata and Validation function to separate files
* [x] Remove `filename` String from the summary stats - not much point
because it is accessible from the mbtiles struct itself

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-11-16 18:36:40 -05:00
Yuri Astrakhan
b0e19d3787
Improve bbox formatting (#1001)
Formats bbox using the most appropriate precision for that zoom level
2023-11-13 08:53:15 +00:00
Yuri Astrakhan
021cddcccd
mbtiles summary tool cleanup (#1000)
After some thinking, it seems `mbtiles summary` (aliased as `mbtiles
info`) would be a bit better than stats. I renamed and adjusted
documentation, consolidating it in one doc page.

Other changes:
* use file system's file size, reporting 'unknown' if needed
* report page count
* moved bbox computation into a separate function
* inlined a number of things for readability
2023-11-13 07:50:10 +00:00
Lucas
0398336114
Add mbtiles statistics command (#986)
- [x] Add tile statistics to mbtiles tools
- [x] Add test
- [x] Use 4326 instead of 3857 for tile bounds
- [x] Add document
- [x] Use size-format to prettify output 
- [x]  Statistics  struct Refactor
- [x] Cleanup and reformat

Closes #964 

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-11-13 06:03:40 +00:00
Yuri Astrakhan
dec09bdabf
Do not serialize missing configs (#999)
Make sure minimum config does not save unused sub-systems.
2023-11-13 04:40:34 +00:00
Yuri Astrakhan
f7c69f8302
Minor cleanup and just reorg (#992)
Only run git push diff on linux to make development on non-linux easier,
and a few preparations for pmtiles over http serving

Fixes #855
2023-11-08 19:46:23 -05:00
Yuri Astrakhan
fe79756097 Bump lock, improve test.sh 2023-11-05 21:23:08 -05:00
Yuri Astrakhan
3a1f7acc95 fix image test extension 2023-10-30 21:52:11 -04:00
Yuri Astrakhan
47b3106ac6
Rename martin-mbtiles crate to mbtiles (#976)
This simplifies usage as both a crate and as a tool
2023-10-28 08:50:49 +00:00
Yuri Astrakhan
76a1b7fb33 fix CI 2023-10-28 04:01:48 -04:00
Yuri Astrakhan
0178ca731d better logging on failure 2023-10-28 02:06:37 -04:00
Yuri Astrakhan
9b112ae7b9
Implement dynamic font support /font/<name>/<start>-<end> (#755)
This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles

This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.

### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
  * convert them to SDFs and package them into a protobuf

---------

Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
2023-10-28 05:10:48 +00:00
Yuri Astrakhan
7aa169ed67
[BREAKING] New bounds calculation methods (#958)
* Remove `--disable-bounds` flag and `disable_bounds` config parameters.
* Add `--auto-bounds` / `-b` CLI parameter and `auto_bounds` config
value:
* `quick`: Compute table geometry bounds, but abort if it takes longer
than 5 seconds (default)
* `calc`: Compute table geometry bounds. The startup time may be
significant. Make sure all GEO columns have indexes
* `skip`: Skip bounds calculation. The bounds will be set to the whole
world
* `-b` is now mapped to `--auto-bounds` param, but it will fail if used
by itself because it now requires a value.

Fixes #955
2023-10-22 03:30:18 -04:00
Yuri Astrakhan
502413aecb
Fix martin integration tests (#952)
Turns out some integration tests were not running after `martin` was
moved to subdir
2023-10-20 11:40:08 +08:00
Yuri Astrakhan
c2a12a3799
Implement sprite catalog reporting (#951)
The `/catalog` now shows available sprites, which also paves way for the
future font support.

Lots of small refactorings to streamline tile source management. Now
each tile source can produce its own catalog entry, making the whole
thing much simpler.

Fixes #949
2023-10-19 06:13:52 +00:00
Yuri Astrakhan
a1f794ae4a
mbtiles: Add --apply-patch to copy, rename apply-diff (#945)
* `mbtiles apply-diff` is now `apply-patch` (old name is still
supported)
* `mbtiles copy` can now take `--apply-patch <file>` to apply the patch
while copying from source to destination. This way, the source file will
remain unmodified.
2023-10-16 12:40:44 -04:00
Yuri Astrakhan
8b34cd374c
Add metadat copy/apply-diff, new testing framework (#921)
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.

## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content

There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).

## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
2023-10-10 11:10:17 -04:00
Yuri Astrakhan
64d652f5af
Improve SSL mode testing (#913) 2023-09-30 22:49:56 -04:00
Yuri Astrakhan
ae8e0709d7 ssl dockercompose test config 2023-09-30 12:38:57 -04:00
Yuri Astrakhan
d1fe026639 Fix justfile ssl test, print hba config 2023-09-30 12:06:49 -04:00
Yuri Astrakhan
8d7204c53d
Switch to Rustls (#474)
Fixes #471
2023-09-30 17:58:49 +08:00
Yuri Astrakhan
6f08aa9465
Fix mbtiles validation, CI, and logging (#903) 2023-09-29 14:37:18 -04:00
Yuri Astrakhan
1a386b7071 Fix unit tests to use blobs only 2023-09-28 16:57:50 -04:00
Lucas
b2b3e2ce36
Add clip_geom, buffer and extent under auto_publish conf (#887)
`postgres.auto_publish.tables` now lets users customize  `clip_geom`, `buffer`, and `extent` instead of using their default values.

See also #872

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-09-21 10:18:09 -04:00
Lucas
fa1bf4bdbd
Retrieve table/view comments as tilejson (#871) 2023-09-12 12:59:59 +08:00
Yuri Astrakhan
6cd9296bfc
add mbtiles meta-all command (#865)
Print all MBTiles metadata fields in a free style YAML, including some
auto-detected tile info
2023-09-05 23:12:45 -04:00
Yuri Astrakhan
0fce9641c0
A few cleanups on tile hashing (#861)
* Rename `global_hash` to `agg_tiles_hash`

This is still a big sticking point: what should be the name for the
metadata key for this value? The value represents the hash of all
`z,x,y,tile` over all rows of the `tiles` table (or view). Should it
include `md5` in its name, or should the hash be auto-detected by its
length? (details in #856)

* Generate it based on `tiles` table/view
* validate or generate, but not both (it will always fail otherwise)
* break up logic for per-tile, total, and integrity checks
* delete unused sqlx prep file
2023-09-05 16:25:59 -04:00
Yuri Astrakhan
7a9ea5d0d4
Cleanup multi-feature styling (#859)
| Old Style | New style |
|---|---|
|
![image](https://github.com/maplibre/martin/assets/1641515/1cfc8951-76b8-4cbf-af1a-810d15e04b2b)
|
![image](https://github.com/maplibre/martin/assets/1641515/5d43164b-7084-4d4f-a343-a75f0a3f4dae)
|
2023-09-03 13:30:38 -04:00