This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles
This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.
### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
* convert them to SDFs and package them into a protobuf
---------
Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
The `/catalog` now shows available sprites, which also paves way for the
future font support.
Lots of small refactorings to streamline tile source management. Now
each tile source can produce its own catalog entry, making the whole
thing much simpler.
Fixes#949
Modify `/catalog` endpoint to return an object instead of a list. This
allows future expansion of the catalog schema, e.g. adding new types of
data.
The new schema:
```yaml
{
"tiles" {
"function_zxy_query": {
"name": "public.function_zxy_query",
"content_type": "application/x-protobuf"
},
"points1": {
"name": "public.points1.geom",
"content_type": "image/webp"
},
...
},
}
```
Resolves#682
- [x] Get id_column string from config.yaml and use for id column
- [x] Support for list of strings
- [x] Add info/warnings if column is not there or is of wrong type
- [x] if column for the feature ID is found, remove it from properties
(see inline comment)
- [x] cleanup logging messages
- [x] need more tests to catch other edge cases
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
* on `--save-config`, only save configured `auto_publish` settings
* alias `from_schemas` as `from_schema`
* add integration testing for `auto_publish`
* if integration test DB preloading fails, try to clean up the test DB
* A few more info traces
This change should benefit testing of the #790 cc: @Binabh
Dynamically create image sprites for MapLibre rendering, given a
directory with images.
### TODO
* [x] Work with @flother to merge these PRs
* [x] https://github.com/flother/spreet/pull/59 (must have)
* [x] https://github.com/flother/spreet/pull/57
* [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use
### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`
Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.
### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:
```yaml
# Publish sprite images
sprites:
paths:
# scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
- /path/to/my_images
sources:
# named source matching source name to a directory
my_sprites: /path/to/some_dir
```
Implement #705
* [BREAKING] Use source ID (table name) as the default layer ID, instead
of `schema.table.column`
* Add support for the optional `layer_id` table config parameter
Fix#595
* make tilejson's `name` be the same as the ID of the source (even if
aliased)
* `/catalog` will always show ID, but now it will hide the `name` if it
is the same as the `id`
* make `description` be the longer version, e.g. `public.table.column`
format - not guaranteed to be stable
* make `vector_layers` have the fields auto-discovered in the PG table
* preserve the order of the serialized json fields
Fixes#583
Merge after #548
Adds a new [.pmtiles](https://protomaps.com/docs/pmtiles/) backend.
Supports all formats like png, vector, etc.
From CLI, can be as easy as adding a path to a directory that contains a
.pmtiles file:
```bash
# All *.pmtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```
From configuration file, the path can be specified in a number of ways:
```yaml
pmtiles:
paths:
# scan this whole dir, matching all *.pmtiles files
- /dir-path
# specific pmtiles file will be published as pmtiles2 source
- /path/to/pmtiles2.pmtiles
sources:
# named source matching source name to a single file
pm-src1: /tmp/pmtiles.pmtiles
# named source, where the filename is explicitly set. This way we will be able to add more options later
pm-src2:
path: /tmp/pmtiles.pmtiles
```
Fixes#508
* introduce a new Connections object to track all positional strings
passed as the CLI arguments
* each tile provider can now indicate if it can take a positional CLI
arg, and if the value can be shared between multiple providers, i.e. if
its a directory that could contain files for multiple providers
* make xyz use better types - u8 for zoom, u32 for x&y. Postgres casts
those to INT2 and INT8
* minor bug in pre-push git hook to abort in case of a testingerror
* added GIF detection/type
* combine MVT and compression concepts into one enum more explicitly. It
is not ideal (technically they are separate concerns), but it keeps it a
bit simpler for now for multiple providers.
* set content encoding and content type on HTTP responses if known, and
also include them in the `/catalog` response (json)
* raise an error if the user attempts to merge non-concatenatable tiles
from multiple sources. We may want to implement it in the future, e.g.
combine multiple semi-transparent PNGs. Or even combine GIF & PNG & JPEG
* do not set content-type on empty responses (http 204)
* add tilejson outputs to testing
* fixed SQL to work on older PG versions
* re-enable CI to test expected `test.sh` output against the one stored
in the `tests/expected`
* add postgres in docker tests on linux - one for the oldest supported
DB, and another using the more recent version
* minor justfile cleanup
* ensure config files are sorted alphabetically
Rework CI to run tests locally using the VM-installed Postgres on all
target platforms.
### CI jobs
* Build release versions on Linux/Win/Mac and save build results as
output artifacts
* In a separate VMs (Linux/Win/Mac)
* use
[nyurik/action-setup-postgis](https://github.com/nyurik/action-setup-postgis)
to install postgis and run tests using the built artifacts
* run `cargo test` on Linux only
* copy built artifacts from the build step, and run tests using the
release martin binary
* package and publish if this is a release
### Other changes
* Port some minor changes from the rewrite to porting easier
* minor cleanups
* remove all "expected" data files - too unstable to be usable
* Add justfile to simplify running all the tests
* Save all PBF outputs to the text files
* Consolidate all tests to reuse the same code
* Consolidate database initialization
* updated readme with the new instructions
Note that while this PR creates "expected" files, the CI cannot validate
the generated results because the output is not stable. Eventually we
may try to output just the non-geometry values to have reasonable tests
comparing against the expected results.
