Fixes#1245
Large book and README refactoring, adding quick started section, list of available features to the README, and moving most of the content from README to the book.
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
* **postgres** - enable PostgreSQL/PostGIS tile sources
* **pmtiles** - enable PMTile tile sources
* **mbtiles** - enable MBTile tile sources
* **fonts** - enable font sources
* **sprites** - enable sprite sources
* Use justfile in the CI
Fixes#1093
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.
PMTiles config now also allows `http` and `https` protocol.
```
# Publish PMTiles files
pmtiles:
paths:
# specific pmtiles file will be published as mypmtiles source
# (use last portion of the URL without extension)
- http://example.org/path/to/mypmtiles.pmtiles
sources:
# named source matching source name to a single file
pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
```
fixes#884
---------
Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.
`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.
## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content
There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).
## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
Dynamically create image sprites for MapLibre rendering, given a
directory with images.
### TODO
* [x] Work with @flother to merge these PRs
* [x] https://github.com/flother/spreet/pull/59 (must have)
* [x] https://github.com/flother/spreet/pull/57
* [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use
### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`
Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.
### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:
```yaml
# Publish sprite images
sprites:
paths:
# scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
- /path/to/my_images
sources:
# named source matching source name to a directory
my_sprites: /path/to/some_dir
```
Implement #705
Quoting DATABASE_URL content variable prevents an issue from the command
export. In case of a string (i.e. password) with given characters (i.e.
'('), it can make the command fail. Quoting it prevents that from
happening.
The issue happened on MacOS with an M1 chip.
- [x] add slack chat icon
- [x] merging maplibre/leaflet/mapbox demos into `using with clients`
- [x] cut off a lot contents
- [x] downgrade contents from h1 to h2
- [x] remove TOC .[GitHub will now automatically generate a table of
contents in the header when there are 2 or more
headings](https://github.blog/changelog/2021-04-13-table-of-contents-support-in-markdown-files/)
- [x] add link to the book in readme
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
* [BREAKING] Use source ID (table name) as the default layer ID, instead
of `schema.table.column`
* Add support for the optional `layer_id` table config parameter
Fix#595
* make tilejson's `name` be the same as the ID of the source (even if
aliased)
* `/catalog` will always show ID, but now it will hide the `name` if it
is the same as the `id`
* make `description` be the longer version, e.g. `public.table.column`
format - not guaranteed to be stable
* make `vector_layers` have the fields auto-discovered in the PG table
* preserve the order of the serialized json fields
Fixes#583
Compression middleware turned out to be hard to use for image cases - it
simply looks at the content-encoding, and if not set, tries to compress
if accepted by the client.
Instead, now individual routes are configured with either that
middleware, or for tiles, I decompress and optionally recompress if
applicable.
Now encoding is tracked separately from the tile content, making it
cleaner too. Plus lots of tests for mbtiles & pmtiles.
Fixes#577
Implements #384 - ability to limit the number of features included in a
tile from a Postgres table/view.
This allows zoomed-out view of a table with a reasonable speed because
each tile could be limited in size, rather than include millions of
features.
If set on a CLI, overrides whatever is set in the config file (if
given).
Any naming suggestions?
* added manual coverage justfile command
* a lot of small refactorings of config and argument parsing
* feature: support jsonb query param for functions
* cleaned up public/private access
* make all tests populate with a predefined values to avoid issues with
random data
* Support `postgres` config key to be either a list or an object
* Use `PathBuf` instead of `String` where dealing with files
* Merge `SrvConfigBuilder` into `SrvConfig`
* Parse keep_alive as u64
* More config tests
