* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.
`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files
# Other
* Always sort JSON-serialized keys for consistency
* this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
After some thinking, it seems `mbtiles summary` (aliased as `mbtiles
info`) would be a bit better than stats. I renamed and adjusted
documentation, consolidating it in one doc page.
Other changes:
* use file system's file size, reporting 'unknown' if needed
* report page count
* moved bbox computation into a separate function
* inlined a number of things for readability
This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles
This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.
### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
* convert them to SDFs and package them into a protobuf
---------
Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
* Remove `--disable-bounds` flag and `disable_bounds` config parameters.
* Add `--auto-bounds` / `-b` CLI parameter and `auto_bounds` config
value:
* `quick`: Compute table geometry bounds, but abort if it takes longer
than 5 seconds (default)
* `calc`: Compute table geometry bounds. The startup time may be
significant. Make sure all GEO columns have indexes
* `skip`: Skip bounds calculation. The bounds will be set to the whole
world
* `-b` is now mapped to `--auto-bounds` param, but it will fail if used
by itself because it now requires a value.
Fixes#955
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.
## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content
There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).
## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
* Do not open the same mbtiles file more than once at the same time:
reuse the (unsafe) sqlite handle to create a `RusqliteConnection`
instead.
* The copying should set `agg_tiles_hash` in all cases because now it
uses the always available `tiles` table/view.
* a few minor cleanups and renames related to that
`postgres.auto_publish.tables` now lets users customize `clip_geom`, `buffer`, and `extent` instead of using their default values.
See also #872
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
Modify `/catalog` endpoint to return an object instead of a list. This
allows future expansion of the catalog schema, e.g. adding new types of
data.
The new schema:
```yaml
{
"tiles" {
"function_zxy_query": {
"name": "public.function_zxy_query",
"content_type": "application/x-protobuf"
},
"points1": {
"name": "public.points1.geom",
"content_type": "image/webp"
},
...
},
}
```
If a PostgreSQL function has an SQL comment, it will try to parse as
JSON and use its values to override the auto-generated TileJSON. It is
recommended to use this form when creating comments to ensure valid JSON
values.
```sql
DO $do$ BEGIN
EXECUTE 'COMMENT ON FUNCTION YOUR_FUNCTION (ARG1_TYPE,ARG2_TYPE,..ARGN_TYPE) IS $tj$' || $$
{
"description": "description override",
...
}
$$::json || '$tj$';
END $do$;
```
Partially implements #822
This fixes Demo https://martin.maplibre.org/ site (the new code is
already in production there). Main fixes:
* uses HTTPS again, just like the previous site
* uses nginx
* fixes all the relative paths, maplibre, minor other things
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)
---------
Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
* Use `postgis/postgis` image for the DB
* Build frontend image inside the docker
* Upgrade all packages except styled-components - that one causes some
errors
* Add yarn.lock to the repo
* Add justfile for simplicity
* Bump martin docker image version in a few places
Resolves#682
- [x] Get id_column string from config.yaml and use for id column
- [x] Support for list of strings
- [x] Add info/warnings if column is not there or is of wrong type
- [x] if column for the feature ID is found, remove it from properties
(see inline comment)
- [x] cleanup logging messages
- [x] need more tests to catch other edge cases
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
* rename configuration `auto_publish.tables.id_format` and
`auto_publish.functions.id_format` fields from `id_format` to
`source_id_format` fields. The `id_format` will continue to be supported
(read) from the configuration, but it will be auto-converted to the new
name on save. It is an error to have both in the same config file.
* The rename was discussed in #682
* internal refactorings: consolidate PG-related utilities, rename a few
vars, move PG errors to their own file.
This is partially made due to #790 (thanks @Binabh!) - and should be
merged before that to make that PR easier.
* Use same docker image for postgis & psql
* Improve docker book
* rename just target `mdbook` to `book` (more obvious choice)
* Cleanup justfile to not duplicate "cargo install" check & installation
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
