Commit Graph

71 Commits

Author SHA1 Message Date
Yuri Astrakhan
2def6288f1
PMTiles cache, refactor file configs, modularize (#1094)
* Make it possible to have configuration specific to the file-based
config sections: pmtiles, mbtiles, and sprites.
* Implement PMTiles directory cache shared between all pmtiles (both
http and local), with configurable max cache size (in MB), or 0 to
disable. Defaults to 32MB (?)
* PMTiles now share web client instance, which optimizes connection
reuse in case multiple pmtiles reside on the same host
* Major refactoring to allow modular reuse, enabling the following build
features:
    * **postgres** - enable PostgreSQL/PostGIS tile sources
    * **pmtiles** - enable PMTile tile sources
    * **mbtiles** - enable MBTile tile sources
    * **fonts** - enable font sources
    * **sprites** - enable sprite sources
* Use justfile in the CI

Fixes #1093
2023-12-25 05:52:04 +00:00
Yuri Astrakhan
1a8e7c89a4
Implement http pmtiles (#991)
PMTiles is a web-optimized format, allowing the actual file to be read
with HTTP range requests. Supporting this use case instantly allows
Martin to function as a lambda executable accessing PMTiles, but without
any significant investment into devops or hosting large file.

PMTiles config now also allows `http` and `https` protocol.

```
# Publish PMTiles files
pmtiles:
  paths:
    # specific pmtiles file will be published as mypmtiles source
    # (use last portion of the URL without extension)
    - http://example.org/path/to/mypmtiles.pmtiles
  sources:
    # named source matching source name to a single file
    pm-src1: https://example.org/path/to/some_pmtiles.pmtiles
 ```

fixes #884

---------

Co-authored-by: Kyle Slugg-Urbino <35903887+kyleslugg@users.noreply.github.com>
2023-12-22 06:01:50 +00:00
Yuri Astrakhan
f88db05582
Fix SQL comments when func or table is pre-configured (#1045)
When a SQL comment is set on a table or a function to customize
tilejson, and that tbl/func is pre-configured as part of the config
file, the comment was silently ignored. Now both table and function
cases are handled correctly.

Also, update docs to not include function parameters - makes SQL example
a bit simpler.

Thanks @jjcfrancisco for reporting!

Fixes: #1044
2023-12-07 07:24:51 +00:00
Yuri Astrakhan
afc0fb8977 update doc version 2023-12-05 23:47:04 -05:00
Lucas
4814d3369c
Add Markdown linting and url check to CI and justfile (#1030)
Fix #979 with
[markdownlint-cli2](https://github.com/DavidAnson/markdownlint-cli2) and
[markdown-link-check](https://github.com/actions-marketplace-validations/gaurav-nelson_github-action-markdown-link-check)

## Link check
- [x] Add link check to just clippy
- [x] Add relative links check with
[markdown-link-check](https://github.com/actions-marketplace-validations/gaurav-nelson_github-action-markdown-link-check)
- [x] Ignore `CHANGELOG.md`
- [x] Ignore `localhost` and `127.0.0.1`
- [x] Ignore `http://opensource.org`
- [x] Add to justfile

## Lintting

- [x] Add markdown linting
- [x] Add markdown lint to just fmt
- [x] Ignore `CHANGELOG.md`?
- [x] Ignore [Line
Length](https://github.com/DavidAnson/markdownlint/blob/v0.32.1/doc/md013.md)
- [x] Ignore [First line in a file should be a top-level
heading](https://github.com/DavidAnson/markdownlint/blob/v0.32.1/doc/md041.md)
- [x] Ignore [Inline
HTML](https://github.com/DavidAnson/markdownlint/blob/v0.32.1/doc/md033.md)
- [x] Ignore [Images should have alternate
text](https://github.com/DavidAnson/markdownlint/blob/v0.32.1/doc/md045.md)
- [x] Ignore [Heading levels should only increment by one level at a
time](https://github.com/DavidAnson/markdownlint/blob/v0.32.1/doc/md001.md)
- [x] Rename lint config file.
- [x] Add to justfile

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-12-04 22:11:03 +00:00
Yuri Astrakhan
aeb08c8b22
add agg-tiles-hash calc to martin-cp (#1033)
* Automatically computes `agg-tiles-hash` metadata value after copying
tiles
* Add a `--skip-agg-tiles-hash` option
2023-12-02 09:12:19 +08:00
Yuri Astrakhan
d311888f94
Update Docker build process (#1022)
* Fix docker file builds to use debian
* add a few more things to `.dockerignore`
* remove unused `martin` from docker-compose.yml
* delete unused arm64.Dockerfile

This should also fix #1021
2023-11-23 21:14:32 -05:00
Yuri Astrakhan
d6219a6526
Add compression and query to martin-cp (#1019)
* Use gzip compression for MVT tiles by default
* Allow user to set compression with `--encoding` parameter (same as
browser's header, e.g. use `br,gzip' to encode with brotli as first
choice, unless already encoded as gzip)
* Allow user to pass a query to the Postgres functions with
`--url-query`
* A bit of a cleanup for `mbtiles summary` output
2023-11-22 05:26:53 +00:00
Lucas
566e8fa948
Revert doc book links - may break a few new ones (#1018)
- [x] Remove chapter numbers
- [x] Update file name of `using-endpoints.md` back
- [x] Update links

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-11-21 08:06:27 +00:00
Yuri Astrakhan
c6a7ef67c9
Rename dst-type to mbtiles-type (#1014) 2023-11-20 17:38:28 -05:00
Lucas
0f13fb6504
Add cargo binstall support (#1012)
Try to fix #1005
- [x] Add cargo binstall support
- [x] Update links
- [x] Update [home-brew](https://github.com/maplibre/homebrew-martin) by
another [PR](https://github.com/maplibre/homebrew-martin/pull/8)
- [x] To check is there anything to do for `mbtiles`
2023-11-20 02:56:09 -05:00
Yuri Astrakhan
f88fd10aa8
Add martin-cp tool to bulk-copy tiles from any Martin-supported sources (#1011)
This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.

`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
2023-11-20 06:27:51 +00:00
Yuri Astrakhan
140ed25679
mbtiles improvements, minor changes (#1013)
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files

# Other
* Always sort JSON-serialized keys for consistency
  * this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
2023-11-19 10:06:37 +00:00
Yuri Astrakhan
b0e19d3787
Improve bbox formatting (#1001)
Formats bbox using the most appropriate precision for that zoom level
2023-11-13 08:53:15 +00:00
Yuri Astrakhan
021cddcccd
mbtiles summary tool cleanup (#1000)
After some thinking, it seems `mbtiles summary` (aliased as `mbtiles
info`) would be a bit better than stats. I renamed and adjusted
documentation, consolidating it in one doc page.

Other changes:
* use file system's file size, reporting 'unknown' if needed
* report page count
* moved bbox computation into a separate function
* inlined a number of things for readability
2023-11-13 07:50:10 +00:00
Lucas
0398336114
Add mbtiles statistics command (#986)
- [x] Add tile statistics to mbtiles tools
- [x] Add test
- [x] Use 4326 instead of 3857 for tile bounds
- [x] Add document
- [x] Use size-format to prettify output 
- [x]  Statistics  struct Refactor
- [x] Cleanup and reformat

Closes #964 

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-11-13 06:03:40 +00:00
Yuri Astrakhan
f7c69f8302
Minor cleanup and just reorg (#992)
Only run git push diff on linux to make development on non-linux easier,
and a few preparations for pmtiles over http serving

Fixes #855
2023-11-08 19:46:23 -05:00
Yuri Astrakhan
09dd2bea62 cleanup trailing spaces 2023-10-31 00:56:13 -04:00
Yuri Astrakhan
d9f01d15d4 cleanup docs 2023-10-31 00:53:39 -04:00
Lucas
c6170c5913
Improve MBTiles documentation (#961)
Fixes #918 

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-10-31 00:41:21 -04:00
Yuri Astrakhan
65b1cdcb83 more doc fixes 2023-10-30 18:25:37 -04:00
Yuri Astrakhan
127c989f37 fix font docs 2023-10-30 18:18:03 -04:00
Yuri Astrakhan
f1241e264c fix CI status badges 2023-10-28 16:29:38 -04:00
Yuri Astrakhan
47b3106ac6
Rename martin-mbtiles crate to mbtiles (#976)
This simplifies usage as both a crate and as a tool
2023-10-28 08:50:49 +00:00
Yuri Astrakhan
9b112ae7b9
Implement dynamic font support /font/<name>/<start>-<end> (#755)
This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles

This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.

### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
  * convert them to SDFs and package them into a protobuf

---------

Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
2023-10-28 05:10:48 +00:00
Lucas
525e14b5fa
Update doc file names and title levels (#972)
Fix #962 
- [x] Rename doc files for better file organization/order
- [x] Update doc links
- [x]  Adjust title level
2023-10-27 20:22:57 +08:00
Yuri Astrakhan
7aa169ed67
[BREAKING] New bounds calculation methods (#958)
* Remove `--disable-bounds` flag and `disable_bounds` config parameters.
* Add `--auto-bounds` / `-b` CLI parameter and `auto_bounds` config
value:
* `quick`: Compute table geometry bounds, but abort if it takes longer
than 5 seconds (default)
* `calc`: Compute table geometry bounds. The startup time may be
significant. Make sure all GEO columns have indexes
* `skip`: Skip bounds calculation. The bounds will be set to the whole
world
* `-b` is now mapped to `--auto-bounds` param, but it will fail if used
by itself because it now requires a value.

Fixes #955
2023-10-22 03:30:18 -04:00
Lucas
dee34313ae
Updat doc about homebrew installation (#934) 2023-10-11 22:44:27 -04:00
Yuri Astrakhan
8b34cd374c
Add metadat copy/apply-diff, new testing framework (#921)
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.

## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content

There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).

## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
2023-10-10 11:10:17 -04:00
Yuri Astrakhan
8d7204c53d
Switch to Rustls (#474)
Fixes #471
2023-09-30 17:58:49 +08:00
Yuri Astrakhan
f52cd3c611 Minor mbtiles doc cleanups 2023-09-28 18:19:05 -04:00
Yuri Astrakhan
18504ff2cb
Update agg_tiles_hash docs and minor bug (#901)
* Do not open the same mbtiles file more than once at the same time:
reuse the (unsafe) sqlite handle to create a `RusqliteConnection`
instead.
* The copying should set `agg_tiles_hash` in all cases because now it
uses the always available `tiles` table/view.
* a few minor cleanups and renames related to that
2023-09-28 20:12:11 +02:00
Yuri Astrakhan
d39cc27e76
Delete trailing spaces (#894) 2023-09-26 15:41:40 +08:00
Lucas
b2b3e2ce36
Add clip_geom, buffer and extent under auto_publish conf (#887)
`postgres.auto_publish.tables` now lets users customize  `clip_geom`, `buffer`, and `extent` instead of using their default values.

See also #872

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-09-21 10:18:09 -04:00
Yuri Astrakhan
2f72d38444
Update development.md 2023-09-20 16:19:53 -04:00
Lucas
fa1bf4bdbd
Retrieve table/view comments as tilejson (#871) 2023-09-12 12:59:59 +08:00
Lucas
da05f43048
Update sources-pg-functions.md (#874) 2023-09-11 02:48:03 -04:00
Lucas
0fb5814663
Fix broken link in run-with-nginx.md (#870) 2023-09-08 00:59:01 -04:00
Yuri Astrakhan
f87da9adf8
Minor Linting and rename dst_type (#864) 2023-09-06 11:12:53 -04:00
Yuri Astrakhan
6cd9296bfc
add mbtiles meta-all command (#865)
Print all MBTiles metadata fields in a free style YAML, including some
auto-detected tile info
2023-09-05 23:12:45 -04:00
Yuri Astrakhan
550a46bd0f
(breaking) Future-proof source catalog (#754)
Modify `/catalog` endpoint to return an object instead of a list. This
allows future expansion of the catalog schema, e.g. adding new types of
data.

The new schema:

```yaml
{
  "tiles" {
    "function_zxy_query": {
      "name": "public.function_zxy_query",
      "content_type": "application/x-protobuf"
    },
    "points1": {
      "name": "public.points1.geom",
      "content_type": "image/webp"
    },
    ...
  },
}
```
2023-08-28 04:31:22 +00:00
Yuri Astrakhan
1131c28c99
cleanup func docs, lints (#842)
this also fixes new rust version lints
2023-08-28 09:07:29 +08:00
Lucas
b3fb720a94
Retrieve function source description from comments (#829)
If a PostgreSQL function has an SQL comment, it will try to parse as
JSON and use its values to override the auto-generated TileJSON. It is
recommended to use this form when creating comments to ensure valid JSON
values.

```sql
DO $do$ BEGIN
    EXECUTE 'COMMENT ON FUNCTION YOUR_FUNCTION (ARG1_TYPE,ARG2_TYPE,..ARGN_TYPE) IS $tj$' || $$
    {
      "description": "description override",
      ...
    }
    $$::json || '$tj$';
END $do$;
```

Partially implements #822
2023-08-27 16:46:56 +00:00
Yuri Astrakhan
1507625e57
Fix demo (#841)
This fixes Demo https://martin.maplibre.org/ site (the new code is
already in production there). Main fixes:

* uses HTTPS again, just like the previous site
* uses nginx
* fixes all the relative paths, maplibre, minor other things
2023-08-26 09:52:43 -04:00
rstanciu
d4d3bd8ba5
Add tiles_with_hash view to normalized mbtiles files (#834)
Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-08-23 17:21:46 -07:00
rstanciu
0907c8dc5b
Minor cleanup mbtiles tools (#833)
* Move tile validation to within `Mbtiles`
* Correct `apply-diff` documentation

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
2023-08-23 09:55:11 -07:00
rstanciu
3e2dceb4bf
Add ability to set mbtiles metadata value (#817)
* Add cli command `meta-set` to set metadata `value`

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
2023-08-17 11:36:46 -07:00
rstanciu
720b682b7e
Add flat hashing table format (#808)
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-08-16 10:22:01 -07:00
Yuri Astrakhan
fa50eeac97
Update and fix demo (#809)
* Use `postgis/postgis` image for the DB
* Build frontend image inside the docker
* Upgrade all packages except styled-components - that one causes some
errors
* Add yarn.lock to the repo
* Add justfile for simplicity
* Bump martin docker image version in a few places
2023-08-15 14:41:12 +02:00
Yuri Astrakhan
aa3db46599
Add metrics to reserved list, docs (#802)
minor change to allow for #793
2023-08-13 02:33:56 -04:00