Commit Graph

104 Commits

Author SHA1 Message Date
Yuri Astrakhan
0fce9641c0
A few cleanups on tile hashing (#861)
* Rename `global_hash` to `agg_tiles_hash`

This is still a big sticking point: what should be the name for the
metadata key for this value? The value represents the hash of all
`z,x,y,tile` over all rows of the `tiles` table (or view). Should it
include `md5` in its name, or should the hash be auto-detected by its
length? (details in #856)

* Generate it based on `tiles` table/view
* validate or generate, but not both (it will always fail otherwise)
* break up logic for per-tile, total, and integrity checks
* delete unused sqlx prep file
2023-09-05 16:25:59 -04:00
Yuri Astrakhan
7a9ea5d0d4
Cleanup multi-feature styling (#859)
| Old Style | New style |
|---|---|
|
![image](https://github.com/maplibre/martin/assets/1641515/1cfc8951-76b8-4cbf-af1a-810d15e04b2b)
|
![image](https://github.com/maplibre/martin/assets/1641515/5d43164b-7084-4d4f-a343-a75f0a3f4dae)
|
2023-09-03 13:30:38 -04:00
Lucas
fa451cbf9b
Show tile boundaries in debug.html (#858) 2023-09-04 00:35:34 +08:00
Lucas
d6f3908b7b
Rewrite debug.html to show and inspect all layers (#820)
Try to resolve #536, but still it can't show the function source without
comment...
- [x] Show all layers with correct geometry type
- [x] Inspect feature property
- [x] Allow to scroll popup content with max-height and a vertical
scroll bar
- [x] Allow to stop the popup showing after click outside
- [x] It's possible to select text when the popup it's fixed

Maybe we should have an inspect toggle?

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-09-03 02:12:49 +00:00
rstanciu
875a942fcf
Add integrity and global hash check to validation (#851)
Co-authored-by: rstanciu <rstanciu@rivian.com>
2023-08-31 03:40:40 +00:00
Yuri Astrakhan
9d8c8c948e
fix debug.html sources (#853) 2023-08-30 03:19:33 -04:00
Yuri Astrakhan
550a46bd0f
(breaking) Future-proof source catalog (#754)
Modify `/catalog` endpoint to return an object instead of a list. This
allows future expansion of the catalog schema, e.g. adding new types of
data.

The new schema:

```yaml
{
  "tiles" {
    "function_zxy_query": {
      "name": "public.function_zxy_query",
      "content_type": "application/x-protobuf"
    },
    "points1": {
      "name": "public.points1.geom",
      "content_type": "image/webp"
    },
    ...
  },
}
```
2023-08-28 04:31:22 +00:00
Lucas
b3fb720a94
Retrieve function source description from comments (#829)
If a PostgreSQL function has an SQL comment, it will try to parse as
JSON and use its values to override the auto-generated TileJSON. It is
recommended to use this form when creating comments to ensure valid JSON
values.

```sql
DO $do$ BEGIN
    EXECUTE 'COMMENT ON FUNCTION YOUR_FUNCTION (ARG1_TYPE,ARG2_TYPE,..ARGN_TYPE) IS $tj$' || $$
    {
      "description": "description override",
      ...
    }
    $$::json || '$tj$';
END $do$;
```

Partially implements #822
2023-08-27 16:46:56 +00:00
Yuri Astrakhan
bfaa3fbb17
Warn when source IDs are renamed (#839)
Make sure the users know when a source name was automatically modified,
and why.

See also #826
2023-08-26 09:53:06 -04:00
rstanciu
3e2dceb4bf
Add ability to set mbtiles metadata value (#817)
* Add cli command `meta-set` to set metadata `value`

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
2023-08-17 11:36:46 -07:00
rstanciu
720b682b7e
Add flat hashing table format (#808)
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-08-16 10:22:01 -07:00
Binabh
e3e6b3563f
adds support for id_column in tables for auto_publish (#790)
Resolves #682 

- [x] Get id_column string from config.yaml and use for id column
- [x] Support for list of strings
- [x] Add info/warnings if column is not there or is of wrong type
- [x] if column for the feature ID is found, remove it from properties
(see inline comment)
- [x] cleanup logging messages
- [x] need more tests to catch other edge cases

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-08-13 02:06:23 +00:00
Yuri Astrakhan
14ded482ac add sql test pg_dump info 2023-08-12 21:30:04 -04:00
Yuri Astrakhan
405d4b27a2
Improve auto-publish usability (#795)
* on `--save-config`, only save configured `auto_publish` settings
* alias `from_schemas` as `from_schema`
* add integration testing for `auto_publish`
* if integration test DB preloading fails, try to clean up the test DB
* A few more info traces

This change should benefit testing of the #790 cc: @Binabh
2023-08-03 23:51:10 +02:00
rstanciu
1342b38e75
Add apply-diff command (#747)
Add command `apply-diff` to apply diff file generated from `copy`
command
2023-07-05 21:38:03 +00:00
rstanciu
e004908722
Add diff-with-file option to copy command (#729)
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-07-05 16:55:39 +00:00
Pirmin Kalberer
aabcb0aa0a
Add sqlx-rustls support (#720)
I'd like to add optional rustls support for SQLx. This PR implements the
required features, but tests using `--all-features` fail, because
runtime selection features are not additive. I didn't find a solution
apart from avoiding `--all-features`.

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-07-03 21:29:44 +00:00
Yuri Astrakhan
ff29530804
Fix non-deterministic CI tests (#724)
* a race condition due to duplicated webp.mbtiles and webp.pmtiles
* minor cleanup of CI/dockerignore
* dependency bump
2023-06-22 23:01:57 -04:00
rstanciu
bc140e0216
Add basic copying functionality (#712)
Copy an existing `.mbtiles` file to a new file.

```shell
mbtiles copy  <src_file.mbtiles> <dst_file.mbtiles>
```

Optionally filters by zooms.  Supports de-duplicated and simple mbtiles

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-06-17 00:00:46 +00:00
Yuri Astrakhan
a5c5505713
Add dynamic sprites support (#715)
Dynamically create image sprites for MapLibre rendering, given a
directory with images.

### TODO
* [x] Work with @flother to merge these PRs
  * [x] https://github.com/flother/spreet/pull/59  (must have)
  * [x] https://github.com/flother/spreet/pull/57
  * [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use

### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`

Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.

### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:

```yaml
# Publish sprite images
sprites:
  paths:
    # scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
    - /path/to/my_images
  sources:
    # named source matching source name to a directory
    my_sprites: /path/to/some_dir
```

Implement #705
2023-06-16 08:19:47 -04:00
Yuri Astrakhan
7e20602079
Minor refactor and cleanup (#716)
* moved `IdResolver` to a separate file
* added `just fmt2` build target to format code using nightly mode
* moved all Actix `Data<AppState>` into individual state objects, e.g.
`Data<Sources>` (allows other source types, separate from `Sources`
type)
* move all Source-related code to a new `Sources` struct (a simple
wrapper over a HashMap)
2023-06-15 18:36:41 -04:00
Yuri Astrakhan
454d435589 Move invalid test fixture files 2023-06-03 23:06:37 -04:00
Yuri Astrakhan
b147c27380
Move Martin crate to its own subdir (#699)
This cleans up the code a bit, making all crates to be on the same
level.
2023-06-04 02:50:55 +00:00
Yuri Astrakhan
2cdd373044
Refactor mbtiles lib, mbtiles tool tests & CI (#702)
* Broke up martin-mbtiles into multiple files
* Made all mbtiles functions take a `SqliteExecutor` -- this way they
can be used with any SQLX connection structs - either a pool connection
or an individual non-pooled connection.
* Simplified mbtiles bin a bit - I realized there is really no need to
pretty print the output for the single value retrieval. Easier to just
dump it to console as is.
* Bump martin-mbtiles to v0.2.0
* Minor fixes in tools docs, cargo.toml, and justfile
* MBTiles tool Integration tests and release publishing

Major thanks to the
[stackoverflow](https://stackoverflow.com/questions/76394665/how-to-pass-sqlx-connection-a-mut-trait-as-a-fn-parameter-in-rust/76395111)
quick reply by @cafce25 on how to use generic sql executor!
2023-06-04 00:54:50 +02:00
baishikele
c24bc06229
allow resize menu by drag side on tests/debug.html (#677)
It's about resizing the left panel by dragging along the right side.
2023-05-25 12:07:43 -04:00
baishikele
fe6c207094
In auto-discover, hide geocolumns from tables with multiple ones (#663)
Hi @nyurik I'm trying to resolve #627 .But it's a little bit hard battle
against rust😂
2023-05-22 09:44:21 -04:00
Binabh
fd88852492
adds search; adds resize; adds dynamic layer color; (#660)
- [ ] show lines and polygons
- [ ] show all layers regardless of their name
- [x] different color for each layer
- [x] make left sidebar resizable
- [x] add search box

- closes #536
2023-05-17 23:15:29 -04:00
Yuri Astrakhan
9583f32edb
Fix merging multi-source tilejson (#653)
Improve merging of multiple tilejsons

Fix #652
2023-05-02 04:25:06 -04:00
Yuri Astrakhan
50462d5fbc
[breaking] Use table name as the layer ID for PG tables (#598)
* [BREAKING] Use source ID (table name) as the default layer ID, instead
of `schema.table.column`
* Add support for the optional `layer_id` table config parameter

Fix #595
2023-03-13 07:41:25 -04:00
Yuri Astrakhan
e92722708f
PG TileJSON changes, add vector_layers (#584)
* make tilejson's `name` be the same as the ID of the source (even if
aliased)
* `/catalog` will always show ID, but now it will hide the `name` if it
is the same as the `id`
* make `description` be the longer version, e.g. `public.table.column`
format - not guaranteed to be stable
* make `vector_layers` have the fields auto-discovered in the PG table
* preserve the order of the serialized json fields

Fixes #583
2023-02-22 16:25:48 +00:00
Yuri Astrakhan
3fcad46500
fix (re-)compression of images (#579)
Compression middleware turned out to be hard to use for image cases - it
simply looks at the content-encoding, and if not set, tries to compress
if accepted by the client.

Instead, now individual routes are configured with either that
middleware, or for tiles, I decompress and optionally recompress if
applicable.

Now encoding is tracked separately from the tile content, making it
cleaner too. Plus lots of tests for mbtiles & pmtiles.

Fixes #577
2023-02-20 10:44:22 -05:00
Yuri Astrakhan
1f0ca167bc
Feat: max_feature_count query limit (#576)
Implements #384 - ability to limit the number of features included in a
tile from a Postgres table/view.

This allows zoomed-out view of a table with a reasonable speed because
each tile could be limited in size, rather than include millions of
features.

If set on a CLI, overrides whatever is set in the config file (if
given).

Any naming suggestions?
2023-02-17 11:11:16 -05:00
Chris Thiange
22bf2008e5
Views handling (#571)
* Adds a view to `points1.sql` fixture
* Replaces `table` with `view` in log statements relating to views

---------

Co-authored-by: Chris Thiange <cthiange@gmail.com>
2023-02-07 20:45:01 -05:00
cthiange
612a8d38c4
Issue 540 - missing index warnings (#569)
Warn users when a PG table geometry column has no index - thus accessing it would be slow.  This is only done for tables. Issues with the views are not printed.

## Implementation

This adds two fields to `TableInfo`:
* `geom_idx: Option<bool>` to tell if a geo column has a spatial index
* `is_view: Option<bool>` to distinguish views from other relations

Missing spatial index warnings are logged for non-view relations. Views
will never have indexed columns and, if referencing a table with a
missing index, it will be logged already.

Couldn't figure out how to make `just test` accept the new warning (from
missing index), so I have them logged as INFO for now :)

fixes #540

---------

Co-authored-by: Christophe Thiange <cthiange@gmail.com>
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-02-07 01:05:47 -05:00
Yuri Astrakhan
f23da97691
tests, report unknown cfg, rm catalog vector flds (#551)
* clean up reporting of the un-used config params - instead of printing,
collect them and print in one place if needed (allows testing too)
* remove `vector_layer` in catalog - too verbose, not needed - can be
received via tilejson for individual source
* clean up tests so that they all use the same config yaml
2023-01-12 11:48:15 -05:00
Yuri Astrakhan
9e5ed2fc02
Add .mbtiles support (#549)
Adds a new [.mbtiles](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md)
backend, without the grid support. Uses extensive tile content
detection, i.e. if the content is gzipped, png, jpeg, gif, webp.

From CLI, can be as easy as adding a path to a directory that contains a
.mbtiles file (works just like pmtiles support)

```bash
# All *.mbtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways
(same as pmtiles)

```yaml
mbtiles:
  paths:
    # scan this whole dir, matching all *.mbtiles files
    - /dir-path
    # specific mbtiles file will be published as mbtiles2 source
    - /path/to/mbtiles2.mbtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/mbtiles.mbtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/mbtiles.mbtiles
```

Fixes #494
2023-01-09 00:10:23 -05:00
Yuri Astrakhan
ba65e34bef
Add .pmtiles file support (#517)
Merge after #548 

Adds a new [.pmtiles](https://protomaps.com/docs/pmtiles/) backend.
Supports all formats like png, vector, etc.

From CLI, can be as easy as adding a path to a directory that contains a
.pmtiles file:

```bash
# All *.pmtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways:

```yaml
pmtiles:
  paths:
    # scan this whole dir, matching all *.pmtiles files
    - /dir-path
    # specific pmtiles file will be published as pmtiles2 source
    - /path/to/pmtiles2.pmtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/pmtiles.pmtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/pmtiles.pmtiles
```

Fixes #508
2023-01-08 14:19:11 -05:00
Yuri Astrakhan
e58773f7b0
Refactorings, content-type/enc, cli parsing, tests, minor fixes (#548)
* introduce a new Connections object to track all positional strings
passed as the CLI arguments
* each tile provider can now indicate if it can take a positional CLI
arg, and if the value can be shared between multiple providers, i.e. if
its a directory that could contain files for multiple providers
* make xyz use better types - u8 for zoom, u32 for x&y. Postgres casts
those to INT2 and INT8
* minor bug in pre-push git hook to abort in case of a testingerror
* added GIF detection/type
* combine MVT and compression concepts into one enum more explicitly. It
is not ideal (technically they are separate concerns), but it keeps it a
bit simpler for now for multiple providers.
* set content encoding and content type on HTTP responses if known, and
also include them in the `/catalog` response (json)
* raise an error if the user attempts to merge non-concatenatable tiles
from multiple sources. We may want to implement it in the future, e.g.
combine multiple semi-transparent PNGs. Or even combine GIF & PNG & JPEG
* do not set content-type on empty responses (http 204)
* add tilejson outputs to testing
2023-01-08 16:31:58 +02:00
Yuri Astrakhan
928a700150
Implement postgres auto-publish (#546)
* NEW: support for #512 - pg table/function auto-discovery
  * can filter schemas
* can use patterns like `{schema}.{table}.{column}` and
`{schema}.{function}`
* NEW: add `disable_bounds` bool flag to allow disabling of the bounds
computation
* reworked integration tests to use yaml
2023-01-03 16:09:41 +00:00
Yuri Astrakhan
64ad07ce36
Fix legacy DB, add CI tests (#547)
* fixed SQL to work on older PG versions
* re-enable CI to test expected `test.sh` output against the one stored
in the `tests/expected`
* add postgres in docker tests on linux - one for the oldest supported
DB, and another using the more recent version
* minor justfile cleanup
* ensure config files are sorted alphabetically
2023-01-01 05:03:21 +00:00
Yuri Astrakhan
555a1fccdd
Improve Coverage tooling (#535)
* added manual coverage justfile command
* a lot of small refactorings of config and argument parsing
* feature: support jsonb query param for functions
* cleaned up public/private access
* make all tests populate with a predefined values to avoid issues with
random data
2022-12-27 06:56:27 +00:00
Yuri Astrakhan
2b240b3a34
Implement multi-sourced postgres (#511)
* Support `postgres` config key to be either a list or an object
* Use `PathBuf` instead of `String` where dealing with files
* Merge `SrvConfigBuilder` into `SrvConfig`
* Parse keep_alive as u64
* More config tests
2022-12-22 06:35:29 +00:00
Yuri Astrakhan
cb99b835e9
Enable clippy pedantic, fix lints (#525)
Pedantic lints often offer some good insight into the code. It is
usually easier to sprinkle a few "allow"-s around, than to miss some
important life hack offered by clippy.

Also, make use a different martin port when running integration tests
locally (make sure `git push` works even if martin is running).
2022-12-18 21:24:06 -05:00
Yuri Astrakhan
9efa364eb0
Fix handling for null-returning PG queries (#521)
Handle cases when a query returns a NULL or a table with no rows, or a
single row with a null value in it.

This fully fixes #519 in the main branch
2022-12-15 14:12:55 +02:00
Yuri Astrakhan
68c6259d32
Catch errors in tests, minor fixes (#514)
This extracts some of the code from #511 but without breaking changes

* Use `PathBuf` instead of `String` where dealing with files
* Parse keep_alive as u64
* More config tests to crash if martin output contains warnings or
errors
2022-12-12 09:11:10 -05:00
Yuri Astrakhan
171032c29a
Allow auto discovery per schema (#510)
Configuration refactoring to allow future schema filtering. This
partially solves #47, but it does not expose the configuration to the
user just yet.
2022-12-10 20:11:11 +00:00
Yuri Astrakhan
2ee517d135
Support z,x,y and record-returning funcs, table rework (#380)
Can now handle several additional Postgres functions to get a tile, plus
tons of small fixes

### Multiple result variants
* `getmvt(z,x,y) -> [bytea,md5]`  (single row with two columns)
* `getmvt(z,x,y) -> [bytea]` (single row with a single column)
* `getmvt(z,x,y) -> bytea` (value)

### Multiple input parameter variants
* `getmvt(z, x, y)` or `getmvt(zoom, x, y)` (all 3 vars must be
integers)
* `getmvt(z, x, y, url_query)`, where instead of `url_query` it could be
any other name, but must be of type JSON

### Breaking
* srid is now the same type as PG -- `i32`
* renamed config vals `table_sources` and `function_sources` into
`tables` and `functions`

### Features and fixes
* if postgis is v3.1+, uses margin parameter to extend the search box by
the size of the buffer. I think we should make 3.1 minimal required.
* fixes feature ID issue from #466
* fixes mixed case names for schemas, tables and columns, functions and
parameter names per #389
 

### Notes
* More dynamic SQL generation in code instead of using external SQL
files. Those should only be used when they are not parametrized.
* The new function/table discovery mechanism: query for all functions in
the database, and match up those functions with the ones configured (if
any), plus adds all the rest of the un-declared ones if discovery mode
is on.
* During table and function discovery, the code generates a map of
`(PgSqlInfo, FunctionInfo)` (or table) tupples containing SQL needed to
get the tile.
* Auto-discovery mode is currently hidden - the discovery is on only
when no tables or functions are configured. TBD - how to configure it in
the future
* The new system allows for an easy way to auto-discover for the
specific schemas only, solving #47
* predictable order of table/function instantiation
* bounding boxes computed in parallel for all tables (when not
configured)
* proper identifier escaping
* test cleanup

fixes #378
fixes #466
fixes #65
fixes #389
2022-12-10 16:20:42 +02:00
Yuri Astrakhan
ff7c31e16f
Improve pre-push hook and debug page (#509)
* write custom git pre-push via justfile
* fix tests/debug.html incorrectly loading test page
* minor cleanup of the obsolete just targets
2022-12-10 09:40:01 +02:00
Yuri Astrakhan
c1b65796f6
Require PostGIS 3 and use ST_TileEnvelope (#497)
* All tests and internal code now uses ST_TileEnvelope function
* Remove `tile_bbox`
* Rename test function sources for clarity - this will be needed in a
subsequent PR to add other function tests
2022-11-30 18:57:27 +02:00
Yuri Astrakhan
66b7fdc4ea
Multi-source support, new path structure (#456)
Partial implementation of the #430

* New endpoint structure:
  * `GET /` -- a placeholder for the future home page
* `GET /catalog` -- get a list of available sources, as a list of json
blobs.
* `[{id, name, description, attribution, vector_layer}, ...]` (some
fields might be missing)
* `GET /<id>` -- get tilejson for the given source, or a combination of
sources. No `.json` extension
  * `GET /<id>/<z>/<x>/<y>` -- get a tile. No format extension.
  * `GET /health` -- healthcheck
* Introduce a new tile format support crate (using code from the
maplibre/mbtileserve project)
* Removed the `/rpc/...` routes - all source IDs are accessed in the
same way
* Can print auto-generated configuration or save it to a file
* Refactored to support multiple sources from multiple backends, with a
proper naming conflict resolution

TODO:
* benchmarks need to be rewritten - they were relying on some internal
structures that are no longer there. This might be done as a separate PR
due to a very different internal architecture - might need to rethink
benchmarking approaches.
2022-11-26 04:46:40 -05:00