Commit Graph

32 Commits

Author SHA1 Message Date
Yuri Astrakhan
ff29530804
Fix non-deterministic CI tests (#724)
* a race condition due to duplicated webp.mbtiles and webp.pmtiles
* minor cleanup of CI/dockerignore
* dependency bump
2023-06-22 23:01:57 -04:00
Yuri Astrakhan
a5c5505713
Add dynamic sprites support (#715)
Dynamically create image sprites for MapLibre rendering, given a
directory with images.

### TODO
* [x] Work with @flother to merge these PRs
  * [x] https://github.com/flother/spreet/pull/59  (must have)
  * [x] https://github.com/flother/spreet/pull/57
  * [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use

### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`

Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.

### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:

```yaml
# Publish sprite images
sprites:
  paths:
    # scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
    - /path/to/my_images
  sources:
    # named source matching source name to a directory
    my_sprites: /path/to/some_dir
```

Implement #705
2023-06-16 08:19:47 -04:00
Yuri Astrakhan
454d435589 Move invalid test fixture files 2023-06-03 23:06:37 -04:00
Yuri Astrakhan
e92722708f
PG TileJSON changes, add vector_layers (#584)
* make tilejson's `name` be the same as the ID of the source (even if
aliased)
* `/catalog` will always show ID, but now it will hide the `name` if it
is the same as the `id`
* make `description` be the longer version, e.g. `public.table.column`
format - not guaranteed to be stable
* make `vector_layers` have the fields auto-discovered in the PG table
* preserve the order of the serialized json fields

Fixes #583
2023-02-22 16:25:48 +00:00
Yuri Astrakhan
3fcad46500
fix (re-)compression of images (#579)
Compression middleware turned out to be hard to use for image cases - it
simply looks at the content-encoding, and if not set, tries to compress
if accepted by the client.

Instead, now individual routes are configured with either that
middleware, or for tiles, I decompress and optionally recompress if
applicable.

Now encoding is tracked separately from the tile content, making it
cleaner too. Plus lots of tests for mbtiles & pmtiles.

Fixes #577
2023-02-20 10:44:22 -05:00
Chris Thiange
22bf2008e5
Views handling (#571)
* Adds a view to `points1.sql` fixture
* Replaces `table` with `view` in log statements relating to views

---------

Co-authored-by: Chris Thiange <cthiange@gmail.com>
2023-02-07 20:45:01 -05:00
cthiange
612a8d38c4
Issue 540 - missing index warnings (#569)
Warn users when a PG table geometry column has no index - thus accessing it would be slow.  This is only done for tables. Issues with the views are not printed.

## Implementation

This adds two fields to `TableInfo`:
* `geom_idx: Option<bool>` to tell if a geo column has a spatial index
* `is_view: Option<bool>` to distinguish views from other relations

Missing spatial index warnings are logged for non-view relations. Views
will never have indexed columns and, if referencing a table with a
missing index, it will be logged already.

Couldn't figure out how to make `just test` accept the new warning (from
missing index), so I have them logged as INFO for now :)

fixes #540

---------

Co-authored-by: Christophe Thiange <cthiange@gmail.com>
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-02-07 01:05:47 -05:00
Yuri Astrakhan
9e5ed2fc02
Add .mbtiles support (#549)
Adds a new [.mbtiles](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md)
backend, without the grid support. Uses extensive tile content
detection, i.e. if the content is gzipped, png, jpeg, gif, webp.

From CLI, can be as easy as adding a path to a directory that contains a
.mbtiles file (works just like pmtiles support)

```bash
# All *.mbtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways
(same as pmtiles)

```yaml
mbtiles:
  paths:
    # scan this whole dir, matching all *.mbtiles files
    - /dir-path
    # specific mbtiles file will be published as mbtiles2 source
    - /path/to/mbtiles2.mbtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/mbtiles.mbtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/mbtiles.mbtiles
```

Fixes #494
2023-01-09 00:10:23 -05:00
Yuri Astrakhan
ba65e34bef
Add .pmtiles file support (#517)
Merge after #548 

Adds a new [.pmtiles](https://protomaps.com/docs/pmtiles/) backend.
Supports all formats like png, vector, etc.

From CLI, can be as easy as adding a path to a directory that contains a
.pmtiles file:

```bash
# All *.pmtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways:

```yaml
pmtiles:
  paths:
    # scan this whole dir, matching all *.pmtiles files
    - /dir-path
    # specific pmtiles file will be published as pmtiles2 source
    - /path/to/pmtiles2.pmtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/pmtiles.pmtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/pmtiles.pmtiles
```

Fixes #508
2023-01-08 14:19:11 -05:00
Yuri Astrakhan
928a700150
Implement postgres auto-publish (#546)
* NEW: support for #512 - pg table/function auto-discovery
  * can filter schemas
* can use patterns like `{schema}.{table}.{column}` and
`{schema}.{function}`
* NEW: add `disable_bounds` bool flag to allow disabling of the bounds
computation
* reworked integration tests to use yaml
2023-01-03 16:09:41 +00:00
Yuri Astrakhan
555a1fccdd
Improve Coverage tooling (#535)
* added manual coverage justfile command
* a lot of small refactorings of config and argument parsing
* feature: support jsonb query param for functions
* cleaned up public/private access
* make all tests populate with a predefined values to avoid issues with
random data
2022-12-27 06:56:27 +00:00
Yuri Astrakhan
9efa364eb0
Fix handling for null-returning PG queries (#521)
Handle cases when a query returns a NULL or a table with no rows, or a
single row with a null value in it.

This fully fixes #519 in the main branch
2022-12-15 14:12:55 +02:00
Yuri Astrakhan
2ee517d135
Support z,x,y and record-returning funcs, table rework (#380)
Can now handle several additional Postgres functions to get a tile, plus
tons of small fixes

### Multiple result variants
* `getmvt(z,x,y) -> [bytea,md5]`  (single row with two columns)
* `getmvt(z,x,y) -> [bytea]` (single row with a single column)
* `getmvt(z,x,y) -> bytea` (value)

### Multiple input parameter variants
* `getmvt(z, x, y)` or `getmvt(zoom, x, y)` (all 3 vars must be
integers)
* `getmvt(z, x, y, url_query)`, where instead of `url_query` it could be
any other name, but must be of type JSON

### Breaking
* srid is now the same type as PG -- `i32`
* renamed config vals `table_sources` and `function_sources` into
`tables` and `functions`

### Features and fixes
* if postgis is v3.1+, uses margin parameter to extend the search box by
the size of the buffer. I think we should make 3.1 minimal required.
* fixes feature ID issue from #466
* fixes mixed case names for schemas, tables and columns, functions and
parameter names per #389
 

### Notes
* More dynamic SQL generation in code instead of using external SQL
files. Those should only be used when they are not parametrized.
* The new function/table discovery mechanism: query for all functions in
the database, and match up those functions with the ones configured (if
any), plus adds all the rest of the un-declared ones if discovery mode
is on.
* During table and function discovery, the code generates a map of
`(PgSqlInfo, FunctionInfo)` (or table) tupples containing SQL needed to
get the tile.
* Auto-discovery mode is currently hidden - the discovery is on only
when no tables or functions are configured. TBD - how to configure it in
the future
* The new system allows for an easy way to auto-discover for the
specific schemas only, solving #47
* predictable order of table/function instantiation
* bounding boxes computed in parallel for all tables (when not
configured)
* proper identifier escaping
* test cleanup

fixes #378
fixes #466
fixes #65
fixes #389
2022-12-10 16:20:42 +02:00
Yuri Astrakhan
c1b65796f6
Require PostGIS 3 and use ST_TileEnvelope (#497)
* All tests and internal code now uses ST_TileEnvelope function
* Remove `tile_bbox`
* Rename test function sources for clarity - this will be needed in a
subsequent PR to add other function tests
2022-11-30 18:57:27 +02:00
Yuri Astrakhan
b8ac719be8
Rework and consolidate CI (#467)
Rework CI to run tests locally using the VM-installed Postgres on all
target platforms.

### CI jobs
* Build release versions on Linux/Win/Mac and save build results as
output artifacts
* In a separate VMs (Linux/Win/Mac)
* use
[nyurik/action-setup-postgis](https://github.com/nyurik/action-setup-postgis)
to install postgis and run tests using the built artifacts
  * run `cargo test` on Linux only
* copy built artifacts from the build step, and run tests using the
release martin binary
  * package and publish if this is a release

### Other changes
* Port some minor changes from the rewrite to porting easier
* minor cleanups
* remove all "expected" data files - too unstable to be usable
2022-10-31 16:28:21 -04:00
Yuri Astrakhan
a0a411f088
Consolidate DB init and tests (#463)
* Add justfile to simplify running all the tests
* Save all PBF outputs to the text files
* Consolidate all tests to reuse the same code
* Consolidate database initialization
* updated readme with the new instructions

Note that while this PR creates "expected" files, the CI cannot validate
the generated results because the output is not stable. Eventually we
may try to output just the non-geometry values to have reasonable tests
comparing against the expected results.
2022-10-28 14:52:39 -04:00
Yuri Astrakhan
9b56fe7b9b
Refactor CI tests into separate scripts (#432)
This PR re-uses some ideas by @gbip from #448

* move all CI github workflow tests into the dedicated shell scripts
* consolitade two database initialization scripts into one
2022-10-25 14:54:36 -04:00
Stepan Kuzmin
b3401785b2
feat: add default SRID support (#308)
* feat: add default SRID support
2022-02-13 15:43:52 +03:00
Stepan Kuzmin
fc9170d164
feat: add multiple geometry columns support in table sources (#269)
* feat: add multiple geometry columns support in table sources

* test: add multiple geometry tests
2021-10-21 12:20:33 +03:00
Stepan Kuzmin
206307fd22
test: add config and multiple zoom level tests (#262)
* test: add config and multiple zoom level tests

* test: add more fixtures

* test: use ST_CurveToLine in function source fixtures

* test: update vtzero binaries
2021-10-12 17:56:45 +03:00
Stepan Kuzmin
40b0a0c26a
feat: add bounds to tilejson endpoints (#260) (h/t @jaspervercnocke) 2021-10-10 12:09:56 +03:00
Stepan Kuzmin
1a25129925
chore: improve debug.html (#257) 2021-10-07 19:54:25 +03:00
Paul FLORENCE
49942734af
fix: invalid json escaping (#224) (h/t @gbip)
* Fix invalid json escaping

* test: add test for function_source with query params
2021-07-16 12:09:18 +03:00
Stepan Kuzmin
3c01125fc5
feat: add composite sources support (#184)
* feat: add composite sources support WIP

* feat: handle empty composite sources

* fix: decompose queries

* docs: add docs on composite sources

* ci: push docker image

* test: add composite source tests
2021-04-24 20:19:37 +03:00
Stepan Kuzmin
5a807e40e2
feat: upgrade actix-web to 1.0 (#33)
* test: 💍 add function sources tests

* test: 💍 limit postgres pool size in tests

* feat: 🎸 upgrade actix-web to 1.0
2019-10-26 20:37:49 +03:00
Stepan Kuzmin
73d8e86860 chore: update dependencies 2018-10-17 12:11:23 +03:00
Stepan Kuzmin
be2cc3ea6f doc: update README.md 2018-10-12 18:18:34 +03:00
Stepan Kuzmin
12f1db9835 chore: add debug page 2018-10-09 15:00:25 +03:00
Stepan Kuzmin
334cf72107 test: add table and function source tests 2018-08-25 16:43:31 +03:00
Stepan Kuzmin
ccc986d037 test: add sources_not_found_test 2018-08-25 15:45:04 +03:00
Stepan Kuzmin
1e3422c09a WIP: switch to actix 2018-02-21 13:37:31 +03:00
Stepan Kuzmin
8ad3fccd8a fix tests 2018-01-30 12:36:48 +03:00