Commit Graph

166 Commits

Author SHA1 Message Date
Yuri Astrakhan
928a700150
Implement postgres auto-publish (#546)
* NEW: support for #512 - pg table/function auto-discovery
  * can filter schemas
* can use patterns like `{schema}.{table}.{column}` and
`{schema}.{function}`
* NEW: add `disable_bounds` bool flag to allow disabling of the bounds
computation
* reworked integration tests to use yaml
2023-01-03 16:09:41 +00:00
Yuri Astrakhan
64ad07ce36
Fix legacy DB, add CI tests (#547)
* fixed SQL to work on older PG versions
* re-enable CI to test expected `test.sh` output against the one stored
in the `tests/expected`
* add postgres in docker tests on linux - one for the oldest supported
DB, and another using the more recent version
* minor justfile cleanup
* ensure config files are sorted alphabetically
2023-01-01 05:03:21 +00:00
Yuri Astrakhan
555a1fccdd
Improve Coverage tooling (#535)
* added manual coverage justfile command
* a lot of small refactorings of config and argument parsing
* feature: support jsonb query param for functions
* cleaned up public/private access
* make all tests populate with a predefined values to avoid issues with
random data
2022-12-27 06:56:27 +00:00
Yuri Astrakhan
2b240b3a34
Implement multi-sourced postgres (#511)
* Support `postgres` config key to be either a list or an object
* Use `PathBuf` instead of `String` where dealing with files
* Merge `SrvConfigBuilder` into `SrvConfig`
* Parse keep_alive as u64
* More config tests
2022-12-22 06:35:29 +00:00
Yuri Astrakhan
cb99b835e9
Enable clippy pedantic, fix lints (#525)
Pedantic lints often offer some good insight into the code. It is
usually easier to sprinkle a few "allow"-s around, than to miss some
important life hack offered by clippy.

Also, make use a different martin port when running integration tests
locally (make sure `git push` works even if martin is running).
2022-12-18 21:24:06 -05:00
Yuri Astrakhan
9efa364eb0
Fix handling for null-returning PG queries (#521)
Handle cases when a query returns a NULL or a table with no rows, or a
single row with a null value in it.

This fully fixes #519 in the main branch
2022-12-15 14:12:55 +02:00
Yuri Astrakhan
68c6259d32
Catch errors in tests, minor fixes (#514)
This extracts some of the code from #511 but without breaking changes

* Use `PathBuf` instead of `String` where dealing with files
* Parse keep_alive as u64
* More config tests to crash if martin output contains warnings or
errors
2022-12-12 09:11:10 -05:00
Yuri Astrakhan
171032c29a
Allow auto discovery per schema (#510)
Configuration refactoring to allow future schema filtering. This
partially solves #47, but it does not expose the configuration to the
user just yet.
2022-12-10 20:11:11 +00:00
Yuri Astrakhan
2ee517d135
Support z,x,y and record-returning funcs, table rework (#380)
Can now handle several additional Postgres functions to get a tile, plus
tons of small fixes

### Multiple result variants
* `getmvt(z,x,y) -> [bytea,md5]`  (single row with two columns)
* `getmvt(z,x,y) -> [bytea]` (single row with a single column)
* `getmvt(z,x,y) -> bytea` (value)

### Multiple input parameter variants
* `getmvt(z, x, y)` or `getmvt(zoom, x, y)` (all 3 vars must be
integers)
* `getmvt(z, x, y, url_query)`, where instead of `url_query` it could be
any other name, but must be of type JSON

### Breaking
* srid is now the same type as PG -- `i32`
* renamed config vals `table_sources` and `function_sources` into
`tables` and `functions`

### Features and fixes
* if postgis is v3.1+, uses margin parameter to extend the search box by
the size of the buffer. I think we should make 3.1 minimal required.
* fixes feature ID issue from #466
* fixes mixed case names for schemas, tables and columns, functions and
parameter names per #389
 

### Notes
* More dynamic SQL generation in code instead of using external SQL
files. Those should only be used when they are not parametrized.
* The new function/table discovery mechanism: query for all functions in
the database, and match up those functions with the ones configured (if
any), plus adds all the rest of the un-declared ones if discovery mode
is on.
* During table and function discovery, the code generates a map of
`(PgSqlInfo, FunctionInfo)` (or table) tupples containing SQL needed to
get the tile.
* Auto-discovery mode is currently hidden - the discovery is on only
when no tables or functions are configured. TBD - how to configure it in
the future
* The new system allows for an easy way to auto-discover for the
specific schemas only, solving #47
* predictable order of table/function instantiation
* bounding boxes computed in parallel for all tables (when not
configured)
* proper identifier escaping
* test cleanup

fixes #378
fixes #466
fixes #65
fixes #389
2022-12-10 16:20:42 +02:00
Yuri Astrakhan
ff7c31e16f
Improve pre-push hook and debug page (#509)
* write custom git pre-push via justfile
* fix tests/debug.html incorrectly loading test page
* minor cleanup of the obsolete just targets
2022-12-10 09:40:01 +02:00
Yuri Astrakhan
c1b65796f6
Require PostGIS 3 and use ST_TileEnvelope (#497)
* All tests and internal code now uses ST_TileEnvelope function
* Remove `tile_bbox`
* Rename test function sources for clarity - this will be needed in a
subsequent PR to add other function tests
2022-11-30 18:57:27 +02:00
Yuri Astrakhan
66b7fdc4ea
Multi-source support, new path structure (#456)
Partial implementation of the #430

* New endpoint structure:
  * `GET /` -- a placeholder for the future home page
* `GET /catalog` -- get a list of available sources, as a list of json
blobs.
* `[{id, name, description, attribution, vector_layer}, ...]` (some
fields might be missing)
* `GET /<id>` -- get tilejson for the given source, or a combination of
sources. No `.json` extension
  * `GET /<id>/<z>/<x>/<y>` -- get a tile. No format extension.
  * `GET /health` -- healthcheck
* Introduce a new tile format support crate (using code from the
maplibre/mbtileserve project)
* Removed the `/rpc/...` routes - all source IDs are accessed in the
same way
* Can print auto-generated configuration or save it to a file
* Refactored to support multiple sources from multiple backends, with a
proper naming conflict resolution

TODO:
* benchmarks need to be rewritten - they were relying on some internal
structures that are no longer there. This might be done as a separate PR
due to a very different internal architecture - might need to rethink
benchmarking approaches.
2022-11-26 04:46:40 -05:00
Yuri Astrakhan
0e0763a739
Make SSL support optional (#488)
By default, Martin is now compiled without openssl, simplifying
debugging and simple case usage, whereas the docker build and CI
publishing would still use openssl
2022-11-19 09:50:27 -05:00
Yuri Astrakhan
6b114cc7f1
Docker improvements and CI tests (#472)
* Change docker image to use `entrypoint` -- so that Martin can be used
as a command:

```bash
docker run maplibre/martin <parameters>
```

* The docker image is now tested the same way as in the CI tests
* Added a few changes to the justfile

Fixes #436
2022-11-02 19:00:05 +01:00
Yuri Astrakhan
b8ac719be8
Rework and consolidate CI (#467)
Rework CI to run tests locally using the VM-installed Postgres on all
target platforms.

### CI jobs
* Build release versions on Linux/Win/Mac and save build results as
output artifacts
* In a separate VMs (Linux/Win/Mac)
* use
[nyurik/action-setup-postgis](https://github.com/nyurik/action-setup-postgis)
to install postgis and run tests using the built artifacts
  * run `cargo test` on Linux only
* copy built artifacts from the build step, and run tests using the
release martin binary
  * package and publish if this is a release

### Other changes
* Port some minor changes from the rewrite to porting easier
* minor cleanups
* remove all "expected" data files - too unstable to be usable
2022-10-31 16:28:21 -04:00
Yuri Astrakhan
a0a411f088
Consolidate DB init and tests (#463)
* Add justfile to simplify running all the tests
* Save all PBF outputs to the text files
* Consolidate all tests to reuse the same code
* Consolidate database initialization
* updated readme with the new instructions

Note that while this PR creates "expected" files, the CI cannot validate
the generated results because the output is not stable. Eventually we
may try to output just the non-geometry values to have reasonable tests
comparing against the expected results.
2022-10-28 14:52:39 -04:00
Yuri Astrakhan
9b56fe7b9b
Refactor CI tests into separate scripts (#432)
This PR re-uses some ideas by @gbip from #448

* move all CI github workflow tests into the dedicated shell scripts
* consolitade two database initialization scripts into one
2022-10-25 14:54:36 -04:00
Yuri Astrakhan
def4a6f1ec
Warn on unknown cfg, CI cleanup (#461)
* Detect all unrecognized config file values, and report them. Ideally
we want to use `serde-ignored` crate, but it doesn't work with flattened
structs (yet). So using a bad workaround.
* CI test has been using all sorts of somewhat duplicated temporary pbf
files - cleaned up to `tmp.pbf`, and made sure curl only shows errors,
not download stats.
* In CI, crash psql instead of silently ignoring errors
* Don't serialize optional config values as nulls
* Tiny error message cleanup
2022-10-24 19:45:26 +02:00
Pat Sier
f5e19ddf9a
Migrate tests to use maplibre-gl (#449)
Migrates the HTML test files to use `maplibre-gl` instead of `mapbox-gl`. I kept the same version as in `tests/debug-maplibre.html` and used the Carto Positron style on `tests/debug.html` because it was the closet to Mapbox's light style.
2022-10-16 14:56:29 -04:00
Yuri Astrakhan
911076e3bc
Mostly noop code refactorings (#439)
* remove most of the utils:: and dev:: namespace usage
* rename `configure_db_source` to `configure_db_sources`
* use defaults from other table info objects (using the `..obj` syntax)
* use doc comments vs regular comment in a few places
* refactor server_test.rs to make it more readable and much shorter
  * use a macro for identical code
  * use a test_get() for identical GET request
* rename fn test_foo() into fn foo() for simplicity
2022-10-12 11:52:26 +03:00
Yuri Astrakhan
92f768bb3e
Configuration cleanup: step 2 (#425)
A few minor simplifications in the config and appstate:

* default srid seems to be unused.
* simplify Config struct to have non-optional table and function sources. Ok to be empty.
* add a parsing unit test
* rename configs to distinct names for simplicity

I am making this as a separate PR to keep things easier -- the big upcoming PR will use a dynamic dispatch system for all types of sources
2022-10-05 22:12:44 +03:00
Yuri Astrakhan
aed3274162
Breakup config for pg and srv (#420)
Getting ready for multi-backend system, where each backend would have its own configuration.

All 3 config structs (clap args, config, and configbuilder) are now separated into 3 files - the main one, one for service  (actix stuff), and one for PostgreSQL.

The files have been moved to `/pg` and `/srv`, but otherwise the logic should be mostly intact.
2022-10-01 07:48:11 +00:00
Yuri Astrakhan
47ed143d8f
Clean up configuration system to use Clap (#415)
* Use latest Clap-derive (currently v4rc, but should be public within a few days)
* reorganize configuration to streamline different config sources into one Config (using multiple ConfigBuilders)
2022-09-28 04:19:23 -04:00
Yuri Astrakhan
1e8e676e44
feat: use latest actix, switch to bb8 (async everywhere), remove actions (#377)
* remove all actions and other low-level magic code, making it more straightforward for the most common usage
* replace r2d2 with bb8 to make it all async
* use first significant version in cargo.toml - this makes it easier to maintain

This fixes #349
2022-08-15 09:54:48 -04:00
Yuri Astrakhan
7a14d6a3ed
feat!: remove --watch support (#381)
BREAKING CHANGE: Remove --watch support and ignore the command line parameter
2022-08-10 12:23:44 -04:00
Yuri Astrakhan
ad6c5297dd
sort use-statements during fmt (#376)
This does not force automatic use statement sorting,
but it sorts all them now, and we can manually keep them ordered until
the fmt features becomes stable.
2022-08-06 12:48:10 -04:00
Yuri Astrakhan
4a26366a60
chore: use tilejson 0.3.1 and simplify (#341) (h/t @nyurik)
* bounds merging is now built-in into Bounds as Add/AddAssign
* use Bounds::MAX instead of hard-coding it everywhere
2022-05-30 14:06:44 +03:00
Yuri Astrakhan
623f574a10
chore: remove extern, optimize use, fmt str (#338) (h/t @nyurik) 2022-05-29 20:00:49 +03:00
zhangzhang
5e2b6279b9
add maplibre example (#340) (h/t @sharkAndshark) 2022-05-29 20:00:05 +03:00
Yuri Astrakhan
66876d5254
chore: migrate to TileJSON v0.3 (#330) (h/t @nyurik)
* use `tilejson!` to generate json object
* use `Bounds` everywhere when appropriate
2022-05-27 14:04:27 +03:00
Yuri Astrakhan
d0655fd2b3
Use 2021 edition, modern format (#331) (h/t @nyurik)
* switch to 2021 rust edition
* use `format!("{var}")` instead of `"{}", var`
2022-05-27 14:03:48 +03:00
Stepan Kuzmin
b3401785b2
feat: add default SRID support (#308)
* feat: add default SRID support
2022-02-13 15:43:52 +03:00
Stepan Kuzmin
c8141e1c65
test: add function source tests (#271) 2021-10-23 17:02:18 +03:00
Stepan Kuzmin
069c8edf6b
test: add table source tests (#270) 2021-10-23 16:16:50 +03:00
Stepan Kuzmin
fc9170d164
feat: add multiple geometry columns support in table sources (#269)
* feat: add multiple geometry columns support in table sources

* test: add multiple geometry tests
2021-10-21 12:20:33 +03:00
Stepan Kuzmin
f2d56c2f7d
fix: tiles attribute in tilejson with x-rewrite-url (#266) 2021-10-15 18:19:36 +03:00
Stepan Kuzmin
194a83e63f
feat: add minzoom and maxzoom support (#265) 2021-10-13 14:51:29 +03:00
Stepan Kuzmin
206307fd22
test: add config and multiple zoom level tests (#262)
* test: add config and multiple zoom level tests

* test: add more fixtures

* test: use ST_CurveToLine in function source fixtures

* test: update vtzero binaries
2021-10-12 17:56:45 +03:00
Stepan Kuzmin
40b0a0c26a
feat: add bounds to tilejson endpoints (#260) (h/t @jaspervercnocke) 2021-10-10 12:09:56 +03:00
Stepan Kuzmin
1a25129925
chore: improve debug.html (#257) 2021-10-07 19:54:25 +03:00
Paul FLORENCE
49942734af
fix: invalid json escaping (#224) (h/t @gbip)
* Fix invalid json escaping

* test: add test for function_source with query params
2021-07-16 12:09:18 +03:00
Stepan Kuzmin
3c01125fc5
feat: add composite sources support (#184)
* feat: add composite sources support WIP

* feat: handle empty composite sources

* fix: decompose queries

* docs: add docs on composite sources

* ci: push docker image

* test: add composite source tests
2021-04-24 20:19:37 +03:00
AdriSolid
7622731ccf
docs: add deckgl example (#186) (h/t @AdriSolid)
* docs: add deckgl example

* Update README.md

Co-authored-by: Stepan Kuzmin <to.stepan.kuzmin@gmail.com>
2021-02-25 13:33:44 +03:00
Stepan Kuzmin
e953aa35a6
docs: add Leaflet example (#185) 2021-02-23 12:01:45 +03:00
Mazedur Rahman
fce1a9e722
feat: add server heartbeat for monitoring the service health (#168) (h/t @yamaszone)
* Added a simple health check to detect service heartbeat
* Test coverage for healthz endpoint
2021-01-16 16:11:19 +03:00
Stepan Kuzmin
7d5dfa8174
chore: drop mdillon postgis in favor of official image (#157) 2020-12-27 16:35:05 +03:00
Stepan Kuzmin
c1c59e158d
feat: add ssl/tls support #63 (#90)
* feat: add ssl/tls support #63

* feat: add danger-accept-invalid-certs option

* ci: switch to ekidd/rust-musl-builder:stable

* ci: enable docker/build-push-action
2020-06-02 09:49:21 +03:00
Stepan Kuzmin
1e34676a14
ci: refactor CI (#86)
* style: run fmt

* ci: run release only on tags

* ci: switch to actions/checkout@v2

* ci: add grcov
* ci: update docker job
2020-05-05 14:13:48 +03:00
Stepan Kuzmin
ad3076812d
feat: upgrade to actix-web 2.0 (#80)
* chore: update actix-web to 2.0

* fix: use default CORS middleware
2020-04-26 17:57:13 +03:00
Stepan Kuzmin
5a807e40e2
feat: upgrade actix-web to 1.0 (#33)
* test: 💍 add function sources tests

* test: 💍 limit postgres pool size in tests

* feat: 🎸 upgrade actix-web to 1.0
2019-10-26 20:37:49 +03:00
Stepan Kuzmin
3222a1d074
ci: 🎡 add vtzero check 2019-09-29 15:32:19 +03:00
Stepan Kuzmin
2f6cc93745 chore: update dependencies 2019-08-30 18:42:24 +03:00
Stepan Kuzmin
5eeef48b30 feat: 🎸 add watch mode #12
Closes: #12
2019-03-16 18:53:21 +03:00
Stepan Kuzmin
cbe9e89909 docs: ✏️ add notes on debugging 2019-02-04 20:22:04 +03:00
Stepan Kuzmin
ca8b330544 docs: ✏️ add config.yaml example to README.md 2019-02-04 13:40:01 +03:00
Stepan Kuzmin
73d8e86860 chore: update dependencies 2018-10-17 12:11:23 +03:00
Stepan Kuzmin
be2cc3ea6f doc: update README.md 2018-10-12 18:18:34 +03:00
Stepan Kuzmin
33be63e661 doc: update README.md 2018-10-11 15:01:47 +03:00
Stepan Kuzmin
12f1db9835 chore: add debug page 2018-10-09 15:00:25 +03:00
Stepan Kuzmin
334cf72107 test: add table and function source tests 2018-08-25 16:43:31 +03:00
Stepan Kuzmin
ccc986d037 test: add sources_not_found_test 2018-08-25 15:45:04 +03:00
Stepan Kuzmin
1e3422c09a WIP: switch to actix 2018-02-21 13:37:31 +03:00
Stepan Kuzmin
35483156e0 fix tests 2018-02-12 13:28:54 +03:00
Stepan Kuzmin
8ad3fccd8a fix tests 2018-01-30 12:36:48 +03:00
Stepan Kuzmin
cd637e1ecc disable cache in tests 2018-01-26 12:49:21 +03:00
Stepan Kuzmin
77cd4ae3d2 add martin_lib crate 2018-01-19 14:06:01 +03:00