Commit Graph

155 Commits

Author SHA1 Message Date
Yuri Astrakhan
6cd9296bfc
add mbtiles meta-all command (#865)
Print all MBTiles metadata fields in a free style YAML, including some
auto-detected tile info
2023-09-05 23:12:45 -04:00
Yuri Astrakhan
0fce9641c0
A few cleanups on tile hashing (#861)
* Rename `global_hash` to `agg_tiles_hash`

This is still a big sticking point: what should be the name for the
metadata key for this value? The value represents the hash of all
`z,x,y,tile` over all rows of the `tiles` table (or view). Should it
include `md5` in its name, or should the hash be auto-detected by its
length? (details in #856)

* Generate it based on `tiles` table/view
* validate or generate, but not both (it will always fail otherwise)
* break up logic for per-tile, total, and integrity checks
* delete unused sqlx prep file
2023-09-05 16:25:59 -04:00
Yuri Astrakhan
7a9ea5d0d4
Cleanup multi-feature styling (#859)
| Old Style | New style |
|---|---|
|
![image](https://github.com/maplibre/martin/assets/1641515/1cfc8951-76b8-4cbf-af1a-810d15e04b2b)
|
![image](https://github.com/maplibre/martin/assets/1641515/5d43164b-7084-4d4f-a343-a75f0a3f4dae)
|
2023-09-03 13:30:38 -04:00
Lucas
fa451cbf9b
Show tile boundaries in debug.html (#858) 2023-09-04 00:35:34 +08:00
Lucas
d6f3908b7b
Rewrite debug.html to show and inspect all layers (#820)
Try to resolve #536, but still it can't show the function source without
comment...
- [x] Show all layers with correct geometry type
- [x] Inspect feature property
- [x] Allow to scroll popup content with max-height and a vertical
scroll bar
- [x] Allow to stop the popup showing after click outside
- [x] It's possible to select text when the popup it's fixed

Maybe we should have an inspect toggle?

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-09-03 02:12:49 +00:00
rstanciu
875a942fcf
Add integrity and global hash check to validation (#851)
Co-authored-by: rstanciu <rstanciu@rivian.com>
2023-08-31 03:40:40 +00:00
Yuri Astrakhan
9d8c8c948e
fix debug.html sources (#853) 2023-08-30 03:19:33 -04:00
Yuri Astrakhan
550a46bd0f
(breaking) Future-proof source catalog (#754)
Modify `/catalog` endpoint to return an object instead of a list. This
allows future expansion of the catalog schema, e.g. adding new types of
data.

The new schema:

```yaml
{
  "tiles" {
    "function_zxy_query": {
      "name": "public.function_zxy_query",
      "content_type": "application/x-protobuf"
    },
    "points1": {
      "name": "public.points1.geom",
      "content_type": "image/webp"
    },
    ...
  },
}
```
2023-08-28 04:31:22 +00:00
Lucas
b3fb720a94
Retrieve function source description from comments (#829)
If a PostgreSQL function has an SQL comment, it will try to parse as
JSON and use its values to override the auto-generated TileJSON. It is
recommended to use this form when creating comments to ensure valid JSON
values.

```sql
DO $do$ BEGIN
    EXECUTE 'COMMENT ON FUNCTION YOUR_FUNCTION (ARG1_TYPE,ARG2_TYPE,..ARGN_TYPE) IS $tj$' || $$
    {
      "description": "description override",
      ...
    }
    $$::json || '$tj$';
END $do$;
```

Partially implements #822
2023-08-27 16:46:56 +00:00
Yuri Astrakhan
bfaa3fbb17
Warn when source IDs are renamed (#839)
Make sure the users know when a source name was automatically modified,
and why.

See also #826
2023-08-26 09:53:06 -04:00
rstanciu
3e2dceb4bf
Add ability to set mbtiles metadata value (#817)
* Add cli command `meta-set` to set metadata `value`

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
2023-08-17 11:36:46 -07:00
rstanciu
720b682b7e
Add flat hashing table format (#808)
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-08-16 10:22:01 -07:00
Binabh
e3e6b3563f
adds support for id_column in tables for auto_publish (#790)
Resolves #682 

- [x] Get id_column string from config.yaml and use for id column
- [x] Support for list of strings
- [x] Add info/warnings if column is not there or is of wrong type
- [x] if column for the feature ID is found, remove it from properties
(see inline comment)
- [x] cleanup logging messages
- [x] need more tests to catch other edge cases

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-08-13 02:06:23 +00:00
Yuri Astrakhan
14ded482ac add sql test pg_dump info 2023-08-12 21:30:04 -04:00
Yuri Astrakhan
405d4b27a2
Improve auto-publish usability (#795)
* on `--save-config`, only save configured `auto_publish` settings
* alias `from_schemas` as `from_schema`
* add integration testing for `auto_publish`
* if integration test DB preloading fails, try to clean up the test DB
* A few more info traces

This change should benefit testing of the #790 cc: @Binabh
2023-08-03 23:51:10 +02:00
rstanciu
1342b38e75
Add apply-diff command (#747)
Add command `apply-diff` to apply diff file generated from `copy`
command
2023-07-05 21:38:03 +00:00
rstanciu
e004908722
Add diff-with-file option to copy command (#729)
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-07-05 16:55:39 +00:00
Pirmin Kalberer
aabcb0aa0a
Add sqlx-rustls support (#720)
I'd like to add optional rustls support for SQLx. This PR implements the
required features, but tests using `--all-features` fail, because
runtime selection features are not additive. I didn't find a solution
apart from avoiding `--all-features`.

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-07-03 21:29:44 +00:00
Yuri Astrakhan
ff29530804
Fix non-deterministic CI tests (#724)
* a race condition due to duplicated webp.mbtiles and webp.pmtiles
* minor cleanup of CI/dockerignore
* dependency bump
2023-06-22 23:01:57 -04:00
rstanciu
bc140e0216
Add basic copying functionality (#712)
Copy an existing `.mbtiles` file to a new file.

```shell
mbtiles copy  <src_file.mbtiles> <dst_file.mbtiles>
```

Optionally filters by zooms.  Supports de-duplicated and simple mbtiles

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-06-17 00:00:46 +00:00
Yuri Astrakhan
a5c5505713
Add dynamic sprites support (#715)
Dynamically create image sprites for MapLibre rendering, given a
directory with images.

### TODO
* [x] Work with @flother to merge these PRs
  * [x] https://github.com/flother/spreet/pull/59  (must have)
  * [x] https://github.com/flother/spreet/pull/57
  * [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use

### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`

Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.

### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:

```yaml
# Publish sprite images
sprites:
  paths:
    # scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
    - /path/to/my_images
  sources:
    # named source matching source name to a directory
    my_sprites: /path/to/some_dir
```

Implement #705
2023-06-16 08:19:47 -04:00
Yuri Astrakhan
7e20602079
Minor refactor and cleanup (#716)
* moved `IdResolver` to a separate file
* added `just fmt2` build target to format code using nightly mode
* moved all Actix `Data<AppState>` into individual state objects, e.g.
`Data<Sources>` (allows other source types, separate from `Sources`
type)
* move all Source-related code to a new `Sources` struct (a simple
wrapper over a HashMap)
2023-06-15 18:36:41 -04:00
Yuri Astrakhan
454d435589 Move invalid test fixture files 2023-06-03 23:06:37 -04:00
Yuri Astrakhan
b147c27380
Move Martin crate to its own subdir (#699)
This cleans up the code a bit, making all crates to be on the same
level.
2023-06-04 02:50:55 +00:00
Yuri Astrakhan
2cdd373044
Refactor mbtiles lib, mbtiles tool tests & CI (#702)
* Broke up martin-mbtiles into multiple files
* Made all mbtiles functions take a `SqliteExecutor` -- this way they
can be used with any SQLX connection structs - either a pool connection
or an individual non-pooled connection.
* Simplified mbtiles bin a bit - I realized there is really no need to
pretty print the output for the single value retrieval. Easier to just
dump it to console as is.
* Bump martin-mbtiles to v0.2.0
* Minor fixes in tools docs, cargo.toml, and justfile
* MBTiles tool Integration tests and release publishing

Major thanks to the
[stackoverflow](https://stackoverflow.com/questions/76394665/how-to-pass-sqlx-connection-a-mut-trait-as-a-fn-parameter-in-rust/76395111)
quick reply by @cafce25 on how to use generic sql executor!
2023-06-04 00:54:50 +02:00
baishikele
c24bc06229
allow resize menu by drag side on tests/debug.html (#677)
It's about resizing the left panel by dragging along the right side.
2023-05-25 12:07:43 -04:00
baishikele
fe6c207094
In auto-discover, hide geocolumns from tables with multiple ones (#663)
Hi @nyurik I'm trying to resolve #627 .But it's a little bit hard battle
against rust😂
2023-05-22 09:44:21 -04:00
Binabh
fd88852492
adds search; adds resize; adds dynamic layer color; (#660)
- [ ] show lines and polygons
- [ ] show all layers regardless of their name
- [x] different color for each layer
- [x] make left sidebar resizable
- [x] add search box

- closes #536
2023-05-17 23:15:29 -04:00
Yuri Astrakhan
9583f32edb
Fix merging multi-source tilejson (#653)
Improve merging of multiple tilejsons

Fix #652
2023-05-02 04:25:06 -04:00
Yuri Astrakhan
50462d5fbc
[breaking] Use table name as the layer ID for PG tables (#598)
* [BREAKING] Use source ID (table name) as the default layer ID, instead
of `schema.table.column`
* Add support for the optional `layer_id` table config parameter

Fix #595
2023-03-13 07:41:25 -04:00
Yuri Astrakhan
e92722708f
PG TileJSON changes, add vector_layers (#584)
* make tilejson's `name` be the same as the ID of the source (even if
aliased)
* `/catalog` will always show ID, but now it will hide the `name` if it
is the same as the `id`
* make `description` be the longer version, e.g. `public.table.column`
format - not guaranteed to be stable
* make `vector_layers` have the fields auto-discovered in the PG table
* preserve the order of the serialized json fields

Fixes #583
2023-02-22 16:25:48 +00:00
Yuri Astrakhan
3fcad46500
fix (re-)compression of images (#579)
Compression middleware turned out to be hard to use for image cases - it
simply looks at the content-encoding, and if not set, tries to compress
if accepted by the client.

Instead, now individual routes are configured with either that
middleware, or for tiles, I decompress and optionally recompress if
applicable.

Now encoding is tracked separately from the tile content, making it
cleaner too. Plus lots of tests for mbtiles & pmtiles.

Fixes #577
2023-02-20 10:44:22 -05:00
Yuri Astrakhan
1f0ca167bc
Feat: max_feature_count query limit (#576)
Implements #384 - ability to limit the number of features included in a
tile from a Postgres table/view.

This allows zoomed-out view of a table with a reasonable speed because
each tile could be limited in size, rather than include millions of
features.

If set on a CLI, overrides whatever is set in the config file (if
given).

Any naming suggestions?
2023-02-17 11:11:16 -05:00
Chris Thiange
22bf2008e5
Views handling (#571)
* Adds a view to `points1.sql` fixture
* Replaces `table` with `view` in log statements relating to views

---------

Co-authored-by: Chris Thiange <cthiange@gmail.com>
2023-02-07 20:45:01 -05:00
cthiange
612a8d38c4
Issue 540 - missing index warnings (#569)
Warn users when a PG table geometry column has no index - thus accessing it would be slow.  This is only done for tables. Issues with the views are not printed.

## Implementation

This adds two fields to `TableInfo`:
* `geom_idx: Option<bool>` to tell if a geo column has a spatial index
* `is_view: Option<bool>` to distinguish views from other relations

Missing spatial index warnings are logged for non-view relations. Views
will never have indexed columns and, if referencing a table with a
missing index, it will be logged already.

Couldn't figure out how to make `just test` accept the new warning (from
missing index), so I have them logged as INFO for now :)

fixes #540

---------

Co-authored-by: Christophe Thiange <cthiange@gmail.com>
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-02-07 01:05:47 -05:00
Yuri Astrakhan
f23da97691
tests, report unknown cfg, rm catalog vector flds (#551)
* clean up reporting of the un-used config params - instead of printing,
collect them and print in one place if needed (allows testing too)
* remove `vector_layer` in catalog - too verbose, not needed - can be
received via tilejson for individual source
* clean up tests so that they all use the same config yaml
2023-01-12 11:48:15 -05:00
Yuri Astrakhan
9e5ed2fc02
Add .mbtiles support (#549)
Adds a new [.mbtiles](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md)
backend, without the grid support. Uses extensive tile content
detection, i.e. if the content is gzipped, png, jpeg, gif, webp.

From CLI, can be as easy as adding a path to a directory that contains a
.mbtiles file (works just like pmtiles support)

```bash
# All *.mbtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways
(same as pmtiles)

```yaml
mbtiles:
  paths:
    # scan this whole dir, matching all *.mbtiles files
    - /dir-path
    # specific mbtiles file will be published as mbtiles2 source
    - /path/to/mbtiles2.mbtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/mbtiles.mbtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/mbtiles.mbtiles
```

Fixes #494
2023-01-09 00:10:23 -05:00
Yuri Astrakhan
ba65e34bef
Add .pmtiles file support (#517)
Merge after #548 

Adds a new [.pmtiles](https://protomaps.com/docs/pmtiles/) backend.
Supports all formats like png, vector, etc.

From CLI, can be as easy as adding a path to a directory that contains a
.pmtiles file:

```bash
# All *.pmtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways:

```yaml
pmtiles:
  paths:
    # scan this whole dir, matching all *.pmtiles files
    - /dir-path
    # specific pmtiles file will be published as pmtiles2 source
    - /path/to/pmtiles2.pmtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/pmtiles.pmtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/pmtiles.pmtiles
```

Fixes #508
2023-01-08 14:19:11 -05:00
Yuri Astrakhan
e58773f7b0
Refactorings, content-type/enc, cli parsing, tests, minor fixes (#548)
* introduce a new Connections object to track all positional strings
passed as the CLI arguments
* each tile provider can now indicate if it can take a positional CLI
arg, and if the value can be shared between multiple providers, i.e. if
its a directory that could contain files for multiple providers
* make xyz use better types - u8 for zoom, u32 for x&y. Postgres casts
those to INT2 and INT8
* minor bug in pre-push git hook to abort in case of a testingerror
* added GIF detection/type
* combine MVT and compression concepts into one enum more explicitly. It
is not ideal (technically they are separate concerns), but it keeps it a
bit simpler for now for multiple providers.
* set content encoding and content type on HTTP responses if known, and
also include them in the `/catalog` response (json)
* raise an error if the user attempts to merge non-concatenatable tiles
from multiple sources. We may want to implement it in the future, e.g.
combine multiple semi-transparent PNGs. Or even combine GIF & PNG & JPEG
* do not set content-type on empty responses (http 204)
* add tilejson outputs to testing
2023-01-08 16:31:58 +02:00
Yuri Astrakhan
928a700150
Implement postgres auto-publish (#546)
* NEW: support for #512 - pg table/function auto-discovery
  * can filter schemas
* can use patterns like `{schema}.{table}.{column}` and
`{schema}.{function}`
* NEW: add `disable_bounds` bool flag to allow disabling of the bounds
computation
* reworked integration tests to use yaml
2023-01-03 16:09:41 +00:00
Yuri Astrakhan
64ad07ce36
Fix legacy DB, add CI tests (#547)
* fixed SQL to work on older PG versions
* re-enable CI to test expected `test.sh` output against the one stored
in the `tests/expected`
* add postgres in docker tests on linux - one for the oldest supported
DB, and another using the more recent version
* minor justfile cleanup
* ensure config files are sorted alphabetically
2023-01-01 05:03:21 +00:00
Yuri Astrakhan
555a1fccdd
Improve Coverage tooling (#535)
* added manual coverage justfile command
* a lot of small refactorings of config and argument parsing
* feature: support jsonb query param for functions
* cleaned up public/private access
* make all tests populate with a predefined values to avoid issues with
random data
2022-12-27 06:56:27 +00:00
Yuri Astrakhan
2b240b3a34
Implement multi-sourced postgres (#511)
* Support `postgres` config key to be either a list or an object
* Use `PathBuf` instead of `String` where dealing with files
* Merge `SrvConfigBuilder` into `SrvConfig`
* Parse keep_alive as u64
* More config tests
2022-12-22 06:35:29 +00:00
Yuri Astrakhan
cb99b835e9
Enable clippy pedantic, fix lints (#525)
Pedantic lints often offer some good insight into the code. It is
usually easier to sprinkle a few "allow"-s around, than to miss some
important life hack offered by clippy.

Also, make use a different martin port when running integration tests
locally (make sure `git push` works even if martin is running).
2022-12-18 21:24:06 -05:00
Yuri Astrakhan
9efa364eb0
Fix handling for null-returning PG queries (#521)
Handle cases when a query returns a NULL or a table with no rows, or a
single row with a null value in it.

This fully fixes #519 in the main branch
2022-12-15 14:12:55 +02:00
Yuri Astrakhan
68c6259d32
Catch errors in tests, minor fixes (#514)
This extracts some of the code from #511 but without breaking changes

* Use `PathBuf` instead of `String` where dealing with files
* Parse keep_alive as u64
* More config tests to crash if martin output contains warnings or
errors
2022-12-12 09:11:10 -05:00
Yuri Astrakhan
171032c29a
Allow auto discovery per schema (#510)
Configuration refactoring to allow future schema filtering. This
partially solves #47, but it does not expose the configuration to the
user just yet.
2022-12-10 20:11:11 +00:00
Yuri Astrakhan
2ee517d135
Support z,x,y and record-returning funcs, table rework (#380)
Can now handle several additional Postgres functions to get a tile, plus
tons of small fixes

### Multiple result variants
* `getmvt(z,x,y) -> [bytea,md5]`  (single row with two columns)
* `getmvt(z,x,y) -> [bytea]` (single row with a single column)
* `getmvt(z,x,y) -> bytea` (value)

### Multiple input parameter variants
* `getmvt(z, x, y)` or `getmvt(zoom, x, y)` (all 3 vars must be
integers)
* `getmvt(z, x, y, url_query)`, where instead of `url_query` it could be
any other name, but must be of type JSON

### Breaking
* srid is now the same type as PG -- `i32`
* renamed config vals `table_sources` and `function_sources` into
`tables` and `functions`

### Features and fixes
* if postgis is v3.1+, uses margin parameter to extend the search box by
the size of the buffer. I think we should make 3.1 minimal required.
* fixes feature ID issue from #466
* fixes mixed case names for schemas, tables and columns, functions and
parameter names per #389
 

### Notes
* More dynamic SQL generation in code instead of using external SQL
files. Those should only be used when they are not parametrized.
* The new function/table discovery mechanism: query for all functions in
the database, and match up those functions with the ones configured (if
any), plus adds all the rest of the un-declared ones if discovery mode
is on.
* During table and function discovery, the code generates a map of
`(PgSqlInfo, FunctionInfo)` (or table) tupples containing SQL needed to
get the tile.
* Auto-discovery mode is currently hidden - the discovery is on only
when no tables or functions are configured. TBD - how to configure it in
the future
* The new system allows for an easy way to auto-discover for the
specific schemas only, solving #47
* predictable order of table/function instantiation
* bounding boxes computed in parallel for all tables (when not
configured)
* proper identifier escaping
* test cleanup

fixes #378
fixes #466
fixes #65
fixes #389
2022-12-10 16:20:42 +02:00
Yuri Astrakhan
ff7c31e16f
Improve pre-push hook and debug page (#509)
* write custom git pre-push via justfile
* fix tests/debug.html incorrectly loading test page
* minor cleanup of the obsolete just targets
2022-12-10 09:40:01 +02:00
Yuri Astrakhan
c1b65796f6
Require PostGIS 3 and use ST_TileEnvelope (#497)
* All tests and internal code now uses ST_TileEnvelope function
* Remove `tile_bbox`
* Rename test function sources for clarity - this will be needed in a
subsequent PR to add other function tests
2022-11-30 18:57:27 +02:00
Yuri Astrakhan
66b7fdc4ea
Multi-source support, new path structure (#456)
Partial implementation of the #430

* New endpoint structure:
  * `GET /` -- a placeholder for the future home page
* `GET /catalog` -- get a list of available sources, as a list of json
blobs.
* `[{id, name, description, attribution, vector_layer}, ...]` (some
fields might be missing)
* `GET /<id>` -- get tilejson for the given source, or a combination of
sources. No `.json` extension
  * `GET /<id>/<z>/<x>/<y>` -- get a tile. No format extension.
  * `GET /health` -- healthcheck
* Introduce a new tile format support crate (using code from the
maplibre/mbtileserve project)
* Removed the `/rpc/...` routes - all source IDs are accessed in the
same way
* Can print auto-generated configuration or save it to a file
* Refactored to support multiple sources from multiple backends, with a
proper naming conflict resolution

TODO:
* benchmarks need to be rewritten - they were relying on some internal
structures that are no longer there. This might be done as a separate PR
due to a very different internal architecture - might need to rethink
benchmarking approaches.
2022-11-26 04:46:40 -05:00
Yuri Astrakhan
0e0763a739
Make SSL support optional (#488)
By default, Martin is now compiled without openssl, simplifying
debugging and simple case usage, whereas the docker build and CI
publishing would still use openssl
2022-11-19 09:50:27 -05:00
Yuri Astrakhan
6b114cc7f1
Docker improvements and CI tests (#472)
* Change docker image to use `entrypoint` -- so that Martin can be used
as a command:

```bash
docker run maplibre/martin <parameters>
```

* The docker image is now tested the same way as in the CI tests
* Added a few changes to the justfile

Fixes #436
2022-11-02 19:00:05 +01:00
Yuri Astrakhan
b8ac719be8
Rework and consolidate CI (#467)
Rework CI to run tests locally using the VM-installed Postgres on all
target platforms.

### CI jobs
* Build release versions on Linux/Win/Mac and save build results as
output artifacts
* In a separate VMs (Linux/Win/Mac)
* use
[nyurik/action-setup-postgis](https://github.com/nyurik/action-setup-postgis)
to install postgis and run tests using the built artifacts
  * run `cargo test` on Linux only
* copy built artifacts from the build step, and run tests using the
release martin binary
  * package and publish if this is a release

### Other changes
* Port some minor changes from the rewrite to porting easier
* minor cleanups
* remove all "expected" data files - too unstable to be usable
2022-10-31 16:28:21 -04:00
Yuri Astrakhan
a0a411f088
Consolidate DB init and tests (#463)
* Add justfile to simplify running all the tests
* Save all PBF outputs to the text files
* Consolidate all tests to reuse the same code
* Consolidate database initialization
* updated readme with the new instructions

Note that while this PR creates "expected" files, the CI cannot validate
the generated results because the output is not stable. Eventually we
may try to output just the non-geometry values to have reasonable tests
comparing against the expected results.
2022-10-28 14:52:39 -04:00
Yuri Astrakhan
9b56fe7b9b
Refactor CI tests into separate scripts (#432)
This PR re-uses some ideas by @gbip from #448

* move all CI github workflow tests into the dedicated shell scripts
* consolitade two database initialization scripts into one
2022-10-25 14:54:36 -04:00
Yuri Astrakhan
def4a6f1ec
Warn on unknown cfg, CI cleanup (#461)
* Detect all unrecognized config file values, and report them. Ideally
we want to use `serde-ignored` crate, but it doesn't work with flattened
structs (yet). So using a bad workaround.
* CI test has been using all sorts of somewhat duplicated temporary pbf
files - cleaned up to `tmp.pbf`, and made sure curl only shows errors,
not download stats.
* In CI, crash psql instead of silently ignoring errors
* Don't serialize optional config values as nulls
* Tiny error message cleanup
2022-10-24 19:45:26 +02:00
Pat Sier
f5e19ddf9a
Migrate tests to use maplibre-gl (#449)
Migrates the HTML test files to use `maplibre-gl` instead of `mapbox-gl`. I kept the same version as in `tests/debug-maplibre.html` and used the Carto Positron style on `tests/debug.html` because it was the closet to Mapbox's light style.
2022-10-16 14:56:29 -04:00
Yuri Astrakhan
911076e3bc
Mostly noop code refactorings (#439)
* remove most of the utils:: and dev:: namespace usage
* rename `configure_db_source` to `configure_db_sources`
* use defaults from other table info objects (using the `..obj` syntax)
* use doc comments vs regular comment in a few places
* refactor server_test.rs to make it more readable and much shorter
  * use a macro for identical code
  * use a test_get() for identical GET request
* rename fn test_foo() into fn foo() for simplicity
2022-10-12 11:52:26 +03:00
Yuri Astrakhan
92f768bb3e
Configuration cleanup: step 2 (#425)
A few minor simplifications in the config and appstate:

* default srid seems to be unused.
* simplify Config struct to have non-optional table and function sources. Ok to be empty.
* add a parsing unit test
* rename configs to distinct names for simplicity

I am making this as a separate PR to keep things easier -- the big upcoming PR will use a dynamic dispatch system for all types of sources
2022-10-05 22:12:44 +03:00
Yuri Astrakhan
aed3274162
Breakup config for pg and srv (#420)
Getting ready for multi-backend system, where each backend would have its own configuration.

All 3 config structs (clap args, config, and configbuilder) are now separated into 3 files - the main one, one for service  (actix stuff), and one for PostgreSQL.

The files have been moved to `/pg` and `/srv`, but otherwise the logic should be mostly intact.
2022-10-01 07:48:11 +00:00
Yuri Astrakhan
47ed143d8f
Clean up configuration system to use Clap (#415)
* Use latest Clap-derive (currently v4rc, but should be public within a few days)
* reorganize configuration to streamline different config sources into one Config (using multiple ConfigBuilders)
2022-09-28 04:19:23 -04:00
Yuri Astrakhan
1e8e676e44
feat: use latest actix, switch to bb8 (async everywhere), remove actions (#377)
* remove all actions and other low-level magic code, making it more straightforward for the most common usage
* replace r2d2 with bb8 to make it all async
* use first significant version in cargo.toml - this makes it easier to maintain

This fixes #349
2022-08-15 09:54:48 -04:00
Yuri Astrakhan
7a14d6a3ed
feat!: remove --watch support (#381)
BREAKING CHANGE: Remove --watch support and ignore the command line parameter
2022-08-10 12:23:44 -04:00
Yuri Astrakhan
ad6c5297dd
sort use-statements during fmt (#376)
This does not force automatic use statement sorting,
but it sorts all them now, and we can manually keep them ordered until
the fmt features becomes stable.
2022-08-06 12:48:10 -04:00
Yuri Astrakhan
4a26366a60
chore: use tilejson 0.3.1 and simplify (#341) (h/t @nyurik)
* bounds merging is now built-in into Bounds as Add/AddAssign
* use Bounds::MAX instead of hard-coding it everywhere
2022-05-30 14:06:44 +03:00
Yuri Astrakhan
623f574a10
chore: remove extern, optimize use, fmt str (#338) (h/t @nyurik) 2022-05-29 20:00:49 +03:00
zhangzhang
5e2b6279b9
add maplibre example (#340) (h/t @sharkAndshark) 2022-05-29 20:00:05 +03:00
Yuri Astrakhan
66876d5254
chore: migrate to TileJSON v0.3 (#330) (h/t @nyurik)
* use `tilejson!` to generate json object
* use `Bounds` everywhere when appropriate
2022-05-27 14:04:27 +03:00
Yuri Astrakhan
d0655fd2b3
Use 2021 edition, modern format (#331) (h/t @nyurik)
* switch to 2021 rust edition
* use `format!("{var}")` instead of `"{}", var`
2022-05-27 14:03:48 +03:00
Stepan Kuzmin
b3401785b2
feat: add default SRID support (#308)
* feat: add default SRID support
2022-02-13 15:43:52 +03:00
Stepan Kuzmin
c8141e1c65
test: add function source tests (#271) 2021-10-23 17:02:18 +03:00
Stepan Kuzmin
069c8edf6b
test: add table source tests (#270) 2021-10-23 16:16:50 +03:00
Stepan Kuzmin
fc9170d164
feat: add multiple geometry columns support in table sources (#269)
* feat: add multiple geometry columns support in table sources

* test: add multiple geometry tests
2021-10-21 12:20:33 +03:00
Stepan Kuzmin
f2d56c2f7d
fix: tiles attribute in tilejson with x-rewrite-url (#266) 2021-10-15 18:19:36 +03:00
Stepan Kuzmin
194a83e63f
feat: add minzoom and maxzoom support (#265) 2021-10-13 14:51:29 +03:00
Stepan Kuzmin
206307fd22
test: add config and multiple zoom level tests (#262)
* test: add config and multiple zoom level tests

* test: add more fixtures

* test: use ST_CurveToLine in function source fixtures

* test: update vtzero binaries
2021-10-12 17:56:45 +03:00
Stepan Kuzmin
40b0a0c26a
feat: add bounds to tilejson endpoints (#260) (h/t @jaspervercnocke) 2021-10-10 12:09:56 +03:00
Stepan Kuzmin
1a25129925
chore: improve debug.html (#257) 2021-10-07 19:54:25 +03:00
Paul FLORENCE
49942734af
fix: invalid json escaping (#224) (h/t @gbip)
* Fix invalid json escaping

* test: add test for function_source with query params
2021-07-16 12:09:18 +03:00
Stepan Kuzmin
3c01125fc5
feat: add composite sources support (#184)
* feat: add composite sources support WIP

* feat: handle empty composite sources

* fix: decompose queries

* docs: add docs on composite sources

* ci: push docker image

* test: add composite source tests
2021-04-24 20:19:37 +03:00
AdriSolid
7622731ccf
docs: add deckgl example (#186) (h/t @AdriSolid)
* docs: add deckgl example

* Update README.md

Co-authored-by: Stepan Kuzmin <to.stepan.kuzmin@gmail.com>
2021-02-25 13:33:44 +03:00
Stepan Kuzmin
e953aa35a6
docs: add Leaflet example (#185) 2021-02-23 12:01:45 +03:00
Mazedur Rahman
fce1a9e722
feat: add server heartbeat for monitoring the service health (#168) (h/t @yamaszone)
* Added a simple health check to detect service heartbeat
* Test coverage for healthz endpoint
2021-01-16 16:11:19 +03:00
Stepan Kuzmin
7d5dfa8174
chore: drop mdillon postgis in favor of official image (#157) 2020-12-27 16:35:05 +03:00
Stepan Kuzmin
c1c59e158d
feat: add ssl/tls support #63 (#90)
* feat: add ssl/tls support #63

* feat: add danger-accept-invalid-certs option

* ci: switch to ekidd/rust-musl-builder:stable

* ci: enable docker/build-push-action
2020-06-02 09:49:21 +03:00
Stepan Kuzmin
1e34676a14
ci: refactor CI (#86)
* style: run fmt

* ci: run release only on tags

* ci: switch to actions/checkout@v2

* ci: add grcov
* ci: update docker job
2020-05-05 14:13:48 +03:00
Stepan Kuzmin
ad3076812d
feat: upgrade to actix-web 2.0 (#80)
* chore: update actix-web to 2.0

* fix: use default CORS middleware
2020-04-26 17:57:13 +03:00
Stepan Kuzmin
5a807e40e2
feat: upgrade actix-web to 1.0 (#33)
* test: 💍 add function sources tests

* test: 💍 limit postgres pool size in tests

* feat: 🎸 upgrade actix-web to 1.0
2019-10-26 20:37:49 +03:00
Stepan Kuzmin
3222a1d074
ci: 🎡 add vtzero check 2019-09-29 15:32:19 +03:00
Stepan Kuzmin
2f6cc93745 chore: update dependencies 2019-08-30 18:42:24 +03:00
Stepan Kuzmin
5eeef48b30 feat: 🎸 add watch mode #12
Closes: #12
2019-03-16 18:53:21 +03:00
Stepan Kuzmin
cbe9e89909 docs: ✏️ add notes on debugging 2019-02-04 20:22:04 +03:00
Stepan Kuzmin
ca8b330544 docs: ✏️ add config.yaml example to README.md 2019-02-04 13:40:01 +03:00
Stepan Kuzmin
73d8e86860 chore: update dependencies 2018-10-17 12:11:23 +03:00
Stepan Kuzmin
be2cc3ea6f doc: update README.md 2018-10-12 18:18:34 +03:00
Stepan Kuzmin
33be63e661 doc: update README.md 2018-10-11 15:01:47 +03:00
Stepan Kuzmin
12f1db9835 chore: add debug page 2018-10-09 15:00:25 +03:00
Stepan Kuzmin
334cf72107 test: add table and function source tests 2018-08-25 16:43:31 +03:00
Stepan Kuzmin
ccc986d037 test: add sources_not_found_test 2018-08-25 15:45:04 +03:00