* [BREAKING] Use source ID (table name) as the default layer ID, instead
of `schema.table.column`
* Add support for the optional `layer_id` table config parameter
Fix#595
* make tilejson's `name` be the same as the ID of the source (even if
aliased)
* `/catalog` will always show ID, but now it will hide the `name` if it
is the same as the `id`
* make `description` be the longer version, e.g. `public.table.column`
format - not guaranteed to be stable
* make `vector_layers` have the fields auto-discovered in the PG table
* preserve the order of the serialized json fields
Fixes#583
Compression middleware turned out to be hard to use for image cases - it
simply looks at the content-encoding, and if not set, tries to compress
if accepted by the client.
Instead, now individual routes are configured with either that
middleware, or for tiles, I decompress and optionally recompress if
applicable.
Now encoding is tracked separately from the tile content, making it
cleaner too. Plus lots of tests for mbtiles & pmtiles.
Fixes#577
Implements #384 - ability to limit the number of features included in a
tile from a Postgres table/view.
This allows zoomed-out view of a table with a reasonable speed because
each tile could be limited in size, rather than include millions of
features.
If set on a CLI, overrides whatever is set in the config file (if
given).
Any naming suggestions?
DeadPool seem to be better maintained, and most importantly, it properly
reports connection errors - something bb8 does not support yet.
I also removed connection timeout config param - it was introduced while
trying to solve bb8 issue - not needed because connection string
supports it already.
This should resolve some of the issues in #496
* Adds a view to `points1.sql` fixture
* Replaces `table` with `view` in log statements relating to views
---------
Co-authored-by: Chris Thiange <cthiange@gmail.com>
Warn users when a PG table geometry column has no index - thus accessing it would be slow. This is only done for tables. Issues with the views are not printed.
## Implementation
This adds two fields to `TableInfo`:
* `geom_idx: Option<bool>` to tell if a geo column has a spatial index
* `is_view: Option<bool>` to distinguish views from other relations
Missing spatial index warnings are logged for non-view relations. Views
will never have indexed columns and, if referencing a table with a
missing index, it will be logged already.
Couldn't figure out how to make `just test` accept the new warning (from
missing index), so I have them logged as INFO for now :)
fixes#540
---------
Co-authored-by: Christophe Thiange <cthiange@gmail.com>
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
This is a partial fix for #496
* BREAKING: Now Martin behaves the same way as `psql` -- by default, if
SSL is available on the server, it will be used, even though it will not
verify that the server has a valid SSL certificate
* Martin now understands `PGSSLCERT`, `PGSSLKEY`, and `PGSSLROOTCERT`
env vars (and corresponding config keys) - same as psql.
* Martin can now process `?sslmode=verify-ca` and `verify-full` (just
like psql). The verify modes require root and/or client cert & key.
* remove `danger_accept_invalid_certs` -- turns out that behavior is
expected by default unless ssl mode is set to verify - which upstream
lib [does not
support](https://github.com/sfackler/rust-postgres/issues/768) - PR
[submitted](https://github.com/sfackler/rust-postgres/pull/988).
* added connection_timeout_ms option for postgres and set it to 5
seconds by default. This way it will fail out earlier.
* added error reporting to bb8 - but it is currently [broken
upstream](https://github.com/djc/bb8/issues/151) - not sure we can fix
it easily, so may need to switch to deadpool later.
* added docker-based TLS test (horray!) - wasn't trivial at all, despite
ending up fairly simple.
* clean up reporting of the un-used config params - instead of printing,
collect them and print in one place if needed (allows testing too)
* remove `vector_layer` in catalog - too verbose, not needed - can be
received via tilejson for individual source
* clean up tests so that they all use the same config yaml
Adds a new [.mbtiles](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md)
backend, without the grid support. Uses extensive tile content
detection, i.e. if the content is gzipped, png, jpeg, gif, webp.
From CLI, can be as easy as adding a path to a directory that contains a
.mbtiles file (works just like pmtiles support)
```bash
# All *.mbtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```
From configuration file, the path can be specified in a number of ways
(same as pmtiles)
```yaml
mbtiles:
paths:
# scan this whole dir, matching all *.mbtiles files
- /dir-path
# specific mbtiles file will be published as mbtiles2 source
- /path/to/mbtiles2.mbtiles
sources:
# named source matching source name to a single file
pm-src1: /tmp/mbtiles.mbtiles
# named source, where the filename is explicitly set. This way we will be able to add more options later
pm-src2:
path: /tmp/mbtiles.mbtiles
```
Fixes#494
Merge after #548
Adds a new [.pmtiles](https://protomaps.com/docs/pmtiles/) backend.
Supports all formats like png, vector, etc.
From CLI, can be as easy as adding a path to a directory that contains a
.pmtiles file:
```bash
# All *.pmtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```
From configuration file, the path can be specified in a number of ways:
```yaml
pmtiles:
paths:
# scan this whole dir, matching all *.pmtiles files
- /dir-path
# specific pmtiles file will be published as pmtiles2 source
- /path/to/pmtiles2.pmtiles
sources:
# named source matching source name to a single file
pm-src1: /tmp/pmtiles.pmtiles
# named source, where the filename is explicitly set. This way we will be able to add more options later
pm-src2:
path: /tmp/pmtiles.pmtiles
```
Fixes#508
* introduce a new Connections object to track all positional strings
passed as the CLI arguments
* each tile provider can now indicate if it can take a positional CLI
arg, and if the value can be shared between multiple providers, i.e. if
its a directory that could contain files for multiple providers
* make xyz use better types - u8 for zoom, u32 for x&y. Postgres casts
those to INT2 and INT8
* minor bug in pre-push git hook to abort in case of a testingerror
* added GIF detection/type
* combine MVT and compression concepts into one enum more explicitly. It
is not ideal (technically they are separate concerns), but it keeps it a
bit simpler for now for multiple providers.
* set content encoding and content type on HTTP responses if known, and
also include them in the `/catalog` response (json)
* raise an error if the user attempts to merge non-concatenatable tiles
from multiple sources. We may want to implement it in the future, e.g.
combine multiple semi-transparent PNGs. Or even combine GIF & PNG & JPEG
* do not set content-type on empty responses (http 204)
* add tilejson outputs to testing
* NEW: support for #512 - pg table/function auto-discovery
* can filter schemas
* can use patterns like `{schema}.{table}.{column}` and
`{schema}.{function}`
* NEW: add `disable_bounds` bool flag to allow disabling of the bounds
computation
* reworked integration tests to use yaml
* fixed SQL to work on older PG versions
* re-enable CI to test expected `test.sh` output against the one stored
in the `tests/expected`
* add postgres in docker tests on linux - one for the oldest supported
DB, and another using the more recent version
* minor justfile cleanup
* ensure config files are sorted alphabetically
* added manual coverage justfile command
* a lot of small refactorings of config and argument parsing
* feature: support jsonb query param for functions
* cleaned up public/private access
* make all tests populate with a predefined values to avoid issues with
random data
* Support `postgres` config key to be either a list or an object
* Use `PathBuf` instead of `String` where dealing with files
* Merge `SrvConfigBuilder` into `SrvConfig`
* Parse keep_alive as u64
* More config tests
Pedantic lints often offer some good insight into the code. It is
usually easier to sprinkle a few "allow"-s around, than to miss some
important life hack offered by clippy.
Also, make use a different martin port when running integration tests
locally (make sure `git push` works even if martin is running).
* Ensure the source ID only contains ascii alphanumerics, dots, dashes,
and underscores.
* optimize the most common case of getting a single source
* optimize the case of sources with no query parameters
* Since this is a library, all errors should have a strongly typed enum.
* table bounds computing function was not escaping identifiers
* table bounds computation was also silently ignoring all errors
This extracts some of the code from #511 but without breaking changes
* Use `PathBuf` instead of `String` where dealing with files
* Parse keep_alive as u64
* More config tests to crash if martin output contains warnings or
errors
Can now handle several additional Postgres functions to get a tile, plus
tons of small fixes
### Multiple result variants
* `getmvt(z,x,y) -> [bytea,md5]` (single row with two columns)
* `getmvt(z,x,y) -> [bytea]` (single row with a single column)
* `getmvt(z,x,y) -> bytea` (value)
### Multiple input parameter variants
* `getmvt(z, x, y)` or `getmvt(zoom, x, y)` (all 3 vars must be
integers)
* `getmvt(z, x, y, url_query)`, where instead of `url_query` it could be
any other name, but must be of type JSON
### Breaking
* srid is now the same type as PG -- `i32`
* renamed config vals `table_sources` and `function_sources` into
`tables` and `functions`
### Features and fixes
* if postgis is v3.1+, uses margin parameter to extend the search box by
the size of the buffer. I think we should make 3.1 minimal required.
* fixes feature ID issue from #466
* fixes mixed case names for schemas, tables and columns, functions and
parameter names per #389
### Notes
* More dynamic SQL generation in code instead of using external SQL
files. Those should only be used when they are not parametrized.
* The new function/table discovery mechanism: query for all functions in
the database, and match up those functions with the ones configured (if
any), plus adds all the rest of the un-declared ones if discovery mode
is on.
* During table and function discovery, the code generates a map of
`(PgSqlInfo, FunctionInfo)` (or table) tupples containing SQL needed to
get the tile.
* Auto-discovery mode is currently hidden - the discovery is on only
when no tables or functions are configured. TBD - how to configure it in
the future
* The new system allows for an easy way to auto-discover for the
specific schemas only, solving #47
* predictable order of table/function instantiation
* bounding boxes computed in parallel for all tables (when not
configured)
* proper identifier escaping
* test cleanup
fixes#378fixes#466fixes#65fixes#389
* All tests and internal code now uses ST_TileEnvelope function
* Remove `tile_bbox`
* Rename test function sources for clarity - this will be needed in a
subsequent PR to add other function tests
* Fix incorrect printing of the xyz value - implements Display trait
instead to keep it simpler
* Remove the possibly-incorrect macro capture of a simple string - in
case the string contains inline vars
* Handle optional extra comma properly
* Rename it to a shorter name
* Inline vars where possible
Partial implementation of the #430
* New endpoint structure:
* `GET /` -- a placeholder for the future home page
* `GET /catalog` -- get a list of available sources, as a list of json
blobs.
* `[{id, name, description, attribution, vector_layer}, ...]` (some
fields might be missing)
* `GET /<id>` -- get tilejson for the given source, or a combination of
sources. No `.json` extension
* `GET /<id>/<z>/<x>/<y>` -- get a tile. No format extension.
* `GET /health` -- healthcheck
* Introduce a new tile format support crate (using code from the
maplibre/mbtileserve project)
* Removed the `/rpc/...` routes - all source IDs are accessed in the
same way
* Can print auto-generated configuration or save it to a file
* Refactored to support multiple sources from multiple backends, with a
proper naming conflict resolution
TODO:
* benchmarks need to be rewritten - they were relying on some internal
structures that are no longer there. This might be done as a separate PR
due to a very different internal architecture - might need to rethink
benchmarking approaches.
By default, Martin is now compiled without openssl, simplifying
debugging and simple case usage, whereas the docker build and CI
publishing would still use openssl
Rework CI to run tests locally using the VM-installed Postgres on all
target platforms.
### CI jobs
* Build release versions on Linux/Win/Mac and save build results as
output artifacts
* In a separate VMs (Linux/Win/Mac)
* use
[nyurik/action-setup-postgis](https://github.com/nyurik/action-setup-postgis)
to install postgis and run tests using the built artifacts
* run `cargo test` on Linux only
* copy built artifacts from the build step, and run tests using the
release martin binary
* package and publish if this is a release
### Other changes
* Port some minor changes from the rewrite to porting easier
* minor cleanups
* remove all "expected" data files - too unstable to be usable
* Detect all unrecognized config file values, and report them. Ideally
we want to use `serde-ignored` crate, but it doesn't work with flattened
structs (yet). So using a bad workaround.
* CI test has been using all sorts of somewhat duplicated temporary pbf
files - cleaned up to `tmp.pbf`, and made sure curl only shows errors,
not download stats.
* In CI, crash psql instead of silently ignoring errors
* Don't serialize optional config values as nulls
* Tiny error message cleanup
* remove most of the utils:: and dev:: namespace usage
* rename `configure_db_source` to `configure_db_sources`
* use defaults from other table info objects (using the `..obj` syntax)
* use doc comments vs regular comment in a few places
* refactor server_test.rs to make it more readable and much shorter
* use a macro for identical code
* use a test_get() for identical GET request
* rename fn test_foo() into fn foo() for simplicity
A few minor simplifications in the config and appstate:
* default srid seems to be unused.
* simplify Config struct to have non-optional table and function sources. Ok to be empty.
* add a parsing unit test
* rename configs to distinct names for simplicity
I am making this as a separate PR to keep things easier -- the big upcoming PR will use a dynamic dispatch system for all types of sources
Getting ready for multi-backend system, where each backend would have its own configuration.
All 3 config structs (clap args, config, and configbuilder) are now separated into 3 files - the main one, one for service (actix stuff), and one for PostgreSQL.
The files have been moved to `/pg` and `/srv`, but otherwise the logic should be mostly intact.
* Use latest Clap-derive (currently v4rc, but should be public within a few days)
* reorganize configuration to streamline different config sources into one Config (using multiple ConfigBuilders)
Use attribute-based routing, and support HEAD requests (same as GET)
attribute routes are better because they keep route declaration (i.e. parameters) together with the code that uses it.
* remove all actions and other low-level magic code, making it more straightforward for the most common usage
* replace r2d2 with bb8 to make it all async
* use first significant version in cargo.toml - this makes it easier to maintain
This fixes#349
This does not force automatic use statement sorting,
but it sorts all them now, and we can manually keep them ordered until
the fmt features becomes stable.
