Commit Graph

62 Commits

Author SHA1 Message Date
Yuri Astrakhan
b71c846abc
fix for mbtiles apply-patch with raw bindiff (#1384)
When applying raw (no gzip) patches, `mbtiles` was trying to un-gzip
them first. Now handles it properly.  Also adds a number of tests to catch these cases.
2024-06-26 20:37:20 +00:00
Yuri Astrakhan
ccd550a2cf
Add binary diff for mbtiles (#1358)
* `mbtiles diff` now has an additional `--patch-type` param with
`whole`, `bin-diff-raw` and `bin-diff-gz` values:
* `whole` stores different tiles as before - as whole tiles in the
`tiles` table
* `bin-diff-raw` computes binary difference between tiles, and stores
them as brotli-encoded value in a `bsdiffraw` table, together with a
`xxh3_64` hash of the tile as it will be stored after patching
* `bin-diff-gz` same as `bin-diff-raw`, but assumes the tiles are
gzip-compressed, so it uncompresses them before comparing. The `xxh3_64`
stores the hash of the uncompressed tile. The data will be stored in the
`bsdiffrawgz` table (identical structure with above)

* `mbtiles copy --apply-patch` will automatically detect if
`bsdiffrawgz` or `bsdiffraw` tables exist, and will use binary patching.
* `mbtiles apply-patch` does not support binary patching yet
* `mbtiles copy --diff-with-file ... --patch-type ...` is an alias to
`mbtiles diff --patch-type ...`
2024-06-25 19:34:14 +00:00
Yuri Astrakhan
6320e0fff3
Add agg_tiles_hash_before_apply, warnings, and validate on patch (#1266)
Implement #1244
2024-05-30 18:28:34 +00:00
Yuri Astrakhan
d77533f627
Fix CI: use new nginx and postgis actions (#1351)
I recently released a new nginx github action that should work on all OS
and in all runners. Also, postgis action has been updated.
2024-05-24 22:41:26 -04:00
Yuri Astrakhan
b28dae4466
chore: add empty pg table test (#1259) 2024-03-16 22:45:43 -04:00
Yuri Astrakhan
9505a2107e
chore: increase complexity of the SQL table test (#1256)
a few more characters that have to be properly escaped
2024-03-16 04:23:35 -04:00
Lucas
656728cde1
Add test for function with special character (#1231) 2024-03-09 08:30:57 +08:00
Yuri Astrakhan
c0ec365d5c
Added a test for schema.table.column containing special chars (#1222)
This just adds a test for the unusual case of a table with quotes,
spaces, and dots in their identifiers. Another similar test should be
added for functions.
2024-03-04 02:55:21 -05:00
Yuri Astrakhan
fba9af8749
Bump locks, blessed expected tests (#1212) 2024-02-25 05:46:59 -05:00
Yuri Astrakhan
54f5a8daf0
Add tiles table/view validation to mbtiles validate (#1057)
Make sure all values in the `tiles` table or view are correct:

* zoom_level is between 0 and 30 (max allowed zoom)
* the x,y values are within the limits for the corresponding zoom level
* the column type of z,x,y are all integers
* the `tile_data` is a NULL or a BLOB
2023-12-10 10:59:14 +08:00
Yuri Astrakhan
f88db05582
Fix SQL comments when func or table is pre-configured (#1045)
When a SQL comment is set on a table or a function to customize
tilejson, and that tbl/func is pre-configured as part of the config
file, the comment was silently ignored. Now both table and function
cases are handled correctly.

Also, update docs to not include function parameters - makes SQL example
a bit simpler.

Thanks @jjcfrancisco for reporting!

Fixes: #1044
2023-12-07 07:24:51 +00:00
Yuri Astrakhan
713a51b74d
Upgarde to v0.10 spreet crate, cleanup SpreetResult (#984) 2023-11-29 19:09:05 -05:00
Yuri Astrakhan
f7c69f8302
Minor cleanup and just reorg (#992)
Only run git push diff on linux to make development on non-linux easier,
and a few preparations for pmtiles over http serving

Fixes #855
2023-11-08 19:46:23 -05:00
Yuri Astrakhan
9b112ae7b9
Implement dynamic font support /font/<name>/<start>-<end> (#755)
This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles

This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.

### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
  * convert them to SDFs and package them into a protobuf

---------

Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
2023-10-28 05:10:48 +00:00
Yuri Astrakhan
8b34cd374c
Add metadat copy/apply-diff, new testing framework (#921)
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.

## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content

There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).

## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
2023-10-10 11:10:17 -04:00
Yuri Astrakhan
64d652f5af
Improve SSL mode testing (#913) 2023-09-30 22:49:56 -04:00
Yuri Astrakhan
ae8e0709d7 ssl dockercompose test config 2023-09-30 12:38:57 -04:00
Yuri Astrakhan
d1fe026639 Fix justfile ssl test, print hba config 2023-09-30 12:06:49 -04:00
Yuri Astrakhan
6f08aa9465
Fix mbtiles validation, CI, and logging (#903) 2023-09-29 14:37:18 -04:00
Yuri Astrakhan
1a386b7071 Fix unit tests to use blobs only 2023-09-28 16:57:50 -04:00
Lucas
fa1bf4bdbd
Retrieve table/view comments as tilejson (#871) 2023-09-12 12:59:59 +08:00
Yuri Astrakhan
0fce9641c0
A few cleanups on tile hashing (#861)
* Rename `global_hash` to `agg_tiles_hash`

This is still a big sticking point: what should be the name for the
metadata key for this value? The value represents the hash of all
`z,x,y,tile` over all rows of the `tiles` table (or view). Should it
include `md5` in its name, or should the hash be auto-detected by its
length? (details in #856)

* Generate it based on `tiles` table/view
* validate or generate, but not both (it will always fail otherwise)
* break up logic for per-tile, total, and integrity checks
* delete unused sqlx prep file
2023-09-05 16:25:59 -04:00
rstanciu
875a942fcf
Add integrity and global hash check to validation (#851)
Co-authored-by: rstanciu <rstanciu@rivian.com>
2023-08-31 03:40:40 +00:00
Lucas
b3fb720a94
Retrieve function source description from comments (#829)
If a PostgreSQL function has an SQL comment, it will try to parse as
JSON and use its values to override the auto-generated TileJSON. It is
recommended to use this form when creating comments to ensure valid JSON
values.

```sql
DO $do$ BEGIN
    EXECUTE 'COMMENT ON FUNCTION YOUR_FUNCTION (ARG1_TYPE,ARG2_TYPE,..ARGN_TYPE) IS $tj$' || $$
    {
      "description": "description override",
      ...
    }
    $$::json || '$tj$';
END $do$;
```

Partially implements #822
2023-08-27 16:46:56 +00:00
rstanciu
720b682b7e
Add flat hashing table format (#808)
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-08-16 10:22:01 -07:00
Binabh
e3e6b3563f
adds support for id_column in tables for auto_publish (#790)
Resolves #682 

- [x] Get id_column string from config.yaml and use for id column
- [x] Support for list of strings
- [x] Add info/warnings if column is not there or is of wrong type
- [x] if column for the feature ID is found, remove it from properties
(see inline comment)
- [x] cleanup logging messages
- [x] need more tests to catch other edge cases

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-08-13 02:06:23 +00:00
Yuri Astrakhan
14ded482ac add sql test pg_dump info 2023-08-12 21:30:04 -04:00
Yuri Astrakhan
405d4b27a2
Improve auto-publish usability (#795)
* on `--save-config`, only save configured `auto_publish` settings
* alias `from_schemas` as `from_schema`
* add integration testing for `auto_publish`
* if integration test DB preloading fails, try to clean up the test DB
* A few more info traces

This change should benefit testing of the #790 cc: @Binabh
2023-08-03 23:51:10 +02:00
rstanciu
1342b38e75
Add apply-diff command (#747)
Add command `apply-diff` to apply diff file generated from `copy`
command
2023-07-05 21:38:03 +00:00
rstanciu
e004908722
Add diff-with-file option to copy command (#729)
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-07-05 16:55:39 +00:00
Yuri Astrakhan
ff29530804
Fix non-deterministic CI tests (#724)
* a race condition due to duplicated webp.mbtiles and webp.pmtiles
* minor cleanup of CI/dockerignore
* dependency bump
2023-06-22 23:01:57 -04:00
Yuri Astrakhan
a5c5505713
Add dynamic sprites support (#715)
Dynamically create image sprites for MapLibre rendering, given a
directory with images.

### TODO
* [x] Work with @flother to merge these PRs
  * [x] https://github.com/flother/spreet/pull/59  (must have)
  * [x] https://github.com/flother/spreet/pull/57
  * [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use

### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`

Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.

### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:

```yaml
# Publish sprite images
sprites:
  paths:
    # scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
    - /path/to/my_images
  sources:
    # named source matching source name to a directory
    my_sprites: /path/to/some_dir
```

Implement #705
2023-06-16 08:19:47 -04:00
Yuri Astrakhan
454d435589 Move invalid test fixture files 2023-06-03 23:06:37 -04:00
Yuri Astrakhan
e92722708f
PG TileJSON changes, add vector_layers (#584)
* make tilejson's `name` be the same as the ID of the source (even if
aliased)
* `/catalog` will always show ID, but now it will hide the `name` if it
is the same as the `id`
* make `description` be the longer version, e.g. `public.table.column`
format - not guaranteed to be stable
* make `vector_layers` have the fields auto-discovered in the PG table
* preserve the order of the serialized json fields

Fixes #583
2023-02-22 16:25:48 +00:00
Yuri Astrakhan
3fcad46500
fix (re-)compression of images (#579)
Compression middleware turned out to be hard to use for image cases - it
simply looks at the content-encoding, and if not set, tries to compress
if accepted by the client.

Instead, now individual routes are configured with either that
middleware, or for tiles, I decompress and optionally recompress if
applicable.

Now encoding is tracked separately from the tile content, making it
cleaner too. Plus lots of tests for mbtiles & pmtiles.

Fixes #577
2023-02-20 10:44:22 -05:00
Chris Thiange
22bf2008e5
Views handling (#571)
* Adds a view to `points1.sql` fixture
* Replaces `table` with `view` in log statements relating to views

---------

Co-authored-by: Chris Thiange <cthiange@gmail.com>
2023-02-07 20:45:01 -05:00
cthiange
612a8d38c4
Issue 540 - missing index warnings (#569)
Warn users when a PG table geometry column has no index - thus accessing it would be slow.  This is only done for tables. Issues with the views are not printed.

## Implementation

This adds two fields to `TableInfo`:
* `geom_idx: Option<bool>` to tell if a geo column has a spatial index
* `is_view: Option<bool>` to distinguish views from other relations

Missing spatial index warnings are logged for non-view relations. Views
will never have indexed columns and, if referencing a table with a
missing index, it will be logged already.

Couldn't figure out how to make `just test` accept the new warning (from
missing index), so I have them logged as INFO for now :)

fixes #540

---------

Co-authored-by: Christophe Thiange <cthiange@gmail.com>
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-02-07 01:05:47 -05:00
Yuri Astrakhan
9e5ed2fc02
Add .mbtiles support (#549)
Adds a new [.mbtiles](https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md)
backend, without the grid support. Uses extensive tile content
detection, i.e. if the content is gzipped, png, jpeg, gif, webp.

From CLI, can be as easy as adding a path to a directory that contains a
.mbtiles file (works just like pmtiles support)

```bash
# All *.mbtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways
(same as pmtiles)

```yaml
mbtiles:
  paths:
    # scan this whole dir, matching all *.mbtiles files
    - /dir-path
    # specific mbtiles file will be published as mbtiles2 source
    - /path/to/mbtiles2.mbtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/mbtiles.mbtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/mbtiles.mbtiles
```

Fixes #494
2023-01-09 00:10:23 -05:00
Yuri Astrakhan
ba65e34bef
Add .pmtiles file support (#517)
Merge after #548 

Adds a new [.pmtiles](https://protomaps.com/docs/pmtiles/) backend.
Supports all formats like png, vector, etc.

From CLI, can be as easy as adding a path to a directory that contains a
.pmtiles file:

```bash
# All *.pmtiles files in this dir will be published.
# The filename will be used as the source ID
martin ./tests/fixtures
```

From configuration file, the path can be specified in a number of ways:

```yaml
pmtiles:
  paths:
    # scan this whole dir, matching all *.pmtiles files
    - /dir-path
    # specific pmtiles file will be published as pmtiles2 source
    - /path/to/pmtiles2.pmtiles
  sources:
      # named source matching source name to a single file
      pm-src1: /tmp/pmtiles.pmtiles
      # named source, where the filename is explicitly set. This way we will be able to add more options later
      pm-src2:
        path: /tmp/pmtiles.pmtiles
```

Fixes #508
2023-01-08 14:19:11 -05:00
Yuri Astrakhan
928a700150
Implement postgres auto-publish (#546)
* NEW: support for #512 - pg table/function auto-discovery
  * can filter schemas
* can use patterns like `{schema}.{table}.{column}` and
`{schema}.{function}`
* NEW: add `disable_bounds` bool flag to allow disabling of the bounds
computation
* reworked integration tests to use yaml
2023-01-03 16:09:41 +00:00
Yuri Astrakhan
555a1fccdd
Improve Coverage tooling (#535)
* added manual coverage justfile command
* a lot of small refactorings of config and argument parsing
* feature: support jsonb query param for functions
* cleaned up public/private access
* make all tests populate with a predefined values to avoid issues with
random data
2022-12-27 06:56:27 +00:00
Yuri Astrakhan
9efa364eb0
Fix handling for null-returning PG queries (#521)
Handle cases when a query returns a NULL or a table with no rows, or a
single row with a null value in it.

This fully fixes #519 in the main branch
2022-12-15 14:12:55 +02:00
Yuri Astrakhan
2ee517d135
Support z,x,y and record-returning funcs, table rework (#380)
Can now handle several additional Postgres functions to get a tile, plus
tons of small fixes

### Multiple result variants
* `getmvt(z,x,y) -> [bytea,md5]`  (single row with two columns)
* `getmvt(z,x,y) -> [bytea]` (single row with a single column)
* `getmvt(z,x,y) -> bytea` (value)

### Multiple input parameter variants
* `getmvt(z, x, y)` or `getmvt(zoom, x, y)` (all 3 vars must be
integers)
* `getmvt(z, x, y, url_query)`, where instead of `url_query` it could be
any other name, but must be of type JSON

### Breaking
* srid is now the same type as PG -- `i32`
* renamed config vals `table_sources` and `function_sources` into
`tables` and `functions`

### Features and fixes
* if postgis is v3.1+, uses margin parameter to extend the search box by
the size of the buffer. I think we should make 3.1 minimal required.
* fixes feature ID issue from #466
* fixes mixed case names for schemas, tables and columns, functions and
parameter names per #389
 

### Notes
* More dynamic SQL generation in code instead of using external SQL
files. Those should only be used when they are not parametrized.
* The new function/table discovery mechanism: query for all functions in
the database, and match up those functions with the ones configured (if
any), plus adds all the rest of the un-declared ones if discovery mode
is on.
* During table and function discovery, the code generates a map of
`(PgSqlInfo, FunctionInfo)` (or table) tupples containing SQL needed to
get the tile.
* Auto-discovery mode is currently hidden - the discovery is on only
when no tables or functions are configured. TBD - how to configure it in
the future
* The new system allows for an easy way to auto-discover for the
specific schemas only, solving #47
* predictable order of table/function instantiation
* bounding boxes computed in parallel for all tables (when not
configured)
* proper identifier escaping
* test cleanup

fixes #378
fixes #466
fixes #65
fixes #389
2022-12-10 16:20:42 +02:00
Yuri Astrakhan
c1b65796f6
Require PostGIS 3 and use ST_TileEnvelope (#497)
* All tests and internal code now uses ST_TileEnvelope function
* Remove `tile_bbox`
* Rename test function sources for clarity - this will be needed in a
subsequent PR to add other function tests
2022-11-30 18:57:27 +02:00
Yuri Astrakhan
b8ac719be8
Rework and consolidate CI (#467)
Rework CI to run tests locally using the VM-installed Postgres on all
target platforms.

### CI jobs
* Build release versions on Linux/Win/Mac and save build results as
output artifacts
* In a separate VMs (Linux/Win/Mac)
* use
[nyurik/action-setup-postgis](https://github.com/nyurik/action-setup-postgis)
to install postgis and run tests using the built artifacts
  * run `cargo test` on Linux only
* copy built artifacts from the build step, and run tests using the
release martin binary
  * package and publish if this is a release

### Other changes
* Port some minor changes from the rewrite to porting easier
* minor cleanups
* remove all "expected" data files - too unstable to be usable
2022-10-31 16:28:21 -04:00
Yuri Astrakhan
a0a411f088
Consolidate DB init and tests (#463)
* Add justfile to simplify running all the tests
* Save all PBF outputs to the text files
* Consolidate all tests to reuse the same code
* Consolidate database initialization
* updated readme with the new instructions

Note that while this PR creates "expected" files, the CI cannot validate
the generated results because the output is not stable. Eventually we
may try to output just the non-geometry values to have reasonable tests
comparing against the expected results.
2022-10-28 14:52:39 -04:00
Yuri Astrakhan
9b56fe7b9b
Refactor CI tests into separate scripts (#432)
This PR re-uses some ideas by @gbip from #448

* move all CI github workflow tests into the dedicated shell scripts
* consolitade two database initialization scripts into one
2022-10-25 14:54:36 -04:00
Stepan Kuzmin
b3401785b2
feat: add default SRID support (#308)
* feat: add default SRID support
2022-02-13 15:43:52 +03:00
Stepan Kuzmin
fc9170d164
feat: add multiple geometry columns support in table sources (#269)
* feat: add multiple geometry columns support in table sources

* test: add multiple geometry tests
2021-10-21 12:20:33 +03:00
Stepan Kuzmin
206307fd22
test: add config and multiple zoom level tests (#262)
* test: add config and multiple zoom level tests

* test: add more fixtures

* test: use ST_CurveToLine in function source fixtures

* test: update vtzero binaries
2021-10-12 17:56:45 +03:00