Commit Graph

24 Commits

Author SHA1 Message Date
rstanciu
720b682b7e
Add flat hashing table format (#808)
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)

---------

Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-08-16 10:22:01 -07:00
Yuri Astrakhan
fa50eeac97
Update and fix demo (#809)
* Use `postgis/postgis` image for the DB
* Build frontend image inside the docker
* Upgrade all packages except styled-components - that one causes some
errors
* Add yarn.lock to the repo
* Add justfile for simplicity
* Bump martin docker image version in a few places
2023-08-15 14:41:12 +02:00
Yuri Astrakhan
aa3db46599
Add metrics to reserved list, docs (#802)
minor change to allow for #793
2023-08-13 02:33:56 -04:00
Binabh
e3e6b3563f
adds support for id_column in tables for auto_publish (#790)
Resolves #682 

- [x] Get id_column string from config.yaml and use for id column
- [x] Support for list of strings
- [x] Add info/warnings if column is not there or is of wrong type
- [x] if column for the feature ID is found, remove it from properties
(see inline comment)
- [x] cleanup logging messages
- [x] need more tests to catch other edge cases

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-08-13 02:06:23 +00:00
Yuri Astrakhan
74c84fab1f
rename id_format to source_id_format, refactor (#794)
* rename configuration `auto_publish.tables.id_format` and
`auto_publish.functions.id_format` fields from `id_format` to
`source_id_format` fields. The `id_format` will continue to be supported
(read) from the configuration, but it will be auto-converted to the new
name on save. It is an error to have both in the same config file.
  * The rename was discussed in #682

* internal refactorings: consolidate PG-related utilities, rename a few
vars, move PG errors to their own file.

This is partially made due to #790 (thanks @Binabh!) - and should be
merged before that to make that PR easier.
2023-08-03 01:13:38 +00:00
Yuri Astrakhan
ed7d33a76d
Cleanup docker and related docs (#752)
* Use same docker image for postgis & psql
* Improve docker book
* rename just target `mdbook` to `book` (more obvious choice)
* Cleanup justfile to not duplicate "cargo install" check & installation
2023-07-07 20:44:30 +00:00
Yuri Astrakhan
24d43603c1 book cleanup 2023-07-06 14:11:42 -04:00
rstanciu
1342b38e75
Add apply-diff command (#747)
Add command `apply-diff` to apply diff file generated from `copy`
command
2023-07-05 21:38:03 +00:00
rstanciu
e004908722
Add diff-with-file option to copy command (#729)
* Add ability to generate diff file by specifying `--diff-with-file` to
the `copy` tool

---------

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-07-05 16:55:39 +00:00
Yuri Astrakhan
87b62cb223 update sources-sprites docs, minor lint 2023-06-22 23:53:59 -04:00
Yuri Astrakhan
71f9ee6c7d Ver bump, doc cleanup, readying for release 2023-06-22 23:24:31 -04:00
rstanciu
bc140e0216
Add basic copying functionality (#712)
Copy an existing `.mbtiles` file to a new file.

```shell
mbtiles copy  <src_file.mbtiles> <dst_file.mbtiles>
```

Optionally filters by zooms.  Supports de-duplicated and simple mbtiles

---------

Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
2023-06-17 00:00:46 +00:00
Yuri Astrakhan
a5c5505713
Add dynamic sprites support (#715)
Dynamically create image sprites for MapLibre rendering, given a
directory with images.

### TODO
* [x] Work with @flother to merge these PRs
  * [x] https://github.com/flother/spreet/pull/59  (must have)
  * [x] https://github.com/flother/spreet/pull/57
  * [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use

### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`

Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.

### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:

```yaml
# Publish sprite images
sprites:
  paths:
    # scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
    - /path/to/my_images
  sources:
    # named source matching source name to a directory
    my_sprites: /path/to/some_dir
```

Implement #705
2023-06-16 08:19:47 -04:00
Yuri Astrakhan
6d02416720
Improve PG performance by 28% (!!!) (#703)
A very long overdue PostgreSQL querying performance optimization that
should have used cached queries, but ... somehow didn't.

Also, this PR adds two new `just` tasks: `run-release` and `bench-http`

I used [oha](https://github.com/hatoo/oha) for its visual appeal. All
tests were using keep-alive, which I think is relatively accurate
because clients make many tile requests on the same connection. As a
target, I used the same non-empty small tile to reduce the PostgreSQL
indexing load.

❯ just run-release
❯ just bench-http

`bench-http` runs this command:
```
oha -z 120s http://localhost:3000/function_zxy_query/18/235085/122323
```

<pre>
|       before the change          |         after the change         |
|----------------------------------|----------------------------------|
|   Summary:                       | Summary:                         |
|     Success rate:    1.0000      |   Success rate:    1.0000        |
|     Total:    120.0004 secs      |   Total:    120.0002 secs        |
|     Slowest:    0.1339 secs      |   Slowest:    0.3505 secs        |
|     Fastest:    0.0015 secs      |   Fastest:    0.0012 secs        |
|     Average:    0.0076 secs      |   Average:    0.0055 secs        |
|     Requests/sec:    6583.6946   |   Requests/sec:    9073.5398     |
|                                  |                                  |
|     Total data:    113.02 MiB    |   Total data:    155.76 MiB      |
|     Size/request:    150 B       |   Size/request:    150 B         |
|     Size/sec:    964.41 KiB      |   Size/sec:    1.30 MiB          |
|                                  |                                  |
|   Response time histogram:       | Response time histogram:         |
|     0.002 [1]                    |   0.001 [1]                      |
|     0.015 [785706] ■■■■■■■■■■■■■ |   0.036 [1088825] ■■■■■■■■■■■■■  |
|     0.028 [4225]                 |   0.071 [0]                      |
|     0.041 [111]                  |   0.106 [0]                      |
|     0.054 [2]                    |   0.141 [0]                      |
|     0.068 [0]                    |   0.176 [0]                      |
|     0.081 [0]                    |   0.211 [0]                      |
|     0.094 [0]                    |   0.246 [0]                      |
|     0.107 [0]                    |   0.281 [0]                      |
|     0.121 [0]                    |   0.316 [0]                      |
|     0.134 [1]                    |   0.350 [1]                      |
|                                  |                                  |
|   Latency distribution:          | Latency distribution:            |
|     10% in 0.0057 secs           |   10% in 0.0039 secs             |
|     25% in 0.0064 secs           |   25% in 0.0045 secs             |
|     50% in 0.0073 secs           |   50% in 0.0053 secs             |
|     75% in 0.0084 secs           |   75% in 0.0063 secs             |
|     90% in 0.0098 secs           |   90% in 0.0074 secs             |
|     95% in 0.0107 secs           |   95% in 0.0082 secs             |
|     99% in 0.0135 secs           |   99% in 0.0102 secs             |
</pre>

Fixes #678
2023-06-04 15:02:00 -04:00
Yuri Astrakhan
adab3bca95 Fix mdbook compile target 2023-06-04 14:11:44 -04:00
Yuri Astrakhan
300b08b5b0 cleanup installation docs 2023-06-04 00:12:24 -04:00
Yuri Astrakhan
b147c27380
Move Martin crate to its own subdir (#699)
This cleans up the code a bit, making all crates to be on the same
level.
2023-06-04 02:50:55 +00:00
Yuri Astrakhan
2cdd373044
Refactor mbtiles lib, mbtiles tool tests & CI (#702)
* Broke up martin-mbtiles into multiple files
* Made all mbtiles functions take a `SqliteExecutor` -- this way they
can be used with any SQLX connection structs - either a pool connection
or an individual non-pooled connection.
* Simplified mbtiles bin a bit - I realized there is really no need to
pretty print the output for the single value retrieval. Easier to just
dump it to console as is.
* Bump martin-mbtiles to v0.2.0
* Minor fixes in tools docs, cargo.toml, and justfile
* MBTiles tool Integration tests and release publishing

Major thanks to the
[stackoverflow](https://stackoverflow.com/questions/76394665/how-to-pass-sqlx-connection-a-mut-trait-as-a-fn-parameter-in-rust/76395111)
quick reply by @cafce25 on how to use generic sql executor!
2023-06-04 00:54:50 +02:00
rstanciu
78e67c3ad2
Add mbtiles meta-get utility (#701)
* Add functionality to retrieve a metadata value in an mbtiles file by
key; simple implementation for one of the items in #667
* Also, disable TTY in docker-up `just` target
2023-06-03 00:40:22 +00:00
Yuri Astrakhan
290a7c18fa readme cleanups 2023-06-02 17:00:19 -04:00
Yuri Astrakhan
81bb9582ed
Cleaning up mdbook (#697)
A lot of book content cleanup, making it a bit shorter and "logical"
(whatever that even means)

To try, run `just mdbook`
2023-06-02 01:16:41 -04:00
Yuri Astrakhan
5d87b086fe improving readme/docs 2023-06-01 16:15:05 -04:00
baishikele
27f54e2ee1
adjust book style (#691) 2023-05-30 19:13:40 -04:00
baishikele
56dde81cfe
Create mdbook documentation (#681)
Copy readme into multiple book files

Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
2023-05-29 02:06:35 +00:00