This adds a new utility called `martin-cp` that allows copying any
number of tiles from a single source (or a composite source) to an
mbtiles file.
`martin-cp` is a tool for generating tiles in bulk, and save retrieved
tiles into a new or an existing MBTiles file. It can be used to generate
tiles for a large area or multiple areas. If multiple areas overlap, it
will generate tiles only once. `martin-cp` supports the same
configuration file and CLI arguments as Martin server, so it can support
all sources and even combining sources.
# MBTiles
* New `--agg-hash (update|verify|off)` flag replaces
`--update-agg-tiles-hash` (still supported, but not shown in the help
screen). This allows bypassing aggregate hash validation entirely,
without either updating or validating it.
* Simplify MBTiles SQL generation
* MBTiles now uses faster `1 << zoom` everywhere, and a dedicated TMS
inversion fn
* split up metadata insert and delete into separate fn
* consolidated schema initialization
* ensure db settings (like pragma) are always reset on new files
# Other
* Always sort JSON-serialized keys for consistency
* this affects `/catalog` key ordering, but content is the same
* Minor code cleanup
- Add separators to long literal.
- Use first() instead of get(0)
- Move some SQL-making code to their own functions
- Minor mathematics and rounding cleanup
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
This implements dynamic font protobuf generation, allowing users to
request font ranges on the fly, and combining them in any order, e.g.
`Font1,Font2,Font3`, same as with sprites and tiles
This is a first iteration, without any multithreading support. In
theory, this could be done far faster by generating SDFs with multiple
threads.
### Current process
* during init, figure out all glyphs available in each font, and store
them as a bitset
* during request:
* combine requested bitsets to figure out which glyph should come from
which font file
* load those glyphs from files (using a single instance of the freetype
lib)
* convert them to SDFs and package them into a protobuf
---------
Co-authored-by: Lucas <zhangyijunmetro@hotmail.com>
* Fix metadata copying
* Introduce a new metadata field `agg_tiles_hash_after_apply` for diff
files
* Added a lot of new info and debug logging
* Simplified Copying interface - not much value in having all the
complex builder pattern here it seems, might as well use a simple
object.
## Testing
* Generate SQLite DBs in memory on the fly to validate just what we need
* Use `insta` for validating DB content
There is now a function `dump(connection) -> Vec<Entry>` to dump the
content of the entire SQLite DB into text with `serde`. At many steps
through the testing, the DB content is validated with the corresponding
.snap file with `insta` crate (which makes this process mega-simple,
including a simple way to "bless" (update) any changes).
## Discovered bugs
* Seems like normalized files do not get copied properly - they contain
extras that should be removed.
* Rename `TileCopierOptions` -> `TileCopier`
* remove a few un-needed sqlite open to detect mbtiles type
* move `open_and_detect_type` to `MBTiles`
* add `attach_to` to `MBTiles`
* move various table creation fn to mbtiles_queries file
* a few sql format
Bumps [json-patch](https://github.com/idubrov/json-patch) from 1.0.0 to
1.1.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="50a220ab4d"><code>50a220a</code></a>
chore: Release json-patch version 1.1.0</li>
<li><a
href="d3a5a121bb"><code>d3a5a12</code></a>
fix: tests and suggestion</li>
<li><a
href="227dd997c3"><code>227dd99</code></a>
refactor: do not use recursion but keep a stack</li>
<li><a
href="670981783d"><code>6709817</code></a>
Updating codecov and fixing the test.</li>
<li><a
href="0e1f7aab17"><code>0e1f7aa</code></a>
Use utoipa support for serde_json::Value</li>
<li>See full diff in <a
href="https://github.com/idubrov/json-patch/compare/v1.0.0...v1.1.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
If a PostgreSQL function has an SQL comment, it will try to parse as
JSON and use its values to override the auto-generated TileJSON. It is
recommended to use this form when creating comments to ensure valid JSON
values.
```sql
DO $do$ BEGIN
EXECUTE 'COMMENT ON FUNCTION YOUR_FUNCTION (ARG1_TYPE,ARG2_TYPE,..ARGN_TYPE) IS $tj$' || $$
{
"description": "description override",
...
}
$$::json || '$tj$';
END $do$;
```
Partially implements #822
* Add `MbtType::FlatWithHash`
* Support copying, diffing and applying diffs to and from any
`MbtTypes`s
* Support validating tile data if hash is contained in `*.mbtiles` file
(i.e it is of `MbtType::FlatWithHash` or `MbtType::Normalized`)
---------
Co-authored-by: rstanciu <rstanciu@rivian.com>
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
Add deb package by [cargo-deb](https://crates.io/crates/cargo-deb)
- [ ] add deb package metadata
- [ ] more research on systemd integration
To inspect the package:
```shell
# install cargo-deb dependices
sudo apt install dpkg
sudo apt install dpkg-dev
sudo apt install liblzma-dev
cargo install cargo-deb
cd martin
cargo deb
The Debian package will be created in `target/debian/martin_0.8.7_amd64.deb` .
```
To decompress this package:
```shell
dpkg -x martin_0.8.7_amd64.deb path_of_decompress
```
---------
Co-authored-by: Yuri Astrakhan <yuriastrakhan@gmail.com>
This is mostly a noop, just updating to the new way of storing cached
queries and a few other changes per
https://github.com/launchbadge/sqlx/blob/main/CHANGELOG.md#070---2023-06-30
Note that this uses some of the code from #729 (thanks @upsicleclown !)
A minor unrelated change - a reformat of `martin/release.toml`
I'd like to add optional rustls support for SQLx. This PR implements the
required features, but tests using `--all-features` fail, because
runtime selection features are not additive. I didn't find a solution
apart from avoiding `--all-features`.
---------
Co-authored-by: Yuri Astrakhan <YuriAstrakhan@gmail.com>
Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.29.0 to 1.29.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/tokio-rs/tokio/releases">tokio's
releases</a>.</em></p>
<blockquote>
<h2>Tokio v1.29.1</h2>
<h3>Fixed</h3>
<ul>
<li>rt: fix nesting two <code>block_in_place</code> with a
<code>block_on</code> between (<a
href="https://redirect.github.com/tokio-rs/tokio/issues/5837">#5837</a>)</li>
</ul>
<p><a
href="https://redirect.github.com/tokio-rs/tokio/issues/5837">#5837</a>:
<a
href="https://redirect.github.com/tokio-rs/tokio/pull/5837">tokio-rs/tokio#5837</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="1b1b9dc7e3"><code>1b1b9dc</code></a>
chore: prepare Tokio v1.29.1 release</li>
<li><a
href="012c848401"><code>012c848</code></a>
rt: fix nesting block_in_place with block_on (<a
href="https://redirect.github.com/tokio-rs/tokio/issues/5837">#5837</a>)</li>
<li>See full diff in <a
href="https://github.com/tokio-rs/tokio/compare/tokio-1.29.0...tokio-1.29.1">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Dynamically create image sprites for MapLibre rendering, given a
directory with images.
### TODO
* [x] Work with @flother to merge these PRs
* [x] https://github.com/flother/spreet/pull/59 (must have)
* [x] https://github.com/flother/spreet/pull/57
* [x] https://github.com/flother/spreet/pull/56
* [ ] https://github.com/flother/spreet/pull/62 (not required but nice
to have, can upgrade later without any code changes)
* [x] Add docs to the book
* [x] Add CLI param, e.g. `--sprite <dir_path>`
* [x] Don't output `.sprites` in auto-genned config when not in use
### API
Per [MapLibre sprites
API](https://maplibre.org/maplibre-style-spec/sprite/), we need to
support the following:
* `/sprite/<sprite_id>.json` metadata about the sprite file - all coming
from a single directory
* `/sprite/<sprite_id>.png` all images combined into a single PNG
* `/sprite/<sprite_id>@2x.json` same but for high DPI devices
* `/sprite/<sprite_id>@2x.png`
Multiple sprite_id values can be combined into one sprite with the same
pattern as for tile joining:
`/sprite/<sprite_id1>,<sprite_id2>,...,<sprite_idN>[.json|.png|@2x.json|@2x.png]`.
No ID renaming is done, so identical names will override one another.
### Configuration
[Config file](https://maplibre.org/martin/config-file.html) and possibly
CLI should have a simple option to serve sprites. The configuration may
look similar to how mbtiles and pmtiles are configured:
```yaml
# Publish sprite images
sprites:
paths:
# scan this whole dir, matching all image files, and publishing it as "my_images" sprite source
- /path/to/my_images
sources:
# named source matching source name to a directory
my_sprites: /path/to/some_dir
```
Implement #705
* Broke up martin-mbtiles into multiple files
* Made all mbtiles functions take a `SqliteExecutor` -- this way they
can be used with any SQLX connection structs - either a pool connection
or an individual non-pooled connection.
* Simplified mbtiles bin a bit - I realized there is really no need to
pretty print the output for the single value retrieval. Easier to just
dump it to console as is.
* Bump martin-mbtiles to v0.2.0
* Minor fixes in tools docs, cargo.toml, and justfile
* MBTiles tool Integration tests and release publishing
Major thanks to the
[stackoverflow](https://stackoverflow.com/questions/76394665/how-to-pass-sqlx-connection-a-mut-trait-as-a-fn-parameter-in-rust/76395111)
quick reply by @cafce25 on how to use generic sql executor!
* Add functionality to retrieve a metadata value in an mbtiles file by
key; simple implementation for one of the items in #667
* Also, disable TTY in docker-up `just` target
Bumps [criterion](https://github.com/bheisler/criterion.rs) from 0.4.0
to 0.5.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/bheisler/criterion.rs/blob/master/CHANGELOG.md">criterion's
changelog</a>.</em></p>
<blockquote>
<h2>[0.5.0] - 2023-05-23</h2>
<h3>Changed</h3>
<ul>
<li>Replaced lazy_static dependency with once_cell</li>
<li>Improved documentation of the <code>html_reports</code> feature</li>
<li>Replaced atty dependency with is-terminal</li>
<li>MSRV bumped to 1.64</li>
<li>Upgraded clap dependency to v4</li>
<li>Upgraded tempfile dependency to v3.5.0</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>Quick mode (<code>--quick</code>) no longer outputs 1ms for measured
times over 5 seconds</li>
<li>Documentation updates</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="4a560cb8d8"><code>4a560cb</code></a>
release version 0.5 (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/683">#683</a>)</li>
<li><a
href="3e2849ebf8"><code>3e2849e</code></a>
Post v0.4 CHANGELOG updates (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/680">#680</a>)</li>
<li><a
href="637010e50d"><code>637010e</code></a>
Bump clap to version 4 (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/679">#679</a>)</li>
<li><a
href="39d7e2f954"><code>39d7e2f</code></a>
Update to tempfile 3.5 (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/675">#675</a>)</li>
<li><a
href="8ab2752ccf"><code>8ab2752</code></a>
Fix typo in known_limitations.md (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/674">#674</a>)</li>
<li><a
href="a2112308ef"><code>a211230</code></a>
chore: replace atty with is-terminal (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/628">#628</a>)</li>
<li><a
href="a844eb226b"><code>a844eb2</code></a>
Bump MSRV to 1.60 for csv dependency (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/665">#665</a>)</li>
<li><a
href="2f53607378"><code>2f53607</code></a>
Use OR operator to specify multiple licenses (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/643">#643</a>)</li>
<li><a
href="86dcd7d323"><code>86dcd7d</code></a>
Fix bug where time over the maximum would be reported as 1ms in quick
mode. (...</li>
<li><a
href="2942aee124"><code>2942aee</code></a>
Remove unnecessary <code>pub</code> from example code (<a
href="https://redirect.github.com/bheisler/criterion.rs/issues/657">#657</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/bheisler/criterion.rs/compare/0.4.0...0.5.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
