Summary:
Mononoke and hg both have their own implementation wrappers for lz4
compression, unify these to avoid duplication.
Reviewed By: StanislavGlebik
Differential Revision: D14131430
fbshipit-source-id: 3301b755442f9bea00c650c22ea696912a4a24fd
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary:
Previously `max_gen()` function did a linear scan through all the keys, and it
was linear. Let's use `UniqueHeap` datastructure to track maximum generation
number.
Reviewed By: lukaspiatkowski
Differential Revision: D13275471
fbshipit-source-id: 21b026c54d4bc08b26a96102d2b77c58a981930f
Summary:
We've recently found that `known()` wireproto request gets much slower when we
send more traffic to Mononoke jobs. Other wireproto methods looked fine, cpu
and memory usage was fine as well.
Background: `known()` request takes a list of hg commit hashes and returns
which of them Mononoke knows about.
One thing that we've noticed is that `known()` handler sends db requests sequentially.
Experiments with sending `known()` requests with commit hashes that Mononoke
didn't know about confirmed that it's latency got higher the more parallel
requests we sent. We suspect this is because Mononoke has to send a requests to
db master, and we limit the number of master connections.
A thing that should help is batching the requests i.e. do not send many
requests asking if a single hg commit exists, but sending the same request for
many commits at once.
That change also required doing changes to the bonsai-mapping caching layer to
do batch cache requests.
Reviewed By: lukaspiatkowski
Differential Revision: D13194775
fbshipit-source-id: 47c035959c7ee12ab92e89e8e85b723cb72738ae
Summary:
Default ServiceType is ServiceType.Any, so it might go to master in a master
region. This diff changes it.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D13021674
fbshipit-source-id: 928cf59b095549f3048411241116c097e1193c7d
Summary: Additionally use a lower max_number_of_concurrent_connections for read connections to master to avoid overloading it.
Reviewed By: farnz
Differential Revision: D12979366
fbshipit-source-id: 258dbae554155d7a33d619f445293092940aad61
Summary:
We were using incorrect buffer size. That's *very* surprising that our servers
weren't continuously crashing. However, see the test plan - it really looks
like `LZ4_compressBound()` is the correct option here.
Reviewed By: farnz
Differential Revision: D9738590
fbshipit-source-id: d531f32e79ab900f40d46b7cb6dac01dff8e9cdc
Summary:
Backout D9124508.
This is actually more complex than it seems. It breaks non-buck build
everywhere:
- hgbuild on all platforms. POSIX platforms break because `hg archive` will
miss `scm/common`. Windows build breaks because of symlink.
- `make local` on GitHub repo because `failure_ext` is not public. The `pylz4`
Cargo.toml has missing dependencies.
Fixing them correctly seems non-trivial. Therefore let's backout the change to
unblock builds quickly.
The linter change is kept in case we'd like to try again in the future.
Reviewed By: simpkins
Differential Revision: D9225955
fbshipit-source-id: 4170a5f7664ac0f6aa78f3b32f61a09d65e19f63
Summary: Moved the lz4 compression code into a separate module in `scm/common/pylz4` and redirected code referencing the former two files to the new module
Reviewed By: quark-zju, mitrandir77
Differential Revision: D9124508
fbshipit-source-id: e4796cf36d16c3a8c60314c75f26ee942d2f9e65
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
Unify all uses of Sqlite and of Mysql
This superceded D8712926
Reviewed By: farnz
Differential Revision: D8732579
fbshipit-source-id: a02cd04055a915e5f97b540d6d98e2ff2d707875
Summary:
We had a memory leak because context wasn't cleaned afterwards. This diff fixes
it
Reviewed By: farnz
Differential Revision: D8236762
fbshipit-source-id: f82b061f3f541d9104d1185ed04ea21224b7d5bc
Summary: We are going to add CompressContext in the next diff
Reviewed By: farnz
Differential Revision: D8236761
fbshipit-source-id: 0df55b9bc5e9fd78ac8c060576513c1216641ead
Summary: Will be used in remotefilelog getfiles method.
Reviewed By: jsgf
Differential Revision: D6884919
fbshipit-source-id: e8037123a4843322c29b37c6b5749444781f4fa7
Summary:
Add a separate crate that uses lz4 in the same way as python lz4 library. The
main difference is that first 4 bytes are length of the raw data in le32
format. The reason for moving it in a separate crate is to use pylz4 for
remotefilelog getfiles method.
Also removed one panic and replaced it with error.
Reviewed By: jsgf
Differential Revision: D6884918
fbshipit-source-id: 1b05381c045a1f138ab28820175289233b07a91d
