Summary:
Now that Rust macros can be `use`d like normal symbols, `stats` can
simply import the `lazy_static!` macro without requiring its users to do it.
Reviewed By: Imxset21
Differential Revision: D13281897
fbshipit-source-id: a6780fbace07dd784308e642d4a384322a17c367
Summary: It causes test failures. REvert for now until they fixed
Reviewed By: farnz
Differential Revision: D13040073
fbshipit-source-id: fc05373c882baf42f7bd2a3a1c1173e8ba26a952
Summary:
We're seeing long transaction hold times in MyRouter during big
blobimports - let's see if it's just that we've got a huge amount of events
outstanding on single tasks
Reviewed By: HarveyHunt
Differential Revision: D12945071
fbshipit-source-id: 3aca0b8cb649fc572fca8cadec8f0d265be2d564
Summary:
Purpose:
- Sha256 alias link to file_content is a required part of LFS getfiles works correct.
LFS protocol uses SHA-256 to refer to the file content. Mononoke uses Blake2.
To support LFS in Mononoke we need to set up a link from SHA-256 hash of the content to blake2 of the content.
These links are called aliases.
- Aliases are uploading together with file content blobs.
But only for new push operations.
- If repo is blobimported from somewhere, we need to make sure, that all the links are in blobstore.
If repo was blobimported before aliases were added then it may miss aliases for some blobs.
- This tool can be used to
- find if any aliases are missing
- fill missing aliases.
Implementation:
- Run in repo.
- Iterate through all changesets.
- Go through all the file_content blobs in the changesets
- Verify/generate alias256 links to file_content blobs.
Mode supported:
- verify, count the number of errors and print to console
- generate, if blob is missing to add it to the blobstore
Reviewed By: StanislavGlebik
Differential Revision: D10461827
fbshipit-source-id: c2673c139e2f2991081c4024db7b85953d2c5e35
Summary: This will enable doing queries like DELETE, UPDATE or REPLACE without listing all possibilites in the macros
Reviewed By: StanislavGlebik
Differential Revision: D10499501
fbshipit-source-id: 3e2ba433722bd34ffb5960840c509dc27cc9eb5d
Summary:
This diff adds a real implementation for CachingChangesetFetcher. Now it
fetches the data for the cache from the blobstore.
The rest is explained in the comments.
Reviewed By: farnz
Differential Revision: D9908320
fbshipit-source-id: 5427f3ed312cb7753434161423cb27b48744347f
Summary:
In the next diff I'm going to add a separate functions to look in the cache and
insert into the cache, so rename it to avoid confusion
Reviewed By: Imxset21
Differential Revision: D9869648
fbshipit-source-id: f2bd806b14d78660518d841d90a903970028eb37
Summary:
We have plans to add a cache of many changeset entries and store it in the
blobstore. The main reason is to speed up Mononoke's revsets and in turn speed
up getbundle wireproto request.
To cache we first need to serialize them. Let's use thrift serialization for
that.
Reviewed By: lukaspiatkowski
Differential Revision: D9738637
fbshipit-source-id: ba771545de9a955956acb6d169ee7bc424ef271b
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary:
`asynchronize` currently does not warn the event loop that it's
running blocking code, so we can end up starving the thread pool of threads.
We can't use `blocking` directly, because it won't spawn a synchronous task
onto a fresh Tokio task, so your "parallel" futures end up running in series.
Instead, use it inside `asynchronize` so that we can pick up extra threads in
the thread pool as and when we need them due to heavy load.
While in here, fix up `asynchronize` to only work on synchronous tasks and
push the boxing out one layer. Filenodes needs a specific change that's
worth extra eyes.
Reviewed By: jsgf
Differential Revision: D9631141
fbshipit-source-id: 06f79c4cb697288d3fadc96448a9173e38df425f
Summary:
It makes startup unbearably slow, and doesn't add any benefits at all. Revert
it
Reviewed By: purplefox
Differential Revision: D9358741
fbshipit-source-id: 26469941304f737c856a6ffca5e577848ad30955
Summary:
Should be functionally equivalent and semantically more appropriate
This also makes a couple of small API changes:
- The inner function is expected to just return a Result - IntoFuture is
overkill if its supposed to be synchronous in the first place
- `asynchronize` itself returns `impl Future` rather than being intrinsically
boxed.
- Restructure dieselfilenodes::add_filenodes to only asynchronize the insert
itself.
Reviewed By: farnz
Differential Revision: D8959535
fbshipit-source-id: fef9164e3be0069bd0d93573642cd57bb5babb73
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
These are the types that we currently need to be able to serialize if we're to
replace `Asyncmemo`'s caching uses with cachelib. Derive the `Abomonation`
trait for all of them.
Reviewed By: jsgf
Differential Revision: D9082597
fbshipit-source-id: 910e90476a3cc4d18ba758226b8572c3e8d264c6
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
Unify all uses of Sqlite and of Mysql
This superceded D8712926
Reviewed By: farnz
Differential Revision: D8732579
fbshipit-source-id: a02cd04055a915e5f97b540d6d98e2ff2d707875
Summary: Going to master adds a latency. Let's go to master only if we haven't found anything in the replica
Reviewed By: lukaspiatkowski
Differential Revision: D8548598
fbshipit-source-id: b0632ccf7f148d20e66763981c8ef4e615fd7f6c
Summary: this will make it easier to identify stats exported from mononoke vs external libraries
Reviewed By: StanislavGlebik
Differential Revision: D8331418
fbshipit-source-id: c151e76aa386fb13759fced7cc07b03ac67fe051
Summary: Going to take a while to get to everything, but here's a start.
Reviewed By: Imxset21
Differential Revision: D8311107
fbshipit-source-id: ada1908b320a5277eda2587d7e8f26b13b952154
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: It only needs to borrow them.
Reviewed By: kulshrax
Differential Revision: D8244267
fbshipit-source-id: 2a24a3b7c6eb65177e4e26c57650dd7e096b4202
Summary: As a bonus - put the whole rows in the error message to make it easier to spot what is the problem
Reviewed By: farnz
Differential Revision: D8186670
fbshipit-source-id: 719d302883b23733daa30878eaf27ea6df0180d7
Summary: It can be useful to understand how often do we hit a db, write to blobstore, etc
Reviewed By: farnz
Differential Revision: D7806954
fbshipit-source-id: 9c85cbb11e184ae9f7751e05e961980b29b7568a
Summary:
Let's fail only if inconsistent data was inserted - for example, same commit
hash but different parents.
This matches core hg behavior, and also it's completely normal for commit cloud
to send more parent commits than necessary.
Reviewed By: lukaspiatkowski
Differential Revision: D7722649
fbshipit-source-id: 172a0985fb3fda27d55e9dce8916ec3793de5db9
Summary:
This is a nice little improvement, that makes code readable.
`asynchronize()` does all the job of executing the job on the thread pool.
Reviewed By: lukaspiatkowski
Differential Revision: D7722648
fbshipit-source-id: 6d87178c11fa56ea9738738357fb3c3af012c81a
Summary:
Use asyncmemo to cache Changesets.
Unfortunately currently we are using separate asyncmemo cache, so we have to
specify the size for the caches separately. Later we'll have a single cache for
everything, and the number of config knobs will go down.
Reviewed By: lukaspiatkowski
Differential Revision: D7685376
fbshipit-source-id: efe8a3a95fcc72fab4f4af93564e706cd1540c2f
Summary:
Update x86 and aarch64 toolchains, rust-crates-io update.
The `use_nested_groups` feature is now stable, so we don't need to list it in
`#![feature(...)]` lines.
Reviewed By: kulshrax
Differential Revision: D7690730
fbshipit-source-id: 5f74bb01df6bd88a66ac00a82e446c18e3a1c43d
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary:
We don't need parent_id in the index, because we'll search only using cs_id and
seq
Reviewed By: jsgf
Differential Revision: D7487275
fbshipit-source-id: 93ceb238b7d5f42e5d37c1a85eadb8e7aab5c1f4
Summary: There can be a lot of changesets - let's not limit ourselves to 2B
Reviewed By: farnz
Differential Revision: D7429645
fbshipit-source-id: 9501214f6f10b87b8a37cb5b9503a9d231e241be
Summary:
They would hang on any failure rather than reporting properly.
Convert to using async_unit::tokio_unit_test() to make sure unwinds are caught.
Reviewed By: StanislavGlebik
Differential Revision: D7408174
fbshipit-source-id: 39b41bd66557e0d455f87ad29a2df73768cfb9c6
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary:
There's no point passing it by reference since callers don't need to
retain it, and the async implementation needs to move it into another context.
Reviewed By: farnz
Differential Revision: D7350001
fbshipit-source-id: 5947557a84621afae801dc20e3994496244e3a10
Summary:
Use connection pooling for mysql. We can't use it for sqlite because each
"connection" ends up being to a separate db.
Reviewed By: sid0
Differential Revision: D7350000
fbshipit-source-id: 37be71b863810c48ae69da579cf74358058f7d2f
Summary:
For testing, I want to be able to persist state in SQLite (so that
when we have in-process caching, I can restart and see cold cache behaviour).
This means that I want to open a database, and create the tables ignoring
failures. Make it possible
Reviewed By: StanislavGlebik
Differential Revision: D7353231
fbshipit-source-id: 0d0926c501e7951dfaf82998ed58d152e80ace7e
Summary: the diesel schemas should prevent from reading a manifest hash as changeset and vice versa
Reviewed By: StanislavGlebik
Differential Revision: D7339725
fbshipit-source-id: c619ab7ecabfd10f42e64a38c0f16abd114d4a31
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary: Add in the missing code to establish a MySQL connection to the `changesets` crate. Also add unit tests for `MysqlChangesets`.
Reviewed By: jsgf
Differential Revision: D7174371
fbshipit-source-id: b531c5d5efb95eeb5e39188c4d608540f0c0261f
Summary:
Diesel looks quite promising overall, as long as we can ensure async operations
work.
Some basic notes:
* This just looks async but isn't really -- doing that would need a way to send
queries over to a thread pool. (cc kulshrax)
* It is unfortunate that we must use a macro for the exact same `add` code across
MySQL and SQLite. There doesn't appear to be a way out though -- there is no
trait bound that we can apply to the `add` method here.
This is currently pretend-async -- it would be good to make this really async, but that should wait for a future diff IMO
Reviewed By: StanislavGlebik
Differential Revision: D6932028
fbshipit-source-id: ac1cfde3b96bb790f00b73d49c6d90ed128f0255
