Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary:
When I implement a cachemutex to keep the herd from all trying to
write the same blob multiple times, we'll be able to accelerate this. In the
short term, move the implementation up to here so that the change will be
obvious when it's put up for review.
Reviewed By: jsgf
Differential Revision: D8318152
fbshipit-source-id: f117c5fd834cab55ace10ac46bffa881457c840a
Summary:
Memcache is going to provide a faster implementation of `is_present`
that relies on the put cachemutex. We lose all the benefits if `is_present` up
here ends up always calling `get` from lower layers - teach it not to do that.
Reviewed By: jsgf
Differential Revision: D8318151
fbshipit-source-id: 130d4cb17c530c8677bdd563806989c393b4c6cf
Summary: Lack of proper error context makes it hard to understand errors returned from Mononoke
Reviewed By: StanislavGlebik
Differential Revision: D8298154
fbshipit-source-id: 57fe5df7d891b5215fba783255178f14122199cf
Summary: the newly added context should add much more visibility into why an error was returned
Reviewed By: jsgf
Differential Revision: D8286343
fbshipit-source-id: d65387f40da2c14964e85552ae1ec7e75a135c47
Summary:
This change screams for using something like "await", but because we don't have it yet we need to reorganize this code into a chain of `and_then`.
Thanks to that if we attach a context to a future inside `and_then` the context will be properly reported when an error happens inside that future.
This is a bit of a simplification, but the previous code used the equivalent of `map` rather than `and_then` and this resulted in Errors returned from within f.e. `upload_changesets` to have a context of "While resolving B2xInfinitepushBookmarks" attached which was confusing.
Reviewed By: StanislavGlebik
Differential Revision: D8284606
fbshipit-source-id: b8a4bd63b9732caea53089f1b9088f2624a42f87
Summary: this will make it easier to identify stats exported from mononoke vs external libraries
Reviewed By: StanislavGlebik
Differential Revision: D8331418
fbshipit-source-id: c151e76aa386fb13759fced7cc07b03ac67fe051
Summary: This test implements the passing of all current changeset fields to hooks
Reviewed By: StanislavGlebik
Differential Revision: D8298019
fbshipit-source-id: 0e6be3c83b1e4d4c3eab95c76c9041ea2a57f0d3
Summary: Going to take a while to get to everything, but here's a start.
Reviewed By: Imxset21
Differential Revision: D8311107
fbshipit-source-id: ada1908b320a5277eda2587d7e8f26b13b952154
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: Factor out common stuff for local state.
Reviewed By: StanislavGlebik
Differential Revision: D8310823
fbshipit-source-id: e1ce7eebd76d37688e830a5df0486bdfd1e3361c
Summary:
`hg pushbackup` got moved to the `infinitepushbackup` extension.
Also fix some minor output changes.
Reviewed By: StanislavGlebik
Differential Revision: D8310862
fbshipit-source-id: ae8d5579dd4082bd91e80beef4a4b675af9a8986
Summary:
* `.hg` is where mononoke sockets used to reside, but they no longer do.
* `heads` is no longer a thing
* `BlobRepo::new_rocksdb` already takes care of creating `SqliteChangesets` if it doesn't exist.
Reviewed By: StanislavGlebik
Differential Revision: D8289145
fbshipit-source-id: c8b0ad3b9bee5c22f79861474fd08256dbd0fb8f
Summary: Unfortunately even 10 seems to be too little for stress runs.
Reviewed By: StanislavGlebik
Differential Revision: D8310639
fbshipit-source-id: 280824a7c2ec2283d2ef3acfb968e336ca13c93b
Summary: This diff adds unit tests for the runhook command line utility
Reviewed By: jsgf
Differential Revision: D8257571
fbshipit-source-id: 5c390d2a45d895080fce28dcd7943da5d803ff92
Summary:
This diff adds more robust testing for various errors in Lua hooks.
It also contains a little bit of cleanup in the runHook command
Reviewed By: StanislavGlebik
Differential Revision: D8253525
fbshipit-source-id: de9d298e70ec647f2c13e27c9937605ac5b57485
Summary: Changed the about section of new_blobimport to something descriptive.
Reviewed By: kulshrax
Differential Revision: D8307334
fbshipit-source-id: 2198d43cdfaf566b57001be5230a74206306dac1
Summary:
Replaced the help and about messages to summarize what the admin tool
is useful for
Reviewed By: jsgf
Differential Revision: D8302356
fbshipit-source-id: 77d1d4bb50825b0cc2d1ac2ee8d47ee906e04a22
Summary: The data logged there is too large, so print only a subset of it (changesets hashes) and only as context for error
Reviewed By: farnz
Differential Revision: D8235881
fbshipit-source-id: c81eeca2baf26def4d24b288f9d29dd8c23823d8
Summary: This log is by far the most common one and it makes reading logs much harder. It should probably be changed to ODS counters, but for now lets just make it trace!
Reviewed By: farnz
Differential Revision: D8235663
fbshipit-source-id: 3685b260f1c6c43c1fde8501731583debc8d063b
Summary: See the comment for a full explanation.
Reviewed By: jsgf
Differential Revision: D8274912
fbshipit-source-id: 936baa60925fff5ddf1e903c8408410c0a429e77
Summary:
Fuse panics if there is an error. However the errors are quite possible - for
example, if client went away then sending stderr log panics and the whole
Mononoke server fails. This diff fixes it by using ignore res
Reviewed By: lukaspiatkowski
Differential Revision: D8276600
fbshipit-source-id: e0c91a082d80d7c54372171f9893f98b2a1e8304
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary:
The transaction is not necessary in there, since if any of the query fails then the commit is not written in changesets table.
We need to remove it, because with many writes to the filenodes table some of the transactions are timing out while waiting for the database's lock for transation
Reviewed By: farnz
Differential Revision: D8220722
fbshipit-source-id: 003b1369abb1a47e8ad37517745ab659116b95f6
Summary: See previous diff for the movitvation
Reviewed By: jsgf
Differential Revision: D8207273
fbshipit-source-id: 527c1d97546afedf36f84059ccb7c740bd412907
Summary:
Pruner is an FnMut that is called on every entry
that `changed_entry_stream` visits. If Pruner returns false, then
`changed_entry_stream` won't recurse into that.
The primary motivation for that is gettreepack request. We may get requests
with a few `mfnodes`. Currently we just merge a few `get_changed_entry`
streams. The problem is that some entries will be in both streams, and thus
will be generated twice. visited_pruner changes it.
Reviewed By: jsgf
Differential Revision: D8207271
fbshipit-source-id: 1896cb68edbca57651c4767d9326b9bbbab2c980
Summary: It only needs to borrow them.
Reviewed By: kulshrax
Differential Revision: D8244267
fbshipit-source-id: 2a24a3b7c6eb65177e4e26c57650dd7e096b4202
Summary:
We had a memory leak because context wasn't cleaned afterwards. This diff fixes
it
Reviewed By: farnz
Differential Revision: D8236762
fbshipit-source-id: f82b061f3f541d9104d1185ed04ea21224b7d5bc
Summary: We are going to add CompressContext in the next diff
Reviewed By: farnz
Differential Revision: D8236761
fbshipit-source-id: 0df55b9bc5e9fd78ac8c060576513c1216641ead
Summary:
Now hook manager returns result of running hooks as BoxFuture<HashMap<String, HookExecution> where HookExecution is a new richer type representing the result of running a hook.
This provides more info to the user as to why the hook rejected the changeset and a map is simpler to lookup a particular hook failure than a Vec.
Reviewed By: StanislavGlebik
Differential Revision: D8235970
fbshipit-source-id: 9a617b6d459f105aa9dad9782e784459dd716c45
Summary: Hook now returns a future to allow for truly async Rust futures. Also implement Lua hooks as functions and some other small changes.
Reviewed By: StanislavGlebik
Differential Revision: D8235778
fbshipit-source-id: 92753bde4c9b65d15286102827e2f47673feb0c1
Summary:
Big change here is update to rand 0.5. This is a significant API
change. quickcheck still uses rand 0.4, so for quickcheck users I changed it so
that quickcheck re-exports the rand it uses. This means that quickcheck users
are unchanged aside from using quickcheck::rand, whereas direct rand users have
been updated to use the new API.
Reviewed By: farnz
Differential Revision: D8234503
fbshipit-source-id: f9e620851b8dfcc33f22a0af26122adcd5fbde39
Summary:
I got confused myself, so decided to rename it.
dirname here means "all the path without the last path component". So for directory "a/b/c", "a/b" is a dirname. For file "a/b.txt", "a" is a dirname For root dirname is empty.
Reviewed By: jsgf
Differential Revision: D8207274
fbshipit-source-id: 0744ee020d3cf1ecc185efee445f6d40183d6366
Summary:
Introduce the `ReadyState` abstraction to capture repositories being
ready. Also make sure that it "fails open" -- drops and panics cause that
particular handle/future to be marked ready.
Reviewed By: lukaspiatkowski
Differential Revision: D8219667
fbshipit-source-id: 78078055c46ceaa922a4579fe42cbf89fb42a81f
Summary:
There's a bit of Rust/C FFI, but hopefully the comments make it clear
how ownership works.
As part of the lifecycle I also had to handle shutdown safely. This means that `run_service_framework` actually returns now.
Reviewed By: jsgf
Differential Revision: D8133252
fbshipit-source-id: 1e394a60c6e62f5c3d56933f3e7be7a2bfd1e2c0
Summary: Going to add a lot more code to this very soon.
Reviewed By: farnz
Differential Revision: D8152464
fbshipit-source-id: ed6b2752b470a76b5808cf7ee18fbe8939d8ffc2
Summary:
In the future diffs we'll add a trait called Pruner, which can be used to avoid
recursing into some paths. For example, to not recurse into deleted paths, not
recurse into the paths that ends on this string etc.
In this diff we just make a small refactoring to make adding Pruner easier.
The main point of the refactoring is to not call recursive_entry_stream
function.
Reviewed By: jsgf
Differential Revision: D8207269
fbshipit-source-id: 42f246a8a7aaf7a43c40fa05ebea876830391430
Summary:
Just a small helper function to check if the entry is tree or not.
That should work fine for Modified entry, because both of the entries should
have the same type
Reviewed By: jsgf
Differential Revision: D8207272
fbshipit-source-id: 29d2320a06f6cb51bd36899e0da9f928af4bcb06
Summary:
This diff introduces HookManager which knows how to install, uninstall and run hooks.
Hooks are now run in parallel on a cpu_pool.
Reviewed By: lukaspiatkowski
Differential Revision: D8208538
fbshipit-source-id: f18687c14a15cadf4d832318cd66fa400586c29f
Summary:
This is a follow-on from D8014913 (Make in-memory manifests load lazily), addressing jsgf's review comments.
I've made the following stylistic changes:
1. Use `impl Future` for all private APIs instead of `BoxFuture` (making use of `Either` instead of boxing where possible).
2. Use Rust 1.26 style `match` statements with inferred `ref` and `&` whenever possible.
3. Rework the logic in `MemoryManifestEntry::save` to not need `filter_map`.
4. Remove `future::result` and exploit `impl IntoFuture for Result` where possible to reduce code
5. Improve a couple of comments
Reviewed By: jsgf
Differential Revision: D8209186
fbshipit-source-id: c759ad8894fc25616dc6a291d46c487191f96382
Summary: Since signal_parent_ready purpose is to ensure that the child Changesets might perform their own checks it is good enough to send signal_parent_ready after all blobs have been written out, but before the parent did it's own checks
Reviewed By: farnz
Differential Revision: D8202108
fbshipit-source-id: 15ac85bd18bcf9ded61363a6380ad05462c189d6
Summary: If we're going to use memcache to cache things, we need to be able to confirm that the blobstore contains what we expect. Add an admin option that lets us check what's in memcache and/or in the blobstore when using memcache
Reviewed By: StanislavGlebik
Differential Revision: D8124624
fbshipit-source-id: 763786930cf9f4be6d2d8e346ae1ef94dc8e492c
Summary:
Memcache provides us with a low-latency out-of-process cache, shared
between multiple machines. If we use this to provide a write-through cache,
then we get a couple of benefits:
1. Mononoke servers accessing the same memcache backend get low latency access
to commonly requested blobs.
2. Recently written blobs are available at low latency to the server that just
wrote them, hopefully speeding up new_blobimport.
Reviewed By: StanislavGlebik
Differential Revision: D8124623
fbshipit-source-id: 5af085aa8bb63c1366740edfda9020d72b8a9015
Summary: As a bonus - put the whole rows in the error message to make it easier to spot what is the problem
Reviewed By: farnz
Differential Revision: D8186670
fbshipit-source-id: 719d302883b23733daa30878eaf27ea6df0180d7
Summary:
instead of doing for_each which causes the entries to be processed one-by-one use the map + buffer_unordered technique
This has speeden up new_blobimoprt of 2 commits from >2m to <1m30s
Reviewed By: farnz
Differential Revision: D8184176
fbshipit-source-id: 4a0c3124d2398ed41f8b93785bc3c890a23a88aa
Summary:
In previous diffs we've added a function that does a bulk prefetch of
filenodes. Now we can use it in the getfiles request handling code.
Before constructing file history we do a bulk prefetch. Then we use
results of the bulk prefetch as a cache. It means that we first check
whether we have file history node in the prefetched data, and if not it will be
fetched again. It allows us to later plugin a memcache or some other kind of
cache.
repo.rs file is getting bigger and bigger. I'm planning to split it, however
only after I land this diff in order not to deal with merges
Reviewed By: farnz
Differential Revision: D8185701
fbshipit-source-id: 6ca37aeb029236db51d2b5a03cb7053f969cf47e
