Summary:
cachelib can shrink caches to avoid running into OOM, while still
giving as close to full-sized cache performance as possible. Make it possible
for Rust users of cachelib to request cache shrinking on low memory, while
choosing a strategy that suits them.
Reviewed By: Imxset21
Differential Revision: D8895814
fbshipit-source-id: ca4eb5b002c9ed922e7f7f56de002313a0d2303b
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary: Let it be a normal future-style design - infinite loop that spawns futures that handled the request
Reviewed By: jsgf
Differential Revision: D8866166
fbshipit-source-id: 5d90e5c987c419351a7a15013133b47522c345f9
Summary: As a bonus there is no need for spawning threads for separate tokio cores since tokio::runtime has built-in thread pool
Reviewed By: farnz
Differential Revision: D8863343
fbshipit-source-id: 8adf696640aec78e767574e8bf2925699a580ca0
Summary: Iterating over the code on server is a bit painful and it has grown a lot, splitting it should speed up future refactories and make it more maintainable
Reviewed By: jsgf, StanislavGlebik
Differential Revision: D8859811
fbshipit-source-id: 7c56f9f835f45eca322955cb3b9eadd87fbb30a1
Summary: Moving to the crate allows apiserver to reuse the function.
Reviewed By: jsgf
Differential Revision: D8843178
fbshipit-source-id: 9d110c7f2683ff58654187222e7820240bfda98e
Summary:
Deleted the RepoGenCache structure, associated file, and public exports.
Also deleted the containing repoinfo crate, as nothing else was using it now.
Deleted some existing references to it in experimental code which weren't caught in the test plan but were blocking this from landing.
Reviewed By: StanislavGlebik
Differential Revision: D8787103
fbshipit-source-id: 0b90c758ea8175cb0f3ec74c371592b9ca5b192e
Summary:
Removed all references to RepoGenCache from publically callable functions in the revset package. This involved:
- Modifying blobrepo so that its get_generation_number method returned a Generation wrapper instead of a raw usize, to allow it to be used in a cleaner manner in the revset code.
- Simultaneously changing the constructors of all the structures in revset. This seems like a big change, but many of them call each other, passing a RepoGenCache object down the line, so eliminating them all at once made for the cleanest update.
- Modifying helper functions in the revset structures which would create streams of nodes by taking ownership of a RepoGenCache object within a closure. Instead they now take ownership of a clone of the repo. This strategy was already done earlier in the same helper functions, so I am assuming the cost of cloning a repo into a closure is small.
- Modifying the only external usage of revset within the mononoke server code.
This is part of a several step process to completely remove RepoGenCache from the code base. The next steps should be:
- Remove all references to RepoGenCache in the testing macros for revset.
- Delete RepoGenCache and clean up any dangling references to it.
Reviewed By: StanislavGlebik
Differential Revision: D8743560
fbshipit-source-id: 125f851075d836d40224d339e1daee912a39f7e4
Summary:
Those are the changes made to crates in tp2: P59806629
Several changes that were hard to split are here:
- update manifold client to use hyper 0.12.x
- update scribe client to use hyper 0.12.x
- update mononoke manifoldblob to use the updated manifold client
- update mononoke hgcli to use openssl 0.10.x and tokio-openssl
- update mononoke server to use openssl 0.10.x and tokio-openssl
- remove sendwrapper
Reviewed By: jsgf
Differential Revision: D8806931
fbshipit-source-id: 65412d483f77d8c0a0d5692c41c6516bb8f86046
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
hgcli will start logging stuff as well and it will use the same session_uuid as the server.
This also includes logging the user and source hostname.
Reviewed By: farnz
Differential Revision: D8750663
fbshipit-source-id: 7ebc8b6c10b7560d985fd23e9e3f2645f3bd0a1c
Summary: those structures are sshrelay specific, move them there so it's easier to share them in future diffs
Reviewed By: farnz
Differential Revision: D8750666
fbshipit-source-id: b58596e63787d221a3970d5f1648e11d81949925
Summary: This is a preparation to using Preamble more heavily
Reviewed By: StanislavGlebik
Differential Revision: D8750665
fbshipit-source-id: 44d4bcedbe95fe05679faeedf4479ebad4d9359c
Summary: Session UUID will help identify the issues on Mononoke side whenever the client encounters problems
Reviewed By: StanislavGlebik
Differential Revision: D8732396
fbshipit-source-id: 35d04b0d56be0cfc2c608f08287a2b1d236a96e3
Summary:
This diff refactors the server config repository to support storing and loading of hooks. In the new structure each repo lives in its own directory and the config file for the server is called "server.toml".
Hooks can be referenced by relative or absolute paths allowing either local or common hooks to be loaded.
Reviewed By: StanislavGlebik
Differential Revision: D8625178
fbshipit-source-id: 62c8c515a0fbbf7a38cfc68317300d8f42eb4d7a
Summary:
Add a per-repo config flag to repos to be configed without being
enabled. Setting "enabled = false" will make Mononoke completely ignore the
repo config. If not present, "enabled" is assumed to be true.
Reviewed By: farnz
Differential Revision: D8647161
fbshipit-source-id: 2646d41a64917d3e50f662b0b4b628ccfdbb05a8
Summary:
Use tls for connection between hgcli and Mononoke server always, even for
localhost connections[1]
The setup is similar to tls setup of Eden server.
[1] This is not necessary of course, but adding an option to bypass tls
connection may result in accidental use of it in prod. However if it turns out
to be too unusable, we can add such option in the future
Reviewed By: jsgf
Differential Revision: D8644299
fbshipit-source-id: 0898e30e33b718e13a766763479f3adf9323ffe7
Summary: This commit upgraded openssl, enabled alpn for actix-web and added tokio-codec with fixes due to the upgrade.
Reviewed By: StanislavGlebik
Differential Revision: D8682673
fbshipit-source-id: 8c7cadfd6c0c7b016202f6cb038eb4951d0f9333
Summary:
repo.rs file is getting too big. Let's move remotefilelog logic to a separate
file. In the next diff I'm going to modify it to add memcache caching for
getfiles
Reviewed By: farnz
Differential Revision: D8678229
fbshipit-source-id: c12ed23b4b044528d551f9c0f0266114a575d6d6
Summary:
There are so many individual arguments here that it's honestly hard to keep
track.
It is unfortunate that a bunch of string copies have to be done, but not really
a big deal.
Reviewed By: jsgf
Differential Revision: D8675237
fbshipit-source-id: 6a333d01579532a0a88c3e26b2db86b46cf45955
Summary:
It's easier to use scuba when you separate the part of log that identifies the log from the part that can have arbitrary data in it.
It's also easier to search for code after a sample was found
Reviewed By: jsgf
Differential Revision: D8625612
fbshipit-source-id: 7d7e382530dd5d7e5d69c6d34caccda4b6d2921b
Summary: Make it possible to enable and disable tracing in Mononoke via fb303 thrift calls.
Reviewed By: StanislavGlebik
Differential Revision: D8553236
fbshipit-source-id: 6c962fcf7f753f200bf865c403da12b0f9619221
Summary: Migrate Mononoke server to use new `tracing` crate instead of `futures-trace`. This has the effect of making tracing scoped to individual sessions, and also means the traces will be uploaded to Manifold instead of Everstore.
Reviewed By: farnz
Differential Revision: D8354328
fbshipit-source-id: 682f4b1cbbe9c2153ce54aaedc7876d52406527f
Summary: It's no longer test, we use it in prod
Reviewed By: farnz
Differential Revision: D8611639
fbshipit-source-id: dc52e0bcdc26c704c0d9cf820d7aa5deae3e06e4
Summary: Want to make this available at least.
Reviewed By: farnz
Differential Revision: D8591757
fbshipit-source-id: a96a92947e4a2c624caaf87ae007025d938bd1f2
Summary: With this changes we will always get a sample representing result of handling a request written in scuba, regardless if it was success, failure or timeout.
Reviewed By: StanislavGlebik
Differential Revision: D8579277
fbshipit-source-id: 6c7943bb26c8880e4f3df988dcebd12fd57ed062
Summary: more logging more fun
Reviewed By: StanislavGlebik
Differential Revision: D8577655
fbshipit-source-id: 92a160ea8f8c0b8e012a1461fbd3f5d71b4bd171
Summary: This should unify the fields that we send to our Scuba dataset and ensure all of them at least have session_uuid
Reviewed By: StanislavGlebik
Differential Revision: D8577657
fbshipit-source-id: fa1dcf5b6b8c14f73d7610ec75b6c936b3965d85
Summary: This is the scuba sample that is uploaded per whole connection
Reviewed By: farnz
Differential Revision: D8557588
fbshipit-source-id: 0914b902d29bee0172df7404b420b1a2853b28e4
Summary: If someone realy doesn't want a singleton then one would need to use FbWhoAmI::parse_from directly, which is fine.
Reviewed By: farnz
Differential Revision: D8556535
fbshipit-source-id: 3f49970b7b9692277932407ca1ec888a0d9458c0
Summary:
We have too many files, and it slows down gettreepack a lot. Let's filter them
ealier
Note: I haven't found an easy way to make pruner either
`add_pruner_combinator(...)` or `file_pruner`. Let me know if there is a way to
do it
Reviewed By: lukaspiatkowski
Differential Revision: D8526210
fbshipit-source-id: 441c4484b5b46a9e87d894f0c1283aaa986a7f45
Summary:
Background info: pruner is a function that is called on every node during a
tree walk. If it returns false that this node and all of it's subdirectories
are excluded from the walk. visited_pruner is used during gettreepack request
processing when clients requests data for a few manifest nodes at once.
Without visited_pruner some nodes maybe generated twice, and that can make
gettreepack request significantly slower.
While processing gettreepack requests we were inserting all the directories and
files in the visited_pruner's hashmap. However, gettreepack request doesn't
care about files at all, and we were wasting memory for no reason.
This diff fixes it by adding a file_pruner and and_pruner_combinator. File
pruner removes all the files, and only then we apply visited pruner.
Reviewed By: jsgf
Differential Revision: D8517101
fbshipit-source-id: 62f5f6de75a0aac06ec6b1d2cc54c6a99813965b
Summary:
Unfortunately I have to remove tracing. The reason is because tracing doesn't
work with Streams. For now it should be fine because enabling tracing in prod
is still not possible because of the memory and cpu overhead.
Reviewed By: farnz
Differential Revision: D8381855
fbshipit-source-id: e28b4396c81527bdf30fa1703c634688cf645ada
Summary:
Use SinkAsyncWrite that was added in the previous diff.
The main motivation is to make sure that we do not buffer bundle2 response in
memory. For more details look at the summary from the previous diff.
Note that this diff still buffers results in memory, but it prepares the code
for the next diffs, in which we'll finally remove the buffering.
Reviewed By: farnz
Differential Revision: D8379586
fbshipit-source-id: 5985ddb157f7066732ef47a04919eb1fba68c74e
Summary:
visited_pruner is useless if we have just one mfnode, becase it won't filter
out anything, because we'll visit each node only once.
What's even worse is that it uses tons of memory. So let's not use it when we
have just one mfnode
Reviewed By: jsgf
Differential Revision: D8298352
fbshipit-source-id: 818e6b0e89f68472bfd41e34cb67727b47f3010f
Summary:
Unfortunately `HgParents` can't represent all valid parents, because
it can't represent the semantically important case where `p1` is `None` and
`p2` is not. (For incoming changesets we'd like to keep full fidelity with
Mercurial.)
All the Thrift definitions store `p1` and `p2` separately anyway, so just make
that change throughout `RevlogChangeset` and `BlobChangeset`.
Reviewed By: StanislavGlebik
Differential Revision: D8374125
fbshipit-source-id: 63674daaad05d4d4cae3778744dbf1c14b3c2e3b
Summary: There shouldn't be more than one thread writing to the database, because it causes lags in slaves and they race for database locks between themselves. One write connection should be sufficient enough.
Reviewed By: StanislavGlebik
Differential Revision: D8348604
fbshipit-source-id: ceef081ed89611978accfa55969883078d65a58f
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary: See the comment for a full explanation.
Reviewed By: jsgf
Differential Revision: D8274912
fbshipit-source-id: 936baa60925fff5ddf1e903c8408410c0a429e77
Summary:
Fuse panics if there is an error. However the errors are quite possible - for
example, if client went away then sending stderr log panics and the whole
Mononoke server fails. This diff fixes it by using ignore res
Reviewed By: lukaspiatkowski
Differential Revision: D8276600
fbshipit-source-id: e0c91a082d80d7c54372171f9893f98b2a1e8304
Summary:
The new_blobimport job is having difficulties when the pool is too large, because the write transactions are taking too long. If the pool is configured to be 1 for it then everything seems fine and fast enough.
On the other hand the Mononoke server should have bigger connectino pool size to be able to quickly respond for read requests.
Reviewed By: farnz
Differential Revision: D8235413
fbshipit-source-id: 84e0013ce569c3f103a2096001605aab828d178c
Summary: See previous diff for the movitvation
Reviewed By: jsgf
Differential Revision: D8207273
fbshipit-source-id: 527c1d97546afedf36f84059ccb7c740bd412907
Summary:
Big change here is update to rand 0.5. This is a significant API
change. quickcheck still uses rand 0.4, so for quickcheck users I changed it so
that quickcheck re-exports the rand it uses. This means that quickcheck users
are unchanged aside from using quickcheck::rand, whereas direct rand users have
been updated to use the new API.
Reviewed By: farnz
Differential Revision: D8234503
fbshipit-source-id: f9e620851b8dfcc33f22a0af26122adcd5fbde39
