Summary:
The rust-crypto library appears to be unmaintained, switching
- `crypto::digest::Digest` to `digest::Digest`
- `crypto::sha1::Sha1` to `sha1::Sha1`
- `crypto::sha2::Sha256` to `sha2::Sha256`
Reviewed By: jsgf
Differential Revision: D20456962
fbshipit-source-id: 2e3406dedba05245265d96b480c35ba2421aa3fd
Summary: Updated fmt version to be on par with buck build. It was causing inconsistencies.
Reviewed By: vitaut
Differential Revision: D20528011
fbshipit-source-id: d9e04ed2c28b839eaeff24120162c4db4732fa55
Summary:
The rust-crypto library appears to be unmaintained, switching
- `crypto::digest::Digest` to `digest::Digest`
- `crypto::sha1::Sha1` to `sha1::Sha1`
- `crypto::sha2::Sha256` to `sha2::Sha256`
Reviewed By: jsgf
Differential Revision: D20456840
fbshipit-source-id: 90cc031ec5402b60b6eb06a301a3733bd92bbc69
Summary: For LFS blobs, these can be obtained very easily by querying the ContentStore.
Reviewed By: DurhamG
Differential Revision: D20504235
fbshipit-source-id: 937ef20184d6524b1355565f9ab81e40b56d7ab0
Summary:
This diff asyncifies the `loop_fn` call in `run_statistics`.
I was unable to find an existing example of asyncifying an infinite
loop - my solution requires allowing the `Ok` around my `loop`
to be unreachable via the `#[allow(unreachable_code)]` annotation. There may be
a better solution.
We also swap out the `tokio-timer` dependency, which uses old-style
futures, for current-version `tokio` so we can use the new-style future
`tokio::time::delay_for`.
Reviewed By: farnz
Differential Revision: D20527530
fbshipit-source-id: 90d30ec9465402d06d3b4b30c1bbd5e340ac94b6
Summary:
This is only intended for Mercurial .t tests and not in any production
environment.
Reviewed By: DurhamG
Differential Revision: D20504236
fbshipit-source-id: 618e17631b73afa650875cb7217ba7c55fb9f737
Summary:
This will enables the fast-path for comparing LFS blobs without reading the
entire blob.
Reviewed By: DurhamG
Differential Revision: D20504233
fbshipit-source-id: 446cec57fba77e02cc7070203bd759d341fc01ab
Summary:
For now, this is only used for LFS, as this is the only store that can
correctly answer both.
This API will be exposed to Python to be able to have cheap filectx comparison,
and other use cases.
Reviewed By: DurhamG
Differential Revision: D20504234
fbshipit-source-id: 0edb912ce479eb469d679b7df39ba80fceef05f2
Summary:
This enables fetching blobs from the LFS server. For now, this is limited to
fetching them, but the protocol specify ways to also upload. That second part
will matter for commit cloud and when pushing code to the server.
One caveat to this code is that the LFS server is not mocked in tests, and thus
requests are done directly to the server. I chose very small blobs to limit the
disruption to the server, by setting a test specific user-agent, we should be
able to monitor traffic due to tests and potentially rate limit it.
Reviewed By: DurhamG
Differential Revision: D20445628
fbshipit-source-id: beb3acb3f69dd27b54f8df7ccb95b04192deca30
Summary: Most of our keys are prefixed, let's prefix this as well
Differential Revision: D20536231
fbshipit-source-id: 9a2ffecfc7de46d109a9fba2444212735cacbebf
Summary:
Changeset info is less expensive to load than Bonsai, so we would like to use it in SCS as a source of commit info if possible.
This diff adds a method into the Repo object that checks `changeset_info` derivation is enabled for the repo in the `DerivedDataConfig`. If derivation is enabled, then SCS derives this info otherwise it awaits for bonsai and converts it into the changeset info. The bonsai fields aren't copied but moved to the `ChangesetInfo`.
Reviewed By: StanislavGlebik
Differential Revision: D20282403
fbshipit-source-id: b8ddad50dcd5c6de109728b2081ca5a13f440988
Summary:
This diff asyncifies the outermost layer of `statistics_collector`,
so that `main` doesn't need a `compat`, by extracting the
futures portion of `main` into an async function `run_statistics`
and using async futures for the outermost layers of `run_statistics`
logic (everything outside the `loop_fn` call)
Reviewed By: farnz
Differential Revision: D20527529
fbshipit-source-id: 00ad9033584360f45715719f2636dcfac1926004
Summary:
This is the start of migrating blackbox events to tracing events. The
motivation is to have a single data source for log processing (for simplicity)
and the tracing data seems a better fit, since it can represent a tree of
spans, instead of just a flat list. Eventually blackbox might be mostly
a wrapper for tracing data, with some minimal support for logging some indexed
events.
Reviewed By: DurhamG
Differential Revision: D19797710
fbshipit-source-id: 034f17fb5552242b60e759559a202fd26061f1f1
Summary:
The `all()` revset is much slower with narrow-heads for correctness. Use an
alternative that is fast.
Reviewed By: markbt
Differential Revision: D20528063
fbshipit-source-id: c8ae35e67e60407406ca81d67878278392626e9a
Summary: A little step towards asyncifying the filestore. This is just mechanical, without removing clones. TBD: add a diff, which starts to actually use the benefits of new futures.
Reviewed By: farnz
Differential Revision: D20534272
fbshipit-source-id: a038e6f22b666f3f2c9782ee25c0c2582ddced6c
Summary:
Last diff. Fully migrates all of populate_healer.rs to async/await
futures. This makes `put_resume_state()` async, with one `.compat()` call needed
for dealing with `manifold.put()`. Also changes `populate_healer_queue()` to
use the new async `put_resume_state()`. At this point, the only `.compat()`
calls remaining are for interop with ThrifManifoldBlob's interface, and can be
removed once ThriftManifoldBlob is updated or provides async replacement
functions. All explicit old-style future creation sites have been removed in
favor of 0.3 futures.
Reviewed By: krallin
Differential Revision: D20479264
fbshipit-source-id: baad535da3fc8b621d72de567454bcd64862977a
Summary: Moves to using 0.3 futures inside of the populate_healer_queue() function. This leaves only one remaining source of `.compat()` calls inside of populate_healer.rs, which will be removed in the following diff.
Reviewed By: krallin
Differential Revision: D20473834
fbshipit-source-id: 6d76e0673b875fba15611a495d86b9ca0b1695db
Summary:
Run buck build -c rust.clippy=true eden/mononoke/:mononoke#check and fix some
of them manually. I wasn't able to make rustfix to work - will try to see
what's wrong and run it.
The suggestions looks non-controversial
Reviewed By: krallin
Differential Revision: D20520123
fbshipit-source-id: 25d4eb493f2363c5aa77bdb3876da4378483f6cb
Summary: This makes thigs a little more readable.
Reviewed By: krallin
Differential Revision: D20515645
fbshipit-source-id: ae04e18b0f415353431a995ae22844f6e301780c
Summary:
This is going to be used in D20469131, but in a nutshell the idea is to
perform as many checks as possible before actually doing the rechunking.
This way we can avoid churning through the entire blobstore.
Reviewed By: krallin
Differential Revision: D20491189
fbshipit-source-id: 4f7c2a8e02c890db789d25aa819b5c91d08ea7be
Summary:
The way decoders work in Tokio is that they get repeatedly presented whatever
is on the wire right now, and they have to report whether the data being
presented is valid and they'd like to consume it (and otherwise expect Tokio to
provide more data).
It follows that decoders have to be pretty fast, because they will be presented
a bunch of data a bunch of times. Unfortunately, it turns out our SSH Protocol
decoder is everything but.
This hadn't really been a problem until now, because we had ad-hoc decoding for
things like Getpack that might have a large number of parameters, but for now
the designated nodes implementation is decoded in one go through the existing
Gettreepack decoder, so it is important ot make the parsing fast (not to
mention, right now, we buffer the entire request for Getpack as well ... so
maybe we could actually update it to this too!).
Unfortunately, as I mentioned, right now the parsing wasn't fast. The reason is
because it copies parameters to a `Vec<u8>` while it decodes them. So, if
you start decoding and copying, say, 50MB of arguments, before you find out
you're missing a few more bytes, then you just copied 50MB that you need to
throw away.
Unfortunately, the buffer size is 8KiB, so if we say "I need more data", we get
8KiB. That means that if we want to decode a 70MiB request, we're going to make
8960 ( = 70 * 1024 / 8) copies of the data (the first 8KiB, then the first 16,
and so on), which effectively means we are going to copy and throw away ~612GiB
of data (8960 * 70 / 2). That's a lot of work, and indeed it is slow.
Fortunately, our implementation is really close to doing the right thing. Since
everything is length delimited, we can parse pretty quick if we don't make
copies: all we need to do is read the first length, skip ahead, read the second
length, and so on.
This is what this patch does: it extracts the parsing into something that
operates over slices. Then, **assuming the parsing is successful** (and that is
the operative part here), it does the conversion to an owned Vec<u8>.
In O(X) terms .. this means the old parsing is O(N^2) and the new one is O(N).
I actually think we could take this one step further and do the conversion even
later (once we want to start decoding), but for now this is more than fast
enough.
After this patch, it takes < 1 second to parse a 70MiB Gettreepack request.
Before this patch, it took over 2 minutes (which is 3 times longer than it
takes to actually service it).
PS: While in there, I also moved the `gettreepack_directories` function to a
place that makes more sense, which I had introduced earlier in the wrong place
(`parse_star`, `parse_kv` and `params` are a group of things that go together,
it's a bit clowny to have `gettreepack_directories` in the middle of them!).
Reviewed By: kulshrax
Differential Revision: D20517072
fbshipit-source-id: 85b10e82768bf14530a1ddadff8f61a28fdcbcbe
Summary: The `Node` type in Mercurial's Rust code was renamed to `HgId`, with an alias to `Node` to keep older code building. Let's rename the usages in Mononoke to `HgId` to reduce ambiguity and keep the terminology consistent with Mercurial.
Reviewed By: StanislavGlebik
Differential Revision: D20460543
fbshipit-source-id: f6d8e3aef42743370323cde79ec10b21de956313
Summary:
It was (or rather, might have been) useful during debugging of S197766.
Let's now count both "count" (i.e. how often the method was called)
and count how many filenodes were inserted
Reviewed By: krallin
Differential Revision: D20519701
fbshipit-source-id: f19f413171fcbcc300deffbe29baa946ebbe8dce
Summary: Based on comments on D20382825, we need to make sure that `_gettrees()` knows for sure whether on-demand tree fetching is in use in order to properly identify missing nodes in the response.
Reviewed By: quark-zju
Differential Revision: D20520439
fbshipit-source-id: ffa6d62dbe8b6f641b1dacebcb6f94ceae714c1b
Summary: 'new' is not very explicit with the fact that things are not refreshed.
Reviewed By: dtolnay
Differential Revision: D20356129
fbshipit-source-id: ff4a8c6fe4c34e93729c902e4b41afbe3c9deca1
Summary:
Now Segment has no lifetime we can create it directly and return the ownership.
Performance of "building segments" does not seem to change:
# before
building segments 750.129 ms
# after
building segments 712.177 ms
Reviewed By: sfilipco
Differential Revision: D20505200
fbshipit-source-id: 2448814751ad1a754b90267e43262da072bf4a16
Summary:
This allows structures like BTreeMap to own and store Segment.
It was not possible until D19818714, which adds minibytes::Bytes interface for
indexedlog.
In theory this hurts performance a little bit. But the perf difference does not
seem visible by `cargo bench --bench dag_ops`:
# before
building segments 714.420 ms
ancestors 54.045 ms
children 490.386 ms
common_ancestors (spans) 2.579 s
descendants (small subset) 406.374 ms
gca_one (2 ids) 161.260 ms
gca_one (spans) 2.731 s
gca_all (2 ids) 287.857 ms
gca_all (spans) 2.799 s
heads 234.130 ms
heads_ancestors 39.383 ms
is_ancestor 113.847 ms
parents 251.604 ms
parent_ids 11.412 ms
range (2 ids) 117.037 ms
range (spans) 241.156 ms
roots 507.328 ms
# after
building segments 750.129 ms
ancestors 53.341 ms
children 515.607 ms
common_ancestors (spans) 2.664 s
descendants (small subset) 411.556 ms
gca_one (2 ids) 164.466 ms
gca_one (spans) 2.701 s
gca_all (2 ids) 290.516 ms
gca_all (spans) 2.801 s
heads 240.548 ms
heads_ancestors 39.625 ms
is_ancestor 115.735 ms
parents 239.353 ms
parent_ids 11.172 ms
range (2 ids) 115.483 ms
range (spans) 235.694 ms
roots 506.861 ms
Reviewed By: sfilipco
Differential Revision: D20505201
fbshipit-source-id: c34d48f0216fc5b20a1d348a75ace89ace7c080b
Summary: Now that we sort the errors, we don't need this condition anymore.
Reviewed By: xavierd
Differential Revision: D20517578
fbshipit-source-id: 7012de387ee8acee1c1b630991f3a289a3fa48d1
Summary:
EdenFS is reported as `osxfuse_eden` on OSX after D20313385.
Update the fscap table to avoid slow paths testing fs capabilities.
Without this diff, churns on edenfs OSX will trigger undesirable watchman
events.
Reported by: fanzeyi
Reviewed By: fanzeyi
Differential Revision: D20518902
fbshipit-source-id: 2e8e472df16d08b17834b2c966c065bbaad052fe
Summary:
Now that Arun is about to roll this out to the team, we should get some more
logging in place server side. This updates the designated nodes handling code
to report whether it was enabled (and log prior to the request as well).
Reviewed By: HarveyHunt
Differential Revision: D20514429
fbshipit-source-id: 76ce62a296fe27310af75c884a3efebc5f210a8a
Summary:
The later is what is now recommended, and no longer requires a macro to
initialize a lazy value, leading to nicer code.
Reviewed By: DurhamG
Differential Revision: D20491488
fbshipit-source-id: 2e0126c9c61d0885e5deee9dbf112a3cd64376d6
Summary:
Lots of different warnings on this one. Main ones were:
- One bug where .write was used instead of .write_all
- Using .next instead of .nth(0) for iterators,
- Using .cloned() instead of .map(|x| x.clone())
- Using conditions as expressions instead of mut variables
- Using .to_vec() on slices instead of .iter().cloned().collect().
- Using .is_empty instead of comparing .len() against 0.
Reviewed By: DurhamG
Differential Revision: D20469894
fbshipit-source-id: 3666a44ad05e0fbfa68d490595703c022073af63
