Summary:
D21573455 accidentally removed the scuba sampling for scrub_get
operations. Add this back in.
Reviewed By: StanislavGlebik
Differential Revision: D21638972
fbshipit-source-id: eee66dbce161de69246f4da0a15dc2cf00e1ba01
Summary:
See bottom diff in the stack for the motivation. Though you can probably guess
the motivation :)
Reviewed By: farnz
Differential Revision: D21623154
fbshipit-source-id: a0940d766a67080ddcb346c2e3313eb08699edad
Summary: First diff in the stack that removes getfiles since it's no longer needed.
Reviewed By: farnz
Differential Revision: D21623156
fbshipit-source-id: 44f310ec4e4f34845cc5bf1738f1a8ece14e6694
Summary:
This updates a placeholder in the authentication process for requests
in the rust thrift services to allow authentication with CATs. This change
allows for passing in a configurable server identity to be used as the
verifier identity here.
Additionally this sets the verifer identity to be "scm_service_identity"
for the mononoke scs server (the service identity for this service).
This is needed to use the mononoke scs server from corp.
Reviewed By: markbt
Differential Revision: D21560204
fbshipit-source-id: 02e41718307fddab6f0ac2101306bb7f4a59cf7e
Summary:
Currently, we have a gap between updating and subsequently reading the
value of `write_order`. If another blobstore's put operation completes before
we have called `record_put_stats` then we may have an incorrect `write_order`
value.
Whilst this is only a minor issue, let's fix it anyway. :-)
Reviewed By: krallin
Differential Revision: D21619669
fbshipit-source-id: 1b8bacbcb4c195e6765ebdfaa68425f286f88c06
Summary:
After other optimizations the CHashMap version of visit_count was showing as a hot spot.
Given the number of possible NodeTypes is small we can store the visit_count in array indexed by ordinal instead.
Reviewed By: farnz
Differential Revision: D21618518
fbshipit-source-id: 84978778034df11a9a48452adf9269db2dc17145
Summary:
Currently we record them only during pushrebase. Let's record during push as
well.
To simplify things a little bit let's allow only a very simple push case:
1) Single bookmark.
2) All pushed commits should be reachable by this bookmark.
Reviewed By: krallin
Differential Revision: D21451337
fbshipit-source-id: bf2f1e6025ac116fb8096824b7c4c6440d073874
Summary:
This is the fix that might be hard to explain.
First, why we need `generate_additional_bonsai_file_changes()` function at all.
TL;DR is to fix pushrebasing of a diamond merge commit. Merge bonsai changeset
must have a file change entry if p1 and p2 has the same file with different
content, and when merge commit is pushrebased we need to preserve this property.
more details are in D18065013 and in the comments.
Secondly, what was wrong with generate_additional_bonsai_file_changes. While it
was technically correct (i.e. it was producing valid bonsai changeset) it had
an unexpected side-effect for non-diamond merges (e.g. repo imports). It was
adding all changed files between root and onto commit (i.e. all changed files
from commits over which it was rebased). This in turn leads to unnecessary
push failures with "conflicts" errors.
Thirdly, why it was wrong. generate_additional_bonsai_file_changes
a) finds all files changed between onto and root (see comments for more details about onto
and root),
b)then finds those that needs to be added in the pushrebase merge bonsai changeset.
The problem was in step b). It needed to check which files exists in merge
parent but instead it checked which files exists in merge commit itself.
This is because merge commit itself cannot change any of the files that were changed between root
and onto - in that case we get a pushrebase conflict. So all the files come
either from root commit or from another parents. And the entry can become
"stale" only if it existed in one of the parents and later was changed between
root and onto.
Reviewed By: krallin
Differential Revision: D21596758
fbshipit-source-id: f2bea36fcc29fc736caaa7494696a1f6dc848b9e
Summary: Add MPathHash memoization to WrappedPath as the paths are hashed multiple times
Reviewed By: farnz
Differential Revision: D21613720
fbshipit-source-id: b850cf5ea1668b5ff75b07a0489b54f86078677a
Summary:
Currently, there isn't a way to override the value of a tunable during
a unit test. This is unfortunate and may push people to creating integration
tests when unit tests would be best.
Add `with_tunables`, which allows callers to override the tunables values that
a closure accesses. Further, add an example unit test that shows the behaviour.
Thanks to krallin for suggesting this and helping with the implementation
details.
Reviewed By: krallin
Differential Revision: D21595220
fbshipit-source-id: c31da88fb4fcfd38f761960ffe8025132025e6c9
Summary:
Let's add an option to log how many files and trees were fetched in a
particular repo that start with a prefix.
Reviewed By: farnz
Differential Revision: D21617347
fbshipit-source-id: a57f74eadf32781e6c024e18da252c98af21996d
Summary:
Just as for ints and bools, let's add support for strings.
A few notes:
1) we don't have AtomicString, so I decided to use ArcSwap<String>.
However to make code generation simpler (see tunables-derive) I've added a type
alias for that
2) Return type is Arc<String> to avoid unnecessary copies.Another option that we might have here is to put the whole Tunables structure inside ArcSwap, and changing `tunables()` to return Arc<MononookeTunables>
instead of &'static MononokeTunables. However that's a bigger change to make.
Reviewed By: markbt
Differential Revision: D21592390
fbshipit-source-id: 6d3cf340b13f7aef9adb2b1b99ed2bf260033285
Summary:
The blobstore multiplexer contains logic to log blobstore operations to
scuba, as well as updating `PerfCounters`. There are some cases where we don't use the
multiplexed blobstore, which means that we're missing this important logging.
Factor out the logging to a separate crate and implement `LogBlob`, which wraps
another blobstore and performs logging to both scuba and PerfCounters.
Reviewed By: StanislavGlebik
Differential Revision: D21573455
fbshipit-source-id: 490ffd347f1ad19effc93b93f880836467b87651
Summary:
The configerator thrift file was updated to remove a path from hook
config. However, this change wasn't synced to fbsource.
Sync the change and fix up the tests that are broken by this change.
Reviewed By: krallin
Differential Revision: D21594221
fbshipit-source-id: 7b64180914f6c6802e4d70fcb1a5d6ec36eb2eac
Summary:
This adds support for periodically logging that a command is in progress in
Mononoke. The underlying motivation is to make sure that if something is taking
a long time, we can still show some feedback to the user (and log to Scuba). We
might want to log this every 30 seconds.
That said, this is more of an RFC at this stage. I'm thinking it might make
sense to log to Scuba more often and to users less often. It might make sense
to also restrict this to specific commands, such as unbundle:
https://fburl.com/scuba/mononoke_test_perf/atik5959
Reviewed By: StanislavGlebik
Differential Revision: D21549862
fbshipit-source-id: 1d02c5c926abc7e491ac5b8ae0244b5f4620c93e
Summary: Same as the previous diff, but for commands that return a stream.
Reviewed By: StanislavGlebik
Differential Revision: D21549864
fbshipit-source-id: ba8c14db34a651cd4ddbc1c8b9ad382c08cc775d
Summary:
This doesn't do anything on its own, but it's refactoring I need for later in
this stack. It wraps all our commands in a command_future call that gives us an
opportunity to wrap the future being returned. We used to use `start_command`
to get the context, so this just replaces that.
Reviewed By: StanislavGlebik
Differential Revision: D21549863
fbshipit-source-id: 0e613bb1db876d27d662fd6c993d7b7d954b5f2b
Summary:
This allows us to find the closest globalrev (i.e. the highest that is lower
than or equal to the one being requested). Conceptually, this is similar to
prefix lookups, which is why we're including it in the prefix lookup function
method!
Reviewed By: markbt
Differential Revision: D21553081
fbshipit-source-id: 4b08f7e2478210d406b1a043e95f8768a3dcc45a
Summary:
The output streams from the walks are of the form (Key, Payload, Stats).
Where the Key is Node and the Payload is NodeData this is ok, but with the key and payload both tuples it gets hard to read, so this introduces named tuple-like structs for clarity.
Reviewed By: StanislavGlebik
Differential Revision: D21504916
fbshipit-source-id: a856d34af4117d3183ef0741b311c1c34cf9dacc
Summary:
Add a --sample-path-regex option for use in the corpus dumper so we can dump out just a subset of directories from a repo.
This is most useful on large repos.
Reviewed By: farnz
Differential Revision: D21325548
fbshipit-source-id: bfda87aa76fbd325e4e01c2df90b5dcfc906a8f6
Summary:
Track path mtime, as being able to order by mtime is important to be able to use the on disk corpus to evaluate delta compression approaches
The dumped blobs mtime is set based on the last traversed bonsai or hg commit's timestamp. For Bonsai it prefers committer_time if present and if not falls back to author_time.
Reviewed By: farnz
Differential Revision: D21312223
fbshipit-source-id: fa14615603f78675ca54a0f4946cc8480b8eade5
Summary:
Update the corpus walker to dump the sampled bytes as early as possible to the Inflight area of the output dir, then move them to final location once path is known.
When walking large files and manifests this uses a lot less memory that holding the bytes in a map!
Layout is changed is to make comparison by file type easier. we get a top level dir per extension, e.g. all .json files are under FileContent/byext/json
This also reduces the number of bytes taken from the sampling fingerprint used to make directories, 8 was overkill. 3 is enough to limit directory size.
Reviewed By: farnz
Differential Revision: D21168633
fbshipit-source-id: e0e108736611d552302e085d91707cca48436a01
Summary:
Add corpus dumper for space analysis
This reuses the path based tracking from compression-benefit and the size sampling from scrub.
The core new functionality is the dump to disk from inside corpus stream.
Reviewed By: StanislavGlebik
Differential Revision: D20815125
fbshipit-source-id: 01fdc9dd69050baa8488177782cbed9e445aa3f7
Summary:
We now have auto pull logic that covers most unknown rev use-cases. The hint
message is no longer necessary. It's also unclear how to use `hg pull`
correctly. For example, should it be `-r`, `-B remote/foo` or `-B foo`?
Reviewed By: DurhamG
Differential Revision: D21526667
fbshipit-source-id: 40583bfb094e52939130250dd71b96db4d725ad5
Summary:
I want to reuse the functionality provided by `fetch_all_public_changesets`
in building Segmented Changelog. To share the code I am adding a new crate
intended to store utilities in dealing with bulk fetches.
Reviewed By: krallin
Differential Revision: D21471477
fbshipit-source-id: 609907c95b438504d3a0dee64ab5a8b8b3ab3f24
Summary:
If a push or pull operation doesn't involve any changesets for which mutation
information might be relevant, don't spend any time querying the database, and
instead exit early.
Reviewed By: krallin
Differential Revision: D21549937
fbshipit-source-id: a6f992e621456b826acd1bddde3591e751d23b31
Summary:
MySQL returns an error for a query of the form `WHERE value IN ()`. Avoid
these by checking that collections are not empty before making the query.
Reviewed By: krallin
Differential Revision: D21549690
fbshipit-source-id: 1507d36e81f7a743d2a1efb046e52a5479633ab9
Summary:
The `test-infinitepush-mutation.t` test covers the new mutation database, so
add it to the mysql tests.
Reviewed By: krallin
Differential Revision: D21548966
fbshipit-source-id: 0dc1f90129fa61fb6db1c1b5a747efa3d20041f5
Summary:
The `support_bundle2_listkeys` flag controls at runtime whether we support
`listkeys` in bundle2. Since this was added before tunables were available,
it uses a value in the mutable counters SQL store.
We could migrate this to tunables, but in practice we have never disabled it,
so let's just make it the default.
Reviewed By: krallin
Differential Revision: D21546246
fbshipit-source-id: 066a375693757ea841ecf0fddb0cc91dc144fd6f
Summary:
When the client pulls draft commits, include mutation information in the bundle
response.
Reviewed By: farnz
Differential Revision: D20871339
fbshipit-source-id: a89a50426fbd8f9ec08bbe43f16fd0e4e3424e0b
Summary:
Advertise support for `b2x:infinitepushmutation`. When the client sends us
mutation information, store it in the mutation store.
Reviewed By: mitrandir77
Differential Revision: D20871340
fbshipit-source-id: ab0b3a20f43a7d97b3c51dcc10035bf7115579af
Summary: Add the mutation store to blobrepo.
Reviewed By: krallin
Differential Revision: D20871336
fbshipit-source-id: 777cba6c2bdcfb16b711dbad61fc6d0d2f337117
Summary: What it says in the title
Reviewed By: StanislavGlebik
Differential Revision: D21549635
fbshipit-source-id: 75939ebbfb317a9beaa9acd1fc1a7c6f41b0f88f
Summary:
subcommand_single calls `derived_data_utils.regenerate(vec![cs_id])` with the
intention that derived data for this commit will be regenerated. However
previously it didn't work because DerivedDataUtils::derive() was ignoring
regenerate parameter. This diff fixes it.
Reviewed By: krallin
Differential Revision: D21527344
fbshipit-source-id: 56d93135071a7f3789262b7a9d9ad84a0896c895
Summary: Add a `/history` endpoint that serves EdenAPI history data. Like the other endpoints, this one currently buffers the response in memory, and will be modified to return a streaming response in a later diff.
Reviewed By: krallin
Differential Revision: D21489463
fbshipit-source-id: 259d2d1b7d700251fe902f1ac741545e5895404a
Summary: Factor out logic that will be common to many handlers into new functions in the `util` module.
Reviewed By: krallin
Differential Revision: D21489469
fbshipit-source-id: 9aff4e5182748ab0a0bedd6038852692b8e721a7
Summary: Break up the EdenAPI server integration tests to prevent the test from getting too long.
Reviewed By: krallin
Differential Revision: D21464056
fbshipit-source-id: 076aaf8717547fe9188f40c078d577961c02325d
Summary: Add an endpoint that serves trees. Uses the same underlying logic as the files endpoint, and returns the requested nodes in a CBOR DataResponse.
Reviewed By: krallin
Differential Revision: D21412987
fbshipit-source-id: a9bcc169644a5889c3118a3207130228a5246b2f
