Summary:
Move expected_item_size_byte into CachelibSettings, seems like it should be there.
To enable its use also exposes a parse_and_init_cachelib method for callers that have different defaults to default cachelibe settings.
Reviewed By: krallin
Differential Revision: D24761024
fbshipit-source-id: 440082ab77b5b9f879c99b8f764e71bec68d270e
Summary:
We moved to APFS subvolumes a while back, so it's time
to kill this bit.
Reviewed By: chadaustin
Differential Revision: D25260825
fbshipit-source-id: 7487fb6d477f2650e2aedb0c4dfc4f45a10c9807
Summary:
Introduce `edenapithin` crate, which offers C bindings to EdenAPI Client.
There are two top-level `owned` types, `EdenApiClient` and `TreeEntryFetch`, which represent `Box`ed results from calls to EdenAPI's blocking client. The C / C++ code is responsible for calling the associated `free` functions for these types, as well as `OwnedString`, which is used to represent error variants of a `Result` as a string.
Most functionality is provided through functions which operate on and return references into these top-level owned types, providing access into Rust `Result`s and `Vec`s (manually-monomorphized), and EdenApi's `TreeEntry` and `TreeChildEntry`.
Additional non-pointer types are defined in the `types` module, which do not require manual memory management.
C++ bindings are not included currently, but will be introduced soon.
Reviewed By: fanzeyi
Differential Revision: D24866065
fbshipit-source-id: bb15127b84cdbc6487b2d0e1798f37ef62e5b32d
Summary:
Introduce a new API type, `TreeAttributes`, corresponding to the existing type `WireTreeAttributesRequest`, which exposes which optional attributes are available for fetching. An `Option<TreeAttributes>` parameter is added to the `trees` API, and if set to `None`, the client will make a request with TreeAttributes::default().
The Python bindings accept a dictionary for the attributes parameter, and any fields present will overwrite the default settings from TreeAttributes::default(). Unrecognized attributes will be silently ignored.
Reviewed By: kulshrax
Differential Revision: D25041255
fbshipit-source-id: 5c581c20aac06eeb0428fff42bfd93f6aecbb629
Summary: Basic observability for how the segmeted changelog update process is performing.
Reviewed By: krallin
Differential Revision: D25108739
fbshipit-source-id: b1f406eb0c862464b186f933d126e0f3a08144e4
Summary:
The update of the segmented changelog is light weight enough that we can
consider all repositories sharing a common tailer process. With all
repositories sharing a single tailer the the maintenance burden will be lower.
Things that I am particularly unsure about are: tailer configuration setup and
tailer structure. With regards to setup, I am not sure if this is more or less
than what production servers do to instantiate. With regards to structure, I
think that it makes a lot of sense to have a function that takes a single repo
name as parameter but the configuration setup has an influence on the details.
I am also unsure how important it is to paralelize the instantiation of the
blobrepos.
Finally, it is worth mentioning that the SegmentedChangelogTailer waits for
`delay` after an update finishes rather than on a period. The benefit is that
we don't have large updates taking down a process because we schedule the same
large repo update too many timer. The drawback is that scheduling gets messed
up over time and multiple repo updates can end up starting at the same time.
Reviewed By: farnz
Differential Revision: D25100839
fbshipit-source-id: 5fff9f87ba4dc44a17c4a7aaa715d0698b04f5c3
Summary:
Stop implementing the legacy glob() Thrift function, since it's
deprecated, and just noise at this point.
Reviewed By: xavierd
Differential Revision: D25247641
fbshipit-source-id: a022fee169ad54c886d8f300b57bef233453fe8b
Summary: This makes it easier to enable all derived data for scrubbing
Reviewed By: ikostia
Differential Revision: D25188963
fbshipit-source-id: e9c981e33273d6b2eeadcce0d0a341b33e91e42d
Summary:
Show the options for specifying node and edge types in the --help output
These changes removed the last use of lazy_static in the walker, so updated TARGETS as well.
Reviewed By: krallin
Differential Revision: D25188964
fbshipit-source-id: c5ccb4f5a0f3be1b8cb7d51cd5f99236d60d3029
Summary:
It has a build() method and later in stack it will build a mononoke
specific type rather than the clap::App
Differential Revision: D25216827
fbshipit-source-id: 24a531856405a702e7fecf54d60be1ea3d2aa6e7
Summary:
Mercurial is incompatible with TSAN at the moment, due to Rust/C++
compilation issues, Python multiprocess, and Tokio false positives. So
disable our HG tests when running under TSAN.
Reviewed By: fanzeyi
Differential Revision: D25109413
fbshipit-source-id: 86e51ebd287e10f92d6e3b8e7bf191e7946c709a
Summary: Adding `full_idmap_clone` to edenapi and usign that in `debugsegmentclone`.
Reviewed By: quark-zju
Differential Revision: D25139730
fbshipit-source-id: 682055f7c30a94a941acd16e2b8e61b9ea1d0aef
Summary:
This method reconstructs a dag from clone data.
At the moment we only have a clone data construction method in Mononoke. It's
the Dags job to construct and import the clone_data. We'll consolidate that at
a later time.
Reviewed By: quark-zju
Differential Revision: D24954823
fbshipit-source-id: fe92179ec80f71234fc8f1cf7709f5104aabb4fb
Summary:
The server expects post requests. At this time we don't want to cache this data
so POST.
Reviewed By: singhsrb
Differential Revision: D24954824
fbshipit-source-id: 433672189ad97100eee7679894a894ab1d8cff7b
Summary:
The config is something that makes sense for all commands to have access to.
Commands that don't use a repo don't have access to the config that is prepared
by the dispatches. This change is a stop-gap to allow new commands that don't
require a repository to receive the config as an argument.
The construction of the config is something that we should iterate on. I see
the current implementaiton as a workaround.
Reviewed By: quark-zju
Differential Revision: D24954822
fbshipit-source-id: 42254bb201ba8838e7cc107394e8fab53a1a95c7
Summary:
At the moment if we try to sync a bookmark entry but from_cs_id of bookmark
entry doesn't match the value of the bookmark on hg servers then the sync will
fail.
Let's add an option that in the case of this mismatch sets from_cs_id to the
current value on hg servers.
Reviewed By: krallin
Differential Revision: D25242172
fbshipit-source-id: 91180fb86f25d10c9ba2b78d7aa18ed0a52d13a5
Summary: convert mercurial_derived_data to new type futures
Reviewed By: ahornby
Differential Revision: D25220329
fbshipit-source-id: c2532a12e915b315fe6eb72f122dbc37822bbb2a
Summary:
This diff prepares the Mononoke codebase for composition-based extendability of
`ScubaSampleBuilder`. Specifically, in the near future I will add:
- new methods for verbose scuba logging
- new data field (`ObservabilityContext`) to check if verbose logging should
be enabled or disabled
The higher-level goal here is to be able to enable/disable verbose Scuba
logging (either overall or for certain slices of logs, like for a certain
session id) in real time, without restarting Mononoke. To do so, I plan to
expose the aforementioned verbose logging methods, which will run a check
against the stored `ObservabilityContext` and make a decision of whether the
logging is enabled or not. `ObservabilityContext` will of course hide
implementation details from the renamed `ScubaSampleBuilderExt`, and just provide a yes/no
answer based on the current config and sample fields.
At the moment this should be a completely harmless change.
Reviewed By: krallin
Differential Revision: D25211089
fbshipit-source-id: ea03dda82fadb7fc91a2433e12e220582ede5fb8
Summary:
cross repo bookmark validation alarm fired a few times, and looks like it fired
because of the following:
1) find_bookmark_diff compared boomarks and found an inconsistency for bookmark
BM which points to commit A in large repo. Next step is to check bookmark history
2) While find_bookmark_diff was running a new commit B was landed in a large repo
and was backsynced to the small repo, so BM now points to commit B.
3) check_large_bookmark_history is called and it fetches latest bookmark log entries, and
it gets entries for commit A and commit B. check_large_bookmark_history checks
if any of the fetched entries points to a commit in the small repo and if yes then
it also checks if this bookmark update happened not so long ago. And the
problem is in the way it checks the "not so long ago" part. It does so by
finding the time difference between latest bookmark update log entry and any
other bookmark update log entry.
Now, if time difference between these two log entries (for commit B and for
commit A) is more than max_delay_secs (which happens only
if commit rate is low e.g. during the weekends), then the alarm would fire
because the delay between latest bookmark update log entry (the one that moved
BM to commit B) and previous log entry (the one that moved BM to commit A) is too large.
This diff fixes this race by skipping newest entries until we found a bookmark
update log entry which points to the large commit that find_bookmark_diff
returned.
Reviewed By: ikostia
Differential Revision: D25196760
fbshipit-source-id: dfa0dca0001b1c38759ec9f4f790cfa3197ae2cf
Summary: remove dependency on old futures from derived data filenodes
Reviewed By: ahornby
Differential Revision: D25218521
fbshipit-source-id: 4d7eaf42c3ba15ea09276a7f3567128d5216e814
Summary: `derived_data:derived_data` had already been almost converted, I've cleaned up some test so it would be possible to completely remove old futures dependency
Reviewed By: StanislavGlebik
Differential Revision: D25197406
fbshipit-source-id: 064439f42a15f715befc019e5350dda0a2975f2b
Summary:
- convert save_bonsai_changesets to new type futures
- `blobrepo:blobrepo` is free from old futures deps
Reviewed By: StanislavGlebik
Differential Revision: D25197060
fbshipit-source-id: 910bd3f9674094b56e1133d7799cefea56c84123
Summary: Add support for walking fast entries for directories so we can scrub and inspect them
Differential Revision: D25187551
fbshipit-source-id: 812f9fd82459ef49dcd781c318fbe5c398daad21
Summary: Add FastlogFile to walker so it can be inspected and scrubbed. Directory and batch components of fastlog covered in following diffs.
Differential Revision: D25187549
fbshipit-source-id: c046cbf2561cdbbc9563497119e34d1b09d0ebef
Summary:
On Python 3 we only support utf8 files. Python 3 has a way of
representing non-utf8 strings in utf8 format by utilizing surrogateescape, but
these strings cause issues in other parts of the codebase that don't expect
suorrageescaped strings (like the Rust code). Since we simply do not support
these paths, let's filter them out as soon as we get them from Watchman.
Reviewed By: quark-zju
Differential Revision: D25134079
fbshipit-source-id: 8be6893a6114b816097422f4469ac317fa3795d1
Summary: As HG<->Mononoke will go through proxygen, testing showed that Proxygen forces us to use 'Upgrade: websocket' and confirm with the Websocket handshake. Adjust accordingly to do so.
Reviewed By: ahornby
Differential Revision: D25197395
fbshipit-source-id: ca00ac31be92817c6f1a99d7d492469b17b46286
Summary:
This diff adds bundle combining to hg sync job. See motivation for doing that in D25168877 (cebde43d3f).
Main struct here is BookmarkLogEntryBatch. This struct helds a vector of BookmarkUpdateLogEntry that were combined (they are used mostly for logging) and also it contains parameters necessary for producing the bundle, notably from/to changeset ids and bookmarks. This struct has try_append method that decides whether it's possible to combine bundles or not.
Reviewed By: mitrandir77
Differential Revision: D25186110
fbshipit-source-id: 77ce91915f460db73d8a996efe415954eeea2476
Summary:
Use of `RootSkeletonManifest::batch_derive` is wrong, as it doesn't take into
account manifests that are already derived. Just use the normal derivation
method.
Reviewed By: StanislavGlebik
Differential Revision: D25218197
fbshipit-source-id: 60f2faad610d507a9659dc37ba72516431a9c036
Summary: The verify-manifests command can take a while. Add logging to show progress is being made.
Reviewed By: StanislavGlebik
Differential Revision: D25099918
fbshipit-source-id: 2b659f1a6921ed111441b2808ce29998bd2285dd
Summary:
If files are only modified, and not added or deleted, the skeleton manifest
doesn't change. Avoid duplicate writes by skipping the write if the skeleton
manifest we derive has the same ID as one of its parents.
Reviewed By: ahornby
Differential Revision: D25099917
fbshipit-source-id: 62900406711becea88491e706a09c6032109c964
Summary:
Improve backfill_derived_data logging to make it clear when a chunk is started
and when already-derived changesets have been skipped.
Fix the counters to take into account skiped changesets, and make the time
remaining human-readable.
Reviewed By: StanislavGlebik
Differential Revision: D25099919
fbshipit-source-id: b39d4b99ef0e79d253de2aa0c91f93276b561680
Summary:
Add skeleton manifests to `derived-data verify-manifests` and add a
`skeleton-manifests` subcommand that allows listing and traversing of skeleton
manifests.
Since skeleton manfiests don't contain the content id or file type, they are
valid simply if they contain the same set of files as the manifest we are
comparing against.
To allow testing of these items while backfilling is still in progress, add an
`--if-derived` option to these commands that work even if derivation is
disabled, provided the manifest has already been derived.
Reviewed By: ikostia
Differential Revision: D25064785
fbshipit-source-id: a6f923bfc53262a5b2118917f8fdd3e99407e036
Summary:
Avoid clients needing to make a secondary look-up for the commit title or message, which
we already have, by returning it in the blame response if the clients request it.
To do this, we add `format_options` to the blame request parameters, which allows customization
of the returned response. The default value if not specified corresponds to the old
behaviour for backwards compatibility.
Reviewed By: mitrandir77
Differential Revision: D25057231
fbshipit-source-id: 84faad7b2db1f684beb94d9fa799c8cef1a89ce8
Summary: Use these in next diff from walker so we can start to scrub fastlog
Reviewed By: aslpavel
Differential Revision: D25187548
fbshipit-source-id: 4188a0c9bdb51a898817b6f5801fe473b6712385
Summary: The path isn't needed to load the UnodeManifest, it's only needed as route information for compression etc
Reviewed By: aslpavel
Differential Revision: D25187547
fbshipit-source-id: b8199a81c5dae4caceed5d455fa6a9bbc3f037ac
Summary: The path is not needed to load the unode, only for route tracking
Reviewed By: aslpavel
Differential Revision: D25187553
fbshipit-source-id: baf6df3b7f56f5ca5a0ba086097acba4ecd82e75
Summary: Change from the walk_blame boolean to a bitflags struct so can also represent fastlog. Using bitflags crate so as not to grow the node size.
Reviewed By: aslpavel
Differential Revision: D25185267
fbshipit-source-id: b45574cd6c4d048aec321357ae54e2ec404f84fc
Summary: Few things were not on lexical order, clean them up. Makes it easier to read next diff without this mixed in
Reviewed By: aslpavel
Differential Revision: D25188965
fbshipit-source-id: 38b7ea6f0e2c0583e6701550d6db16d85ba1a455
Summary:
Previously, the commitcloud reverse filler would only run for a single
repo. This is fine for some of the high throughput repos (e.g. fbsource).
However, for small repos it doesn't make sense to use a server per repo.
Most of support was already completed with forward filler moving to multirepo
jobs but the bundle prefix could be set once per job which prevents
reversefiller from using it. For each mononoke repo the bundles have different
prefix (repo id is in the prefix) that's why we now take a list of bundle ids
together with a list of repos.
NOTE: If I were implementing this functionality from scratch I'd probably teach
the filler to just read the right bundle prefixed from Mononoke configs but
this way of doing it is just faster to pull and it's backwards compatible with
current tw jobs so the rollout will be easy.
Reviewed By: ikostia
Differential Revision: D25188994
fbshipit-source-id: 1c25e52404d5ec45d721f047c8d71b002cdf0994
Summary:
Currently, the healer would clone a blob at least once if it needs to be healed. This is done because `blobstore.put` needs to take ownership of the `blob`. It is pretty wasteful in several different ways:
- if there's just one `put` needed, we can just use the existing `blob` instance and avoid cloning it
- we clone `blob` for every `stores_to_heal` at the same time, therefore consuming `stores_to_heal.len() * blob.len()` bytes of RAM. On the plus side, this allows us to run those `put`s in parallel, but on the flip side - it uses a lot of RAM. In cases when we need to heal a 4Gb blob to two different blobstores, this would immediately jump memory usage from 4Gb to 12Gb. Taking into account that we use 15Gb containers and heal 100 blobs in parallel, this is pretty bad. One way of dealing with this is doing `put`s one-by-one, cloning for each (except the last one). As each `put` succeeds, we drop the original cloned `blob`, therefore freeing some memory. Formulaically this means that our peak usage per blob is `blob.len() * 2` if there's >1 heal needed and just `blob.len()` if there's 1 heal needed.
So let's support two different strategies of healing: `Sequential` and `Concurrent`. We can choose `Concurrent` when we know that there's plenty of RAM to clone our blob enough times. To know what "plenty of RAM" is, let monitor our RSS and allow passing a "ceiling" on the command line (newly added `--max-rss-bytes` arg).
Note 1: Of course, this all is just best effort: there are obvious TOCTTOU problems with RSS checks, so false positives are possible.
Note 2: A much better approach here would be to have `.put` not take ownership of the blob in the first place, and to only clone from a passed reference when needed (I think it's actually not always needed, like `sqlblob` takes ownership, and then creates chunks by cloning it one more time, IIUC). Unfortunately, this is a much larger change with a lot of complexities, so I'd rather not do it here.
Reviewed By: krallin
Differential Revision: D25196372
fbshipit-source-id: 1128c13f68f2dc877cebcf45d8d96954543daf3c
Summary: convert `BlobRepo::get_bonsai_bookmark` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25188577
fbshipit-source-id: fb6f2b592b9e9f76736bc1af5fa5a08d12744b5f
Summary: Let's use fbinit::compat_test, it makes the tests a bit shorter
Reviewed By: ikostia
Differential Revision: D25196759
fbshipit-source-id: bd8aca4b676a71158269195fd35153a0934b0cbc
Summary: Prefix old futures with `Old` so it would be possible to start conversion of BlobRepo to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25187882
fbshipit-source-id: d66debd2981564b289d50292888f3f6bd3343e94
Summary:
The expected walker validation failures can appear in any order. Sort
them so the comparison is deterministic
Reviewed By: StanislavGlebik
Differential Revision: D25195052
fbshipit-source-id: aeded6675c85186e12d27023fb862d7c52dd764f
Summary: editmergeps.bat was separating a filename with spaces into args[0] and args[1] when calling editmergeps.ps1. Use proper quoting to send files with spaces as a single argument.
Reviewed By: ikostia
Differential Revision: D25194324
fbshipit-source-id: 065f677c9015681b310e1cfc46f52ff563a35f99
Summary: This will make it easier to choose how to migrate this code to futures 0.3
Reviewed By: ahornby
Differential Revision: D25185646
fbshipit-source-id: b39f7540275115fff4e8b6f2e740d544c2c876ef
Summary:
While trying to repro a user report (https://fburl.com/jqvm320o), I ran into a
new hg error: P151186623, i.e.:
```
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("fbcode/thrift/facebook/test/py/TARGETS"), hgid: HgId("55713728544d5955703d604299d77bb1ed50c62d") })'
```
After some investigation (and adding a lot of prints), I noticed that this
was trying to query the EdenAPI server for this filenode. That request should
succeed, given Mononoke knows about this filenode:
```
[torozco@devbig051]~/fbcode % mononoke_exec mononoke_admin fbsource --use-mysql-client filenodes by-id fbcode/thrift/facebook/test/py/TARGETS 55713728544d5955703d604299d77bb1ed50c62d
mononoke_exec: Using config stage prod (set MONONOKE_EXEC_STAGE= to override)
I1126 08:10:02.089614 3697890 [main] eden/mononoke/cmdlib/src/args/mod.rs:1097] using repo "fbsource" repoid RepositoryId(2100)
I1126 08:10:02.995172 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:137] Filenode: HgFileNodeId(HgNodeHash(Sha1(55713728544d5955703d604299d77bb1ed50c62d)))
I1126 08:10:02.995282 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:138] -- path: FilePath(MPath("fbcode/thrift/facebook/test/py/TARGETS"))
I1126 08:10:02.995341 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:139] -- p1: Some(HgFileNodeId(HgNodeHash(Sha1(ccb76adc7db0fc4a395be066fe5464873cdf57e7))))
I1126 08:10:02.995405 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:140] -- p2: None
I1126 08:10:02.995449 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:141] -- copyfrom: None
I1126 08:10:02.995486 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:142] -- linknode: HgChangesetId(HgNodeHash(Sha1(dfe46f7d6cd8bc9b03af8870aca521b1801126f0)))
```
Turns out, the success rate for this method is actually 0% — out of thousands of requests,
not a single one succeeded :(
https://fburl.com/scuba/edenapi_server/cma3c3j0
The root cause here is that the server side is not properly deserializing
requests (actually finding that was a problem of its own, I filed T80406893 for this).
If you manage to get it to print the errors, it says:
```
{"message":"Deserialization failed: missing field `path`","request_id":"f97e2c7c-a432-4696-9a4e-538ed0db0418"}
```
The reason for this is that the server side tries to deserialize the request
as if it was a `WireHistoryRequest`, but actually it's a `HistoryRequest`, so
all the fields have different names (we use numbers in `WireHistoryRequest`).
This diff fixes that. I also introduced a helper method to make this a little
less footgun-y and double-checked the other callsites. There is one callsite
right now that looks like it might be broken (the commit one), but I couldn't
find the client side interface for this (I'm guessing it's not implemented
yet), so for now I left it as-is.
Reviewed By: StanislavGlebik
Differential Revision: D25187639
fbshipit-source-id: fa993579666dda762c0d71ccb56a646c20aee606
Summary:
In the next diff I'd like to allow hg sync job to combine a few bookmark update log entries and send a single bundle for all of them. The goal is to reduce the lag between mononoke and hg servers.
We've already made an attempt to implement bundle combining some time ago, hence why we have things like CombinedBookmarkUpdateLogEntry. However it was never really used in prod - back then the only way to sync a bundle from mononoke to hg servers was to replay a bundle that client has pushed to us, and combining bundles like that was tricky.
Now it's different, because hg sync job has the logic to generates the bundles itself rather than re-use the bundle that client has pushed to us. This makes implementing bundle combinging easier.
This diff doesn't add the actual bundle combining, but it does the refactoring that makes it easier. In particular:
1) Old logic for combining bundles was removed - it was never really used anyway.
1) prepare_bundles() function was added - this function takes a vector of BookmarkUpdateLogEntry and returns a vector of CombinedBookmarkUpdateLogEntry. The idea is to move bundle combining logic from main function to BundlePreparer class, since it has better knowledge of how to do bundle combining (in particular, it knows whether sync job re-uses existing bundle or generates it)
1) prepare_single_bundle() was changed - instead of taking bookmark name, from/to changeset id from BookmarkUpdateLogEntry, it now requires passing them explicitly. This makes adding bundle combining easier in the next diff.
Reviewed By: mitrandir77
Differential Revision: D25168877
fbshipit-source-id: 2935d1795925b4bf0456b9221e2f76ce3987cbd0
Summary: Only the id is used when loading the manifest, not the path. Path is only needed for route information
Reviewed By: mitrandir77
Differential Revision: D25185264
fbshipit-source-id: 1af453fa2716d53ea4fb18a81d59867e98ea07f6
Summary: The path is not needed to load the fsnode, only as route info for things like compression
Reviewed By: mitrandir77
Differential Revision: D25184485
fbshipit-source-id: afd84dcd9dbd82c2c9018e86fb676bc57a5d009b
Summary:
We have `HgBlobChangeset`, `HgFileEnvelope`, `HgManifestEnvelope` ... but we
also have `BlobManifest`. Let's be a little consistent.
Reviewed By: markbt
Differential Revision: D25122288
fbshipit-source-id: 9ae0be49986dbcc31cee9a46bd30093b07795c62
Summary: Add them to the walker so we can scrub and inspect them
Reviewed By: StanislavGlebik
Differential Revision: D25124144
fbshipit-source-id: 5df1ca6e48d18d3d55f68905e93fd75fbae92adb
Summary:
We have 4 different ways of awaiting futures in there: sometimes we create a
runtime, sometimes we use async-unit, sometimes we use `fbinit::test` and
sometimes we use `fbinit::compat_test`. Let's be consistent.
While in there, let's also get rid of `should_panic`, since that's not a very
good way of running tests.
Reviewed By: HarveyHunt
Differential Revision: D25186195
fbshipit-source-id: b64bb61935fb2132d2e5d8dff66fd4efdae1bf64
Summary:
HgBlobEntry is kind of a problematic type right now:
- It has no typing to indicate if it's a file or a manifest
- It always has a file type, but we only sometimes care about it
This results in us using `HgBlobEntry` to sometimes represent
`Entry<HgManifestId, (FileType, HgFileNodeId)>`, and some other times to
represent `Entry<HgManifestId, HgFileNodeId>`.
This makes code a) confusing, b) harder to refactor because you might be having
to change unrelated code that cares for the use case you don't care about (i.e.
with or without the FileType), and c) results in us sometimes just throwing in
a `FileType::Normal` because we don't care about the type, which is prone to
breaking stuff.
So, this diff just removes it, and replaces it with the appropriate types.
Reviewed By: farnz
Differential Revision: D25122291
fbshipit-source-id: e9c060c509357321a8059d95daf22399177140f1
Summary:
I'd like to get rid of HgEntry altogether, as it's not a very useful
abstraction (`Entry` is better). This is one step towards this, by moving the
HgEntry -> Leaf conversion close to where the `HgEntry` is created (which is
where we upload filenodes).
Reviewed By: markbt
Differential Revision: D25122290
fbshipit-source-id: 0e3049392791153b9547091516e702fb04ad7094
Summary:
I'd like to remove the number of places that use HgEntryId, and make its usage
a little more consistent. Indeed, right now, HgEntryId is used in some places
to upload things where we just give it a fake entry type, and sometimes to
represent an actual entry in a manifest; This isn't great.
Let's remove it from here.
Reviewed By: markbt
Differential Revision: D25122289
fbshipit-source-id: 5075383e037e4e890af203d133f0a25118c19823
Summary:
This contains a path, which it turns out is actually never used throughout the
codebase. Remove it.
Reviewed By: markbt
Differential Revision: D25122292
fbshipit-source-id: eab528bfb1e3893bca4d92d62c30499cf9eead6c
Summary: So we can scrub and inspect them
Reviewed By: StanislavGlebik
Differential Revision: D25120995
fbshipit-source-id: d150e55f0d72f584c15dbaf2bd017d19130312b2
Summary: This diff asyncifies build_reporting_handler, and while there also simplifies this function a bit by ignoring cases where log_entries is empty or not specified
Reviewed By: farnz
Differential Revision: D25184396
fbshipit-source-id: 46b5d2f9fb5571e502bcdf5a0fe964fb62426313
Summary:
This diff asyncifies SYNC_LOOP similar to how SYNC_ONCE was asyncified.
The biggest part of SYNC_LOOP is a stream that starts with loop_over_log_entries. Previously it was a very long chain of combinators. This diff makes this chain ~2 times smaller, but unfortunately it can't make it even smaller because of the use of "buffered(...)" method.
Reviewed By: ahornby
Differential Revision: D25123487
fbshipit-source-id: e913bbd1369e4375b5b1d0e6ba462e463a5a44fc
Summary:
The write & read path use different back up state files, which means they can
go out of sync. If that happens, it's a bit annoying because:
- The output of `hg cloud check` is out of sync with that of `hg sl`
- `hg cloud backup -r` says the commit is backedup, and `hg cloud check -r`
says it isn't.
This diff fixes this by just using the `backedup()` revset, which under the
hood reads all state files.
Reviewed By: liubov-dmitrieva
Differential Revision: D25186071
fbshipit-source-id: 0ae2d43227d65a3564b2f796218b55982d5cc467
Summary:
Right now we just get a "deadline exceeded" error, which isn't very amenable to
helping us understand why we timed out. Let's add more logging. Notably,
I'd like to understnad what we've actually received at this point, if anything,
and how long we waited, as I'm starting to suspect this issue doesn't have much
to do with HTTP.
See https://fb.workplace.com/groups/scm/permalink/3361972140519049/ for
more context.
Reviewed By: quark-zju
Differential Revision: D25128159
fbshipit-source-id: b45d3415526fdf21aa80b7eeed98ee9206fbfd12
Summary:
`eden du --clean` currently fails with
```
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
```
Full error: P149352812
It looks like this is because Buck expects to run with a different python, so
here I clear out the PYTHONHOME variable before we start buck.
This reuses out logic used elsewhere to clean up the environment before
calling buck.
Reviewed By: wez
Differential Revision: D24904105
fbshipit-source-id: 73587c52aff3ea00f68339eb44e7042329de2e44
Summary: `lru-disk-cache` depends on an old version of `linked-hash-map` which contains UB in 1.48 (see https://github.com/mozilla/sccache/issues/813). They updated the deps in their repo months ago, but haven't pushed a new version. This diff makes us get `lru-disk-cache` directly from their GitHub instead.
Reviewed By: dtolnay
Differential Revision: D25134582
fbshipit-source-id: 05fd63a76b7095ebeea458645b92a83bbd8c4614
Summary:
It was mistakenly mapping deleted manifest id to a unode id.
I'm surprised the walk.rs code that constructs the DeletedManifestToDeletedManifestChild edge built with this bug. The DeletedManifestId and ManifestUnodeId types presumably coerce to each other.
Reviewed By: markbt
Differential Revision: D25120994
fbshipit-source-id: 1b53037808779c345c163ef32324961938078fc7
Summary:
Soon we are going to use hg sync job for configerator repos, and they might use
Push bookmark move. Let's allow it in sync job
Reviewed By: ikostia
Differential Revision: D25121176
fbshipit-source-id: f6000617b42be8392730ffc56be1947fbdbfff76
Summary:
This adds support for optionally not uploading commits we already have when
they arrive via infinitepush. This can happen if we're replaying bundles.
This works by filtering the commits we have. We still get some futures created
for the underlying uploads, but we never poll them because when we visit
what futures are needed for what commits, we don't select uploads that are
only reachable from commits we filtered out.
Obviously, this isn't super efficient, since the client still has to send us
all this data, but it's better than having them send us all that data then
having us take hours overwriting it all.
Reviewed By: mitrandir77
Differential Revision: D25120844
fbshipit-source-id: f96881d0d98ec622259bedb624fd94b600a2bf1d
Summary: Remove 'static requirement for async methods of Blobstore, propagate this change and fixup low hanging fruits where the code can become 'static free easily.
Reviewed By: ahornby, farnz
Differential Revision: D24839054
fbshipit-source-id: 5d5daa04c23c4c9ae902b669b0a71fe41ee6dee6
Summary:
It feels like invalidating the entry before the directory makes slightly more
sense, so do it in that order.
Reviewed By: chadaustin
Differential Revision: D24800817
fbshipit-source-id: ed053d07bbae6954c276d1ad7a1ff247e5c055d9
Summary:
It turns out the hggit tests weren't passing in Python 3, despite us
having removed them from our py3-broken list. Woops. This fixes them and enables
the tests.
Reviewed By: sfilipco
Differential Revision: D25095189
fbshipit-source-id: acffca34b0d5defa7575ede60621ca2ce0a2afe4
Summary:
Back in March we forced all extras to be strings. It turns out hggit
needs to write binary extras since it stores some binary patches in extras.
To support that, let's encode commit extras using surrogateescaped. That will
allow hggit to put binary data in extras in a later diff.
Reviewed By: sfilipco
Differential Revision: D25095190
fbshipit-source-id: 330bf06b15fc435f70119490557b8c22341b6ed9
Summary:
Apparently shards for some DBs start from 1 (production_filenodes) and for some - from 0 (production_blobstore).
This diff fixed the issue for mysql connections.
Long term we might want to query SMC for the list of shards instead of hardcoding different values in the different places.
Reviewed By: farnz
Differential Revision: D25057136
fbshipit-source-id: 9201a2ec8afe0b66a246a2ee91cc9389630f5ddf
Summary:
Add a TraceBus to HgQueuedBackingStore and allow tracing import events over Thrift.
This powers a new `eden trace hg` command that allows a live view of
tree and blob imports.
Reviewed By: genevievehelsel
Differential Revision: D24732059
fbshipit-source-id: 525152fe39047160a68c1706217a06a00a6dbef1
Summary:
We got a few ubns because one bookmark validator in a single region wasn't able
to connect to mysql and was reporting errors.
This diff fixes by separating logical and infra errors
Reviewed By: ikostia
Differential Revision: D25092364
fbshipit-source-id: 93f4be1a7e0467051b7b8d927eef9b4f5cd6a983
Summary:
This isn't actually being consulted anywhere save for a single test, so let's
just remove it (it's not like the test checks anything important — that field
might not as well exist given we never read it).
Reviewed By: farnz
Differential Revision: D25093494
fbshipit-source-id: 5f4a53f8666fc0e8a89ceade44baa96e71fb813f
Summary:
This is a bit unnecessary as it stands — we roundtrip the path through
execute() just to return it back. This path was used for trace logging, but
given we literally never look at this log, let's just simplify this logging a
little bit.
Reviewed By: StanislavGlebik
Differential Revision: D25089344
fbshipit-source-id: 15b0f1cce8c9b2938429de19ff063e5677794912
Summary:
With segmented changelog, rev can exceed f64 safe range
(Number.MAX_SAFE_INTEGER in Javascript, 9007199254740991, 0x1fffffffffffff).
If rev is used in JSON, then the JSON cannot be parsed with precise rev
information.
This diff adds a compatibility mode so template will map the out-of-range revs
to safe range, and the mapped revs can be detected, and mapped back to be
resolved correctly.
Reviewed By: sfilipco
Differential Revision: D25079001
fbshipit-source-id: 52a5a2c8345a76355b8bf81435034d90c7c31610
Summary: Tests the behaviour of collecting the raw bundles.
Reviewed By: krallin
Differential Revision: D25025255
fbshipit-source-id: 114da273a28d131f5dd24047ed28ea23d076f235
Summary:
The `getdesignatednodes` function returns a boolean indicating whether the requested nodes were actually fetched.
In the case of SSH, this is needed because the server may not support the `designatednodes` capability (only supported by Mononoke). If the fetch fails, Mercurial will fall back to an expensive complete tree fetch.
The HTTP version of this function accidentally omitted `return True`, which meant it implicitly returned `None`, which triggered the fallback path.
Reviewed By: sfilipco
Differential Revision: D25074067
fbshipit-source-id: 089d5382dd566db89ee732cdcb82762c8d43e21a
Summary: The test doesn't compile on Windows, let's just ifdef it.
Reviewed By: genevievehelsel
Differential Revision: D25033804
fbshipit-source-id: 4f312f010f9d0db42cc9ae19df3f668e8e1c4665
Summary:
Converting back and forth between folly::fs::path and AbsolutePath appears to
be problematic on Windows as NUL bytes appears in the paths, causing the tests
to fail. Instead of doing this conversion, let's simply use AbsolutePath everywhere.
Reviewed By: chadaustin
Differential Revision: D25033803
fbshipit-source-id: 6c45c2a20fc4bf18cecc838b219faacfeb8386d8
Summary:
Querying bookmarks in all repos at the exact same time results in us making a
bunch of concurrent queries to MySQL, which in turn results in MyRouter
allocating a bunch of connections.
This is for reporting the age of bookmarks: that stuff can wait... It doesn't
need to be super fast. Let's make it run on repos one at a time to avoid
allocating dozens of connections every 10 seconds (which is more often than
MyRouter seems to close them).
Reviewed By: ikostia
Differential Revision: D25057432
fbshipit-source-id: 8b65ef65752fc9762a26d835ac80f61573003dd7
Summary:
Skip the need to go through SSH/Mercurial servers. Directly connect to Mononoke
via TLS with X509 certificates.
Reviewed By: markbt
Differential Revision: D24756947
fbshipit-source-id: a782c894956d6a6be055eaa78287c040ceb57da7
Summary: Add a test case for walking hg data from non-public changeset
Reviewed By: ikostia
Differential Revision: D25023130
fbshipit-source-id: 34295f77926b32c77095f7c10d6daa8ef59d9550
Summary: Add command line option to exclude derived data types from blobimport so we can use it to create non-public commits without filenode data for tests
Differential Revision: D25023116
fbshipit-source-id: d8e5d6955f11cebec0de2075c22981bf6c6f4af3
Summary: Add DeletedManifest to walker types with edges to linked changeset and its sub-manifests so that they can be scrubbed.
Differential Revision: D25000319
fbshipit-source-id: f146e6132fde0fb13e630d315484cc2c0a964bdc
Summary: Add the mapping from bonsai to deleted manifest to the walker
Reviewed By: ikostia
Differential Revision: D24989424
fbshipit-source-id: 53d622f661629b9b3de91910f4560b641a95a7bf
Summary: Sometimes pretty debug format is too verbose and one per line regular debug format is preferrable.
Reviewed By: ikostia
Differential Revision: D24996432
fbshipit-source-id: 1acda3985658e4c17b57e36734c77b7579e7e28a
Summary: add an output option to walker scrub so we can dump out debug representation of any type the walker can walk
Reviewed By: farnz
Differential Revision: D24996433
fbshipit-source-id: a332d89d65e4d928159155a34bd39b0e2e1131de
Summary: Add gathering of statistics to the three main futures in create_changeset.
Reviewed By: StanislavGlebik
Differential Revision: D25022231
fbshipit-source-id: 26c7cd4a05483e694bdff24229e61a63249f98b5
Summary:
Add a trait that lets us log data from the response of a thrift request.
Use this to log the commit that was created by `repo_create_commit`.
Reviewed By: StanislavGlebik
Differential Revision: D25022232
fbshipit-source-id: c6526b29b1d2072bf7d4c46d80cb1a5bf522d227
Summary:
We don't currently log any information about RepoCreateCommit parameters.
Start logging the parents, date, author and number of changes and deletes.
Reviewed By: farnz
Differential Revision: D25021423
fbshipit-source-id: 2723c208643e074861732a21e149c06ad47879f2
Summary:
Move the check for commits not having case conflicts from upload time to when
the commit is being landed to a public bookmark.
This allows draft commits in commit cloud to contain erroneously-introduced
case conflicts, whilst still protecting public commits from these case conflicts.
Note that the checks we are moving are the checks for whether this commit
contains case conflicts internally (two files added by this commit conflict in
case), or against the content of ancestor commits (the commit adds a file which
conflicts with an existing file). The check for whether this commit and the
commits it is being pushrebased over conflict still happens during pushrebase.
This behaviour is controlled by a pair of tunable flags:
* `check_case_conflicts_on_bookmark_movement` enables the new check at land time
* `skip_case_conflict_check_on_changeset_upload` disables the old check at upload-time
The `check_case_conflicts_on_bookmark_movement` should always be set if the
`skip_case_conflict_check_on_changeset_upload` is set, otherwise users will
be able to land case conflicts. Setting `check_case_conflicts_on_bookmark_movement`
without setting `skip_case_conflict_check_on_changeset_upload` allows both
checks to be enabled if that is necessary.
To handle bookmark-only moves, the `run_hooks_on_additional_changesets` tunable
flag must also be enabled.
Reviewed By: ahornby
Differential Revision: D24990174
fbshipit-source-id: 34e40e389f2c2139ba24ecee75473c362f365864
Summary:
Add a method to `SkeletonManifest` which finds the paths to the first case
conflict, if there is one.
First means lexicographically first within directories, and with the shortest path.
Reviewed By: ahornby, StanislavGlebik
Differential Revision: D24990175
fbshipit-source-id: ec10f66582b81c40740823e32362ca489a6ebb4d
Summary:
Refactor loading of additional changesets so that it is cached. This will
allow us to access them multiple times but only load them once.
Differential Revision: D24990176
fbshipit-source-id: c21cd1a811ede8fe2c2b03444de0f071ecf5a38c
Summary:
We used to produce a confusing error message during glob evaluation
when . or .. was specified as a glob component. Instead, fail early,
with an error message that more directly explains the problem.
Reviewed By: genevievehelsel
Differential Revision: D24969096
fbshipit-source-id: fe70a8f4db1fdce8eec13890d20913b63a516518
Summary:
Be more specific about which PathComponent string failed to validate
in order to help diagnose downstream issues like glob syntax errors.
Reviewed By: genevievehelsel
Differential Revision: D24966004
fbshipit-source-id: cd3bc0aeaeb389caa13c86b91149d48c5afdb306
Summary:
Previous code used flat_map on the result which discarded the errors.
Fix it and add a test
Reviewed By: StanislavGlebik
Differential Revision: D24989246
fbshipit-source-id: 075c8e40dceebc480ad722fb467a8a9e9bef5905
Summary:
There seem to be legitimate cases when we want to have both bypass mechanisms available: via the commit message and via the pushvar. The need to be able to use a commit message arises from the fact that engineers rarely have access to push CLI, so they cannot use pushvars. Still, pushvars are useful if we want to push an existing commit without any changes to its message.
NB. Since I wanted to make `HookBypass` a multi-option, I had to change it from `enum` to `struct`. If I left fields `pub` in that struct (the way it's commonly done in `metaconfig`, I would've allowed `HookBypass` instances with both options `None`. This is not a big deal, but is a bit ugly as we already store `Option<HookBypass>` instead, so it's better if `Some(hook_bypass)` symbolizes that there indeed is some way to bypass the hook. So I opted in for constructors, getters and private fields.
Reviewed By: StanislavGlebik
Differential Revision: D25000578
fbshipit-source-id: ad13076c9690ee8d45fc4a93e32a6026ff5bdd09
Summary:
Dechunker has a feature of saving entire dechunked bundle contents in memory
this is used to save the raw bundles to manifold. Unitl now this feature worked
properly when accesed via `Read` trait methods. When accessed via `BufRead`
trait the logic that was collecting the read contents was skipped.
This manifested in the saved infinitepush bundles being always trimmed to 4kb.
Reviewed By: markbt
Differential Revision: D25020371
fbshipit-source-id: c606c9fb116a1cd00ae7f4558a7249364faa9c13
Summary:
Add Blame to walker so that we can scrub it.
For public commits, if blame is requested we wait for blame to be derived before traversing unodes. This saves traversing the unode tree twice.
For non-public commits they are not expected to have blame, so for those we don't wait for blame to be derived before traversing unodes.
Reviewed By: farnz
Differential Revision: D24896243
fbshipit-source-id: 66226a8e47f115bcda62269ade63874e0fff4ba0
Summary:
This change allows the walk steps to check the underlying WalkState's phase information to see if a changeset is public. Its exposed to the steps via Checker::is_public().
This saves repeat checks from the changeset phase from the bonsai_to_hg_mapping_step if the walk state already knows its public, and makes sure we are passing the get_phase ephemeral_derive flag from just one place (it should always be the opposite of enable_derive).
Reviewed By: farnz
Differential Revision: D24954837
fbshipit-source-id: b911d69837db8ef34fbe2c27f642d6819ea46908
Summary:
delegate FromStr, AsRef and sampling_fingerprint from BlameId to the
inner FileUnodeId
Reviewed By: markbt
Differential Revision: D24917140
fbshipit-source-id: 1f39b15c91c1f90f371baf03d97f03d00d8798ea
Summary:
The NodeTypes variants and the EdgeType variants generated from them were getting rather long which makes them hard to work with.
This change shortens them by
- changing BonsaiChangeset* to Changeset (this also has better consistency with changeset its blobstore keys)
- updating Bonsai*Mapping to *Mapping )except for BonsaiHgMapping where it disambiguates with HgBonsaiMapping)
Due to the way walker output is sorted it made more sense to do these in one diff rather than two.
Reviewed By: markbt
Differential Revision: D24864061
fbshipit-source-id: dbd395a89be828ac97cf056f03787097d8f1491d
Summary: The test wasn't compiling due to these 3 missing headers, add them.
Reviewed By: genevievehelsel
Differential Revision: D24997597
fbshipit-source-id: 6e3be0763362e41be138c670dc88a63bd9e88024
Summary:
We've seen some hangs with http 2 in lfs. Switching to http 1.1 seems
to fix it. Let's make this configurable so we can tweak this if we see it in
edenapi. For now we continue to default to http 2.
Reviewed By: krallin
Differential Revision: D24901201
fbshipit-source-id: 9806e2c37fa299e4bd381ebdcb17d00800408de3
Summary:
To support the ability for Mercurial to talk to Mononoke directly while going
through an HTTP proxy, we need to be able to accept HTTP prefixed requests.
This adds the support for Mononoke side. Mercurial client will be added in
subsequent diff.
Reviewed By: krallin
Differential Revision: D24989335
fbshipit-source-id: 597eaa974cea661332967e34abc80b2b609b94ff
Summary:
This is a preparatory phase to make the refcount usuable on Windows. For more
details, see D24716801 (e50725d2cb)
Reviewed By: chadaustin
Differential Revision: D24764568
fbshipit-source-id: 1e8c6ab00d4c1ec79c347fd5ae7167b2ce1dff68
Summary:
The skip_on_mode_mac argument to cpp_unittest doesn't exist on Windows, and to
be consistent with the rest of the code, we can simply ifdef the code that
either doesn't compile, or doesn't run.
Note: For some reason the accessIncrementsAccessCount test fails when running
on Windows, but only if running after accessAddsProcessToProcessNameCache...
Reviewed By: chadaustin
Differential Revision: D24496450
fbshipit-source-id: fe18fe1d791a27fbe4bd03bd3e8c811feeb23f5f
Summary:
This is what `eden debug` shows:
modified (m, a, t, e, r, i, a, l, i, z, e, d)
Enumerate all potentially-modified inode paths
Reviewed By: chadaustin
Differential Revision: D24908570
fbshipit-source-id: 62e91b6f0212c70de4bb1705539d3674e6e000d9
Summary:
On Windows, `eden debug file_stats` would crash due to the fileSize not being
filled. Since computing the file size is just a matter of calling stat(2) on
the file, let's simply do that.
Reviewed By: chadaustin
Differential Revision: D24908430
fbshipit-source-id: 07ffb97ada15a07565bea397b436fd21d09b5565
