Summary: Basic observability for how the segmeted changelog update process is performing.
Reviewed By: krallin
Differential Revision: D25108739
fbshipit-source-id: b1f406eb0c862464b186f933d126e0f3a08144e4
Summary:
The update of the segmented changelog is light weight enough that we can
consider all repositories sharing a common tailer process. With all
repositories sharing a single tailer the the maintenance burden will be lower.
Things that I am particularly unsure about are: tailer configuration setup and
tailer structure. With regards to setup, I am not sure if this is more or less
than what production servers do to instantiate. With regards to structure, I
think that it makes a lot of sense to have a function that takes a single repo
name as parameter but the configuration setup has an influence on the details.
I am also unsure how important it is to paralelize the instantiation of the
blobrepos.
Finally, it is worth mentioning that the SegmentedChangelogTailer waits for
`delay` after an update finishes rather than on a period. The benefit is that
we don't have large updates taking down a process because we schedule the same
large repo update too many timer. The drawback is that scheduling gets messed
up over time and multiple repo updates can end up starting at the same time.
Reviewed By: farnz
Differential Revision: D25100839
fbshipit-source-id: 5fff9f87ba4dc44a17c4a7aaa715d0698b04f5c3
Summary:
Stop implementing the legacy glob() Thrift function, since it's
deprecated, and just noise at this point.
Reviewed By: xavierd
Differential Revision: D25247641
fbshipit-source-id: a022fee169ad54c886d8f300b57bef233453fe8b
Summary: This makes it easier to enable all derived data for scrubbing
Reviewed By: ikostia
Differential Revision: D25188963
fbshipit-source-id: e9c981e33273d6b2eeadcce0d0a341b33e91e42d
Summary:
Show the options for specifying node and edge types in the --help output
These changes removed the last use of lazy_static in the walker, so updated TARGETS as well.
Reviewed By: krallin
Differential Revision: D25188964
fbshipit-source-id: c5ccb4f5a0f3be1b8cb7d51cd5f99236d60d3029
Summary:
It has a build() method and later in stack it will build a mononoke
specific type rather than the clap::App
Differential Revision: D25216827
fbshipit-source-id: 24a531856405a702e7fecf54d60be1ea3d2aa6e7
Summary:
Mercurial is incompatible with TSAN at the moment, due to Rust/C++
compilation issues, Python multiprocess, and Tokio false positives. So
disable our HG tests when running under TSAN.
Reviewed By: fanzeyi
Differential Revision: D25109413
fbshipit-source-id: 86e51ebd287e10f92d6e3b8e7bf191e7946c709a
Summary: Adding `full_idmap_clone` to edenapi and usign that in `debugsegmentclone`.
Reviewed By: quark-zju
Differential Revision: D25139730
fbshipit-source-id: 682055f7c30a94a941acd16e2b8e61b9ea1d0aef
Summary:
This method reconstructs a dag from clone data.
At the moment we only have a clone data construction method in Mononoke. It's
the Dags job to construct and import the clone_data. We'll consolidate that at
a later time.
Reviewed By: quark-zju
Differential Revision: D24954823
fbshipit-source-id: fe92179ec80f71234fc8f1cf7709f5104aabb4fb
Summary:
The server expects post requests. At this time we don't want to cache this data
so POST.
Reviewed By: singhsrb
Differential Revision: D24954824
fbshipit-source-id: 433672189ad97100eee7679894a894ab1d8cff7b
Summary:
The config is something that makes sense for all commands to have access to.
Commands that don't use a repo don't have access to the config that is prepared
by the dispatches. This change is a stop-gap to allow new commands that don't
require a repository to receive the config as an argument.
The construction of the config is something that we should iterate on. I see
the current implementaiton as a workaround.
Reviewed By: quark-zju
Differential Revision: D24954822
fbshipit-source-id: 42254bb201ba8838e7cc107394e8fab53a1a95c7
Summary:
At the moment if we try to sync a bookmark entry but from_cs_id of bookmark
entry doesn't match the value of the bookmark on hg servers then the sync will
fail.
Let's add an option that in the case of this mismatch sets from_cs_id to the
current value on hg servers.
Reviewed By: krallin
Differential Revision: D25242172
fbshipit-source-id: 91180fb86f25d10c9ba2b78d7aa18ed0a52d13a5
Summary: convert mercurial_derived_data to new type futures
Reviewed By: ahornby
Differential Revision: D25220329
fbshipit-source-id: c2532a12e915b315fe6eb72f122dbc37822bbb2a
Summary:
This diff prepares the Mononoke codebase for composition-based extendability of
`ScubaSampleBuilder`. Specifically, in the near future I will add:
- new methods for verbose scuba logging
- new data field (`ObservabilityContext`) to check if verbose logging should
be enabled or disabled
The higher-level goal here is to be able to enable/disable verbose Scuba
logging (either overall or for certain slices of logs, like for a certain
session id) in real time, without restarting Mononoke. To do so, I plan to
expose the aforementioned verbose logging methods, which will run a check
against the stored `ObservabilityContext` and make a decision of whether the
logging is enabled or not. `ObservabilityContext` will of course hide
implementation details from the renamed `ScubaSampleBuilderExt`, and just provide a yes/no
answer based on the current config and sample fields.
At the moment this should be a completely harmless change.
Reviewed By: krallin
Differential Revision: D25211089
fbshipit-source-id: ea03dda82fadb7fc91a2433e12e220582ede5fb8
Summary:
cross repo bookmark validation alarm fired a few times, and looks like it fired
because of the following:
1) find_bookmark_diff compared boomarks and found an inconsistency for bookmark
BM which points to commit A in large repo. Next step is to check bookmark history
2) While find_bookmark_diff was running a new commit B was landed in a large repo
and was backsynced to the small repo, so BM now points to commit B.
3) check_large_bookmark_history is called and it fetches latest bookmark log entries, and
it gets entries for commit A and commit B. check_large_bookmark_history checks
if any of the fetched entries points to a commit in the small repo and if yes then
it also checks if this bookmark update happened not so long ago. And the
problem is in the way it checks the "not so long ago" part. It does so by
finding the time difference between latest bookmark update log entry and any
other bookmark update log entry.
Now, if time difference between these two log entries (for commit B and for
commit A) is more than max_delay_secs (which happens only
if commit rate is low e.g. during the weekends), then the alarm would fire
because the delay between latest bookmark update log entry (the one that moved
BM to commit B) and previous log entry (the one that moved BM to commit A) is too large.
This diff fixes this race by skipping newest entries until we found a bookmark
update log entry which points to the large commit that find_bookmark_diff
returned.
Reviewed By: ikostia
Differential Revision: D25196760
fbshipit-source-id: dfa0dca0001b1c38759ec9f4f790cfa3197ae2cf
Summary: remove dependency on old futures from derived data filenodes
Reviewed By: ahornby
Differential Revision: D25218521
fbshipit-source-id: 4d7eaf42c3ba15ea09276a7f3567128d5216e814
Summary: `derived_data:derived_data` had already been almost converted, I've cleaned up some test so it would be possible to completely remove old futures dependency
Reviewed By: StanislavGlebik
Differential Revision: D25197406
fbshipit-source-id: 064439f42a15f715befc019e5350dda0a2975f2b
Summary:
- convert save_bonsai_changesets to new type futures
- `blobrepo:blobrepo` is free from old futures deps
Reviewed By: StanislavGlebik
Differential Revision: D25197060
fbshipit-source-id: 910bd3f9674094b56e1133d7799cefea56c84123
Summary: Add support for walking fast entries for directories so we can scrub and inspect them
Differential Revision: D25187551
fbshipit-source-id: 812f9fd82459ef49dcd781c318fbe5c398daad21
Summary: Add FastlogFile to walker so it can be inspected and scrubbed. Directory and batch components of fastlog covered in following diffs.
Differential Revision: D25187549
fbshipit-source-id: c046cbf2561cdbbc9563497119e34d1b09d0ebef
Summary:
On Python 3 we only support utf8 files. Python 3 has a way of
representing non-utf8 strings in utf8 format by utilizing surrogateescape, but
these strings cause issues in other parts of the codebase that don't expect
suorrageescaped strings (like the Rust code). Since we simply do not support
these paths, let's filter them out as soon as we get them from Watchman.
Reviewed By: quark-zju
Differential Revision: D25134079
fbshipit-source-id: 8be6893a6114b816097422f4469ac317fa3795d1
Summary: As HG<->Mononoke will go through proxygen, testing showed that Proxygen forces us to use 'Upgrade: websocket' and confirm with the Websocket handshake. Adjust accordingly to do so.
Reviewed By: ahornby
Differential Revision: D25197395
fbshipit-source-id: ca00ac31be92817c6f1a99d7d492469b17b46286
Summary:
This diff adds bundle combining to hg sync job. See motivation for doing that in D25168877 (cebde43d3f).
Main struct here is BookmarkLogEntryBatch. This struct helds a vector of BookmarkUpdateLogEntry that were combined (they are used mostly for logging) and also it contains parameters necessary for producing the bundle, notably from/to changeset ids and bookmarks. This struct has try_append method that decides whether it's possible to combine bundles or not.
Reviewed By: mitrandir77
Differential Revision: D25186110
fbshipit-source-id: 77ce91915f460db73d8a996efe415954eeea2476
Summary:
Use of `RootSkeletonManifest::batch_derive` is wrong, as it doesn't take into
account manifests that are already derived. Just use the normal derivation
method.
Reviewed By: StanislavGlebik
Differential Revision: D25218197
fbshipit-source-id: 60f2faad610d507a9659dc37ba72516431a9c036
Summary: The verify-manifests command can take a while. Add logging to show progress is being made.
Reviewed By: StanislavGlebik
Differential Revision: D25099918
fbshipit-source-id: 2b659f1a6921ed111441b2808ce29998bd2285dd
Summary:
If files are only modified, and not added or deleted, the skeleton manifest
doesn't change. Avoid duplicate writes by skipping the write if the skeleton
manifest we derive has the same ID as one of its parents.
Reviewed By: ahornby
Differential Revision: D25099917
fbshipit-source-id: 62900406711becea88491e706a09c6032109c964
Summary:
Improve backfill_derived_data logging to make it clear when a chunk is started
and when already-derived changesets have been skipped.
Fix the counters to take into account skiped changesets, and make the time
remaining human-readable.
Reviewed By: StanislavGlebik
Differential Revision: D25099919
fbshipit-source-id: b39d4b99ef0e79d253de2aa0c91f93276b561680
Summary:
Add skeleton manifests to `derived-data verify-manifests` and add a
`skeleton-manifests` subcommand that allows listing and traversing of skeleton
manifests.
Since skeleton manfiests don't contain the content id or file type, they are
valid simply if they contain the same set of files as the manifest we are
comparing against.
To allow testing of these items while backfilling is still in progress, add an
`--if-derived` option to these commands that work even if derivation is
disabled, provided the manifest has already been derived.
Reviewed By: ikostia
Differential Revision: D25064785
fbshipit-source-id: a6f923bfc53262a5b2118917f8fdd3e99407e036
Summary:
Avoid clients needing to make a secondary look-up for the commit title or message, which
we already have, by returning it in the blame response if the clients request it.
To do this, we add `format_options` to the blame request parameters, which allows customization
of the returned response. The default value if not specified corresponds to the old
behaviour for backwards compatibility.
Reviewed By: mitrandir77
Differential Revision: D25057231
fbshipit-source-id: 84faad7b2db1f684beb94d9fa799c8cef1a89ce8
Summary: Use these in next diff from walker so we can start to scrub fastlog
Reviewed By: aslpavel
Differential Revision: D25187548
fbshipit-source-id: 4188a0c9bdb51a898817b6f5801fe473b6712385
Summary: The path isn't needed to load the UnodeManifest, it's only needed as route information for compression etc
Reviewed By: aslpavel
Differential Revision: D25187547
fbshipit-source-id: b8199a81c5dae4caceed5d455fa6a9bbc3f037ac
Summary: The path is not needed to load the unode, only for route tracking
Reviewed By: aslpavel
Differential Revision: D25187553
fbshipit-source-id: baf6df3b7f56f5ca5a0ba086097acba4ecd82e75
Summary: Change from the walk_blame boolean to a bitflags struct so can also represent fastlog. Using bitflags crate so as not to grow the node size.
Reviewed By: aslpavel
Differential Revision: D25185267
fbshipit-source-id: b45574cd6c4d048aec321357ae54e2ec404f84fc
Summary: Few things were not on lexical order, clean them up. Makes it easier to read next diff without this mixed in
Reviewed By: aslpavel
Differential Revision: D25188965
fbshipit-source-id: 38b7ea6f0e2c0583e6701550d6db16d85ba1a455
Summary:
Previously, the commitcloud reverse filler would only run for a single
repo. This is fine for some of the high throughput repos (e.g. fbsource).
However, for small repos it doesn't make sense to use a server per repo.
Most of support was already completed with forward filler moving to multirepo
jobs but the bundle prefix could be set once per job which prevents
reversefiller from using it. For each mononoke repo the bundles have different
prefix (repo id is in the prefix) that's why we now take a list of bundle ids
together with a list of repos.
NOTE: If I were implementing this functionality from scratch I'd probably teach
the filler to just read the right bundle prefixed from Mononoke configs but
this way of doing it is just faster to pull and it's backwards compatible with
current tw jobs so the rollout will be easy.
Reviewed By: ikostia
Differential Revision: D25188994
fbshipit-source-id: 1c25e52404d5ec45d721f047c8d71b002cdf0994
Summary:
Currently, the healer would clone a blob at least once if it needs to be healed. This is done because `blobstore.put` needs to take ownership of the `blob`. It is pretty wasteful in several different ways:
- if there's just one `put` needed, we can just use the existing `blob` instance and avoid cloning it
- we clone `blob` for every `stores_to_heal` at the same time, therefore consuming `stores_to_heal.len() * blob.len()` bytes of RAM. On the plus side, this allows us to run those `put`s in parallel, but on the flip side - it uses a lot of RAM. In cases when we need to heal a 4Gb blob to two different blobstores, this would immediately jump memory usage from 4Gb to 12Gb. Taking into account that we use 15Gb containers and heal 100 blobs in parallel, this is pretty bad. One way of dealing with this is doing `put`s one-by-one, cloning for each (except the last one). As each `put` succeeds, we drop the original cloned `blob`, therefore freeing some memory. Formulaically this means that our peak usage per blob is `blob.len() * 2` if there's >1 heal needed and just `blob.len()` if there's 1 heal needed.
So let's support two different strategies of healing: `Sequential` and `Concurrent`. We can choose `Concurrent` when we know that there's plenty of RAM to clone our blob enough times. To know what "plenty of RAM" is, let monitor our RSS and allow passing a "ceiling" on the command line (newly added `--max-rss-bytes` arg).
Note 1: Of course, this all is just best effort: there are obvious TOCTTOU problems with RSS checks, so false positives are possible.
Note 2: A much better approach here would be to have `.put` not take ownership of the blob in the first place, and to only clone from a passed reference when needed (I think it's actually not always needed, like `sqlblob` takes ownership, and then creates chunks by cloning it one more time, IIUC). Unfortunately, this is a much larger change with a lot of complexities, so I'd rather not do it here.
Reviewed By: krallin
Differential Revision: D25196372
fbshipit-source-id: 1128c13f68f2dc877cebcf45d8d96954543daf3c
Summary: convert `BlobRepo::get_bonsai_bookmark` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25188577
fbshipit-source-id: fb6f2b592b9e9f76736bc1af5fa5a08d12744b5f
Summary: Let's use fbinit::compat_test, it makes the tests a bit shorter
Reviewed By: ikostia
Differential Revision: D25196759
fbshipit-source-id: bd8aca4b676a71158269195fd35153a0934b0cbc
Summary: Prefix old futures with `Old` so it would be possible to start conversion of BlobRepo to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25187882
fbshipit-source-id: d66debd2981564b289d50292888f3f6bd3343e94
Summary:
The expected walker validation failures can appear in any order. Sort
them so the comparison is deterministic
Reviewed By: StanislavGlebik
Differential Revision: D25195052
fbshipit-source-id: aeded6675c85186e12d27023fb862d7c52dd764f
Summary: editmergeps.bat was separating a filename with spaces into args[0] and args[1] when calling editmergeps.ps1. Use proper quoting to send files with spaces as a single argument.
Reviewed By: ikostia
Differential Revision: D25194324
fbshipit-source-id: 065f677c9015681b310e1cfc46f52ff563a35f99
Summary: This will make it easier to choose how to migrate this code to futures 0.3
Reviewed By: ahornby
Differential Revision: D25185646
fbshipit-source-id: b39f7540275115fff4e8b6f2e740d544c2c876ef
Summary:
While trying to repro a user report (https://fburl.com/jqvm320o), I ran into a
new hg error: P151186623, i.e.:
```
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("fbcode/thrift/facebook/test/py/TARGETS"), hgid: HgId("55713728544d5955703d604299d77bb1ed50c62d") })'
```
After some investigation (and adding a lot of prints), I noticed that this
was trying to query the EdenAPI server for this filenode. That request should
succeed, given Mononoke knows about this filenode:
```
[torozco@devbig051]~/fbcode % mononoke_exec mononoke_admin fbsource --use-mysql-client filenodes by-id fbcode/thrift/facebook/test/py/TARGETS 55713728544d5955703d604299d77bb1ed50c62d
mononoke_exec: Using config stage prod (set MONONOKE_EXEC_STAGE= to override)
I1126 08:10:02.089614 3697890 [main] eden/mononoke/cmdlib/src/args/mod.rs:1097] using repo "fbsource" repoid RepositoryId(2100)
I1126 08:10:02.995172 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:137] Filenode: HgFileNodeId(HgNodeHash(Sha1(55713728544d5955703d604299d77bb1ed50c62d)))
I1126 08:10:02.995282 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:138] -- path: FilePath(MPath("fbcode/thrift/facebook/test/py/TARGETS"))
I1126 08:10:02.995341 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:139] -- p1: Some(HgFileNodeId(HgNodeHash(Sha1(ccb76adc7db0fc4a395be066fe5464873cdf57e7))))
I1126 08:10:02.995405 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:140] -- p2: None
I1126 08:10:02.995449 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:141] -- copyfrom: None
I1126 08:10:02.995486 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:142] -- linknode: HgChangesetId(HgNodeHash(Sha1(dfe46f7d6cd8bc9b03af8870aca521b1801126f0)))
```
Turns out, the success rate for this method is actually 0% — out of thousands of requests,
not a single one succeeded :(
https://fburl.com/scuba/edenapi_server/cma3c3j0
The root cause here is that the server side is not properly deserializing
requests (actually finding that was a problem of its own, I filed T80406893 for this).
If you manage to get it to print the errors, it says:
```
{"message":"Deserialization failed: missing field `path`","request_id":"f97e2c7c-a432-4696-9a4e-538ed0db0418"}
```
The reason for this is that the server side tries to deserialize the request
as if it was a `WireHistoryRequest`, but actually it's a `HistoryRequest`, so
all the fields have different names (we use numbers in `WireHistoryRequest`).
This diff fixes that. I also introduced a helper method to make this a little
less footgun-y and double-checked the other callsites. There is one callsite
right now that looks like it might be broken (the commit one), but I couldn't
find the client side interface for this (I'm guessing it's not implemented
yet), so for now I left it as-is.
Reviewed By: StanislavGlebik
Differential Revision: D25187639
fbshipit-source-id: fa993579666dda762c0d71ccb56a646c20aee606
Summary:
In the next diff I'd like to allow hg sync job to combine a few bookmark update log entries and send a single bundle for all of them. The goal is to reduce the lag between mononoke and hg servers.
We've already made an attempt to implement bundle combining some time ago, hence why we have things like CombinedBookmarkUpdateLogEntry. However it was never really used in prod - back then the only way to sync a bundle from mononoke to hg servers was to replay a bundle that client has pushed to us, and combining bundles like that was tricky.
Now it's different, because hg sync job has the logic to generates the bundles itself rather than re-use the bundle that client has pushed to us. This makes implementing bundle combinging easier.
This diff doesn't add the actual bundle combining, but it does the refactoring that makes it easier. In particular:
1) Old logic for combining bundles was removed - it was never really used anyway.
1) prepare_bundles() function was added - this function takes a vector of BookmarkUpdateLogEntry and returns a vector of CombinedBookmarkUpdateLogEntry. The idea is to move bundle combining logic from main function to BundlePreparer class, since it has better knowledge of how to do bundle combining (in particular, it knows whether sync job re-uses existing bundle or generates it)
1) prepare_single_bundle() was changed - instead of taking bookmark name, from/to changeset id from BookmarkUpdateLogEntry, it now requires passing them explicitly. This makes adding bundle combining easier in the next diff.
Reviewed By: mitrandir77
Differential Revision: D25168877
fbshipit-source-id: 2935d1795925b4bf0456b9221e2f76ce3987cbd0
Summary: Only the id is used when loading the manifest, not the path. Path is only needed for route information
Reviewed By: mitrandir77
Differential Revision: D25185264
fbshipit-source-id: 1af453fa2716d53ea4fb18a81d59867e98ea07f6
Summary: The path is not needed to load the fsnode, only as route info for things like compression
Reviewed By: mitrandir77
Differential Revision: D25184485
fbshipit-source-id: afd84dcd9dbd82c2c9018e86fb676bc57a5d009b
Summary:
We have `HgBlobChangeset`, `HgFileEnvelope`, `HgManifestEnvelope` ... but we
also have `BlobManifest`. Let's be a little consistent.
Reviewed By: markbt
Differential Revision: D25122288
fbshipit-source-id: 9ae0be49986dbcc31cee9a46bd30093b07795c62
Summary: Add them to the walker so we can scrub and inspect them
Reviewed By: StanislavGlebik
Differential Revision: D25124144
fbshipit-source-id: 5df1ca6e48d18d3d55f68905e93fd75fbae92adb
Summary:
We have 4 different ways of awaiting futures in there: sometimes we create a
runtime, sometimes we use async-unit, sometimes we use `fbinit::test` and
sometimes we use `fbinit::compat_test`. Let's be consistent.
While in there, let's also get rid of `should_panic`, since that's not a very
good way of running tests.
Reviewed By: HarveyHunt
Differential Revision: D25186195
fbshipit-source-id: b64bb61935fb2132d2e5d8dff66fd4efdae1bf64
Summary:
HgBlobEntry is kind of a problematic type right now:
- It has no typing to indicate if it's a file or a manifest
- It always has a file type, but we only sometimes care about it
This results in us using `HgBlobEntry` to sometimes represent
`Entry<HgManifestId, (FileType, HgFileNodeId)>`, and some other times to
represent `Entry<HgManifestId, HgFileNodeId>`.
This makes code a) confusing, b) harder to refactor because you might be having
to change unrelated code that cares for the use case you don't care about (i.e.
with or without the FileType), and c) results in us sometimes just throwing in
a `FileType::Normal` because we don't care about the type, which is prone to
breaking stuff.
So, this diff just removes it, and replaces it with the appropriate types.
Reviewed By: farnz
Differential Revision: D25122291
fbshipit-source-id: e9c060c509357321a8059d95daf22399177140f1
Summary:
I'd like to get rid of HgEntry altogether, as it's not a very useful
abstraction (`Entry` is better). This is one step towards this, by moving the
HgEntry -> Leaf conversion close to where the `HgEntry` is created (which is
where we upload filenodes).
Reviewed By: markbt
Differential Revision: D25122290
fbshipit-source-id: 0e3049392791153b9547091516e702fb04ad7094
Summary:
I'd like to remove the number of places that use HgEntryId, and make its usage
a little more consistent. Indeed, right now, HgEntryId is used in some places
to upload things where we just give it a fake entry type, and sometimes to
represent an actual entry in a manifest; This isn't great.
Let's remove it from here.
Reviewed By: markbt
Differential Revision: D25122289
fbshipit-source-id: 5075383e037e4e890af203d133f0a25118c19823
Summary:
This contains a path, which it turns out is actually never used throughout the
codebase. Remove it.
Reviewed By: markbt
Differential Revision: D25122292
fbshipit-source-id: eab528bfb1e3893bca4d92d62c30499cf9eead6c
Summary: So we can scrub and inspect them
Reviewed By: StanislavGlebik
Differential Revision: D25120995
fbshipit-source-id: d150e55f0d72f584c15dbaf2bd017d19130312b2
Summary: This diff asyncifies build_reporting_handler, and while there also simplifies this function a bit by ignoring cases where log_entries is empty or not specified
Reviewed By: farnz
Differential Revision: D25184396
fbshipit-source-id: 46b5d2f9fb5571e502bcdf5a0fe964fb62426313
Summary:
This diff asyncifies SYNC_LOOP similar to how SYNC_ONCE was asyncified.
The biggest part of SYNC_LOOP is a stream that starts with loop_over_log_entries. Previously it was a very long chain of combinators. This diff makes this chain ~2 times smaller, but unfortunately it can't make it even smaller because of the use of "buffered(...)" method.
Reviewed By: ahornby
Differential Revision: D25123487
fbshipit-source-id: e913bbd1369e4375b5b1d0e6ba462e463a5a44fc
Summary:
The write & read path use different back up state files, which means they can
go out of sync. If that happens, it's a bit annoying because:
- The output of `hg cloud check` is out of sync with that of `hg sl`
- `hg cloud backup -r` says the commit is backedup, and `hg cloud check -r`
says it isn't.
This diff fixes this by just using the `backedup()` revset, which under the
hood reads all state files.
Reviewed By: liubov-dmitrieva
Differential Revision: D25186071
fbshipit-source-id: 0ae2d43227d65a3564b2f796218b55982d5cc467
Summary:
Right now we just get a "deadline exceeded" error, which isn't very amenable to
helping us understand why we timed out. Let's add more logging. Notably,
I'd like to understnad what we've actually received at this point, if anything,
and how long we waited, as I'm starting to suspect this issue doesn't have much
to do with HTTP.
See https://fb.workplace.com/groups/scm/permalink/3361972140519049/ for
more context.
Reviewed By: quark-zju
Differential Revision: D25128159
fbshipit-source-id: b45d3415526fdf21aa80b7eeed98ee9206fbfd12
Summary:
`eden du --clean` currently fails with
```
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
```
Full error: P149352812
It looks like this is because Buck expects to run with a different python, so
here I clear out the PYTHONHOME variable before we start buck.
This reuses out logic used elsewhere to clean up the environment before
calling buck.
Reviewed By: wez
Differential Revision: D24904105
fbshipit-source-id: 73587c52aff3ea00f68339eb44e7042329de2e44
Summary: `lru-disk-cache` depends on an old version of `linked-hash-map` which contains UB in 1.48 (see https://github.com/mozilla/sccache/issues/813). They updated the deps in their repo months ago, but haven't pushed a new version. This diff makes us get `lru-disk-cache` directly from their GitHub instead.
Reviewed By: dtolnay
Differential Revision: D25134582
fbshipit-source-id: 05fd63a76b7095ebeea458645b92a83bbd8c4614
Summary:
It was mistakenly mapping deleted manifest id to a unode id.
I'm surprised the walk.rs code that constructs the DeletedManifestToDeletedManifestChild edge built with this bug. The DeletedManifestId and ManifestUnodeId types presumably coerce to each other.
Reviewed By: markbt
Differential Revision: D25120994
fbshipit-source-id: 1b53037808779c345c163ef32324961938078fc7
Summary:
Soon we are going to use hg sync job for configerator repos, and they might use
Push bookmark move. Let's allow it in sync job
Reviewed By: ikostia
Differential Revision: D25121176
fbshipit-source-id: f6000617b42be8392730ffc56be1947fbdbfff76
Summary:
This adds support for optionally not uploading commits we already have when
they arrive via infinitepush. This can happen if we're replaying bundles.
This works by filtering the commits we have. We still get some futures created
for the underlying uploads, but we never poll them because when we visit
what futures are needed for what commits, we don't select uploads that are
only reachable from commits we filtered out.
Obviously, this isn't super efficient, since the client still has to send us
all this data, but it's better than having them send us all that data then
having us take hours overwriting it all.
Reviewed By: mitrandir77
Differential Revision: D25120844
fbshipit-source-id: f96881d0d98ec622259bedb624fd94b600a2bf1d
Summary: Remove 'static requirement for async methods of Blobstore, propagate this change and fixup low hanging fruits where the code can become 'static free easily.
Reviewed By: ahornby, farnz
Differential Revision: D24839054
fbshipit-source-id: 5d5daa04c23c4c9ae902b669b0a71fe41ee6dee6
Summary:
It feels like invalidating the entry before the directory makes slightly more
sense, so do it in that order.
Reviewed By: chadaustin
Differential Revision: D24800817
fbshipit-source-id: ed053d07bbae6954c276d1ad7a1ff247e5c055d9
Summary:
It turns out the hggit tests weren't passing in Python 3, despite us
having removed them from our py3-broken list. Woops. This fixes them and enables
the tests.
Reviewed By: sfilipco
Differential Revision: D25095189
fbshipit-source-id: acffca34b0d5defa7575ede60621ca2ce0a2afe4
Summary:
Back in March we forced all extras to be strings. It turns out hggit
needs to write binary extras since it stores some binary patches in extras.
To support that, let's encode commit extras using surrogateescaped. That will
allow hggit to put binary data in extras in a later diff.
Reviewed By: sfilipco
Differential Revision: D25095190
fbshipit-source-id: 330bf06b15fc435f70119490557b8c22341b6ed9
Summary:
Apparently shards for some DBs start from 1 (production_filenodes) and for some - from 0 (production_blobstore).
This diff fixed the issue for mysql connections.
Long term we might want to query SMC for the list of shards instead of hardcoding different values in the different places.
Reviewed By: farnz
Differential Revision: D25057136
fbshipit-source-id: 9201a2ec8afe0b66a246a2ee91cc9389630f5ddf
Summary:
Add a TraceBus to HgQueuedBackingStore and allow tracing import events over Thrift.
This powers a new `eden trace hg` command that allows a live view of
tree and blob imports.
Reviewed By: genevievehelsel
Differential Revision: D24732059
fbshipit-source-id: 525152fe39047160a68c1706217a06a00a6dbef1
Summary:
We got a few ubns because one bookmark validator in a single region wasn't able
to connect to mysql and was reporting errors.
This diff fixes by separating logical and infra errors
Reviewed By: ikostia
Differential Revision: D25092364
fbshipit-source-id: 93f4be1a7e0467051b7b8d927eef9b4f5cd6a983
Summary:
This isn't actually being consulted anywhere save for a single test, so let's
just remove it (it's not like the test checks anything important — that field
might not as well exist given we never read it).
Reviewed By: farnz
Differential Revision: D25093494
fbshipit-source-id: 5f4a53f8666fc0e8a89ceade44baa96e71fb813f
