Summary:
We have 4 different ways of awaiting futures in there: sometimes we create a
runtime, sometimes we use async-unit, sometimes we use `fbinit::test` and
sometimes we use `fbinit::compat_test`. Let's be consistent.
While in there, let's also get rid of `should_panic`, since that's not a very
good way of running tests.
Reviewed By: HarveyHunt
Differential Revision: D25186195
fbshipit-source-id: b64bb61935fb2132d2e5d8dff66fd4efdae1bf64
Summary:
HgBlobEntry is kind of a problematic type right now:
- It has no typing to indicate if it's a file or a manifest
- It always has a file type, but we only sometimes care about it
This results in us using `HgBlobEntry` to sometimes represent
`Entry<HgManifestId, (FileType, HgFileNodeId)>`, and some other times to
represent `Entry<HgManifestId, HgFileNodeId>`.
This makes code a) confusing, b) harder to refactor because you might be having
to change unrelated code that cares for the use case you don't care about (i.e.
with or without the FileType), and c) results in us sometimes just throwing in
a `FileType::Normal` because we don't care about the type, which is prone to
breaking stuff.
So, this diff just removes it, and replaces it with the appropriate types.
Reviewed By: farnz
Differential Revision: D25122291
fbshipit-source-id: e9c060c509357321a8059d95daf22399177140f1
Summary:
I'd like to get rid of HgEntry altogether, as it's not a very useful
abstraction (`Entry` is better). This is one step towards this, by moving the
HgEntry -> Leaf conversion close to where the `HgEntry` is created (which is
where we upload filenodes).
Reviewed By: markbt
Differential Revision: D25122290
fbshipit-source-id: 0e3049392791153b9547091516e702fb04ad7094
Summary:
I'd like to remove the number of places that use HgEntryId, and make its usage
a little more consistent. Indeed, right now, HgEntryId is used in some places
to upload things where we just give it a fake entry type, and sometimes to
represent an actual entry in a manifest; This isn't great.
Let's remove it from here.
Reviewed By: markbt
Differential Revision: D25122289
fbshipit-source-id: 5075383e037e4e890af203d133f0a25118c19823
Summary:
This contains a path, which it turns out is actually never used throughout the
codebase. Remove it.
Reviewed By: markbt
Differential Revision: D25122292
fbshipit-source-id: eab528bfb1e3893bca4d92d62c30499cf9eead6c
Summary: So we can scrub and inspect them
Reviewed By: StanislavGlebik
Differential Revision: D25120995
fbshipit-source-id: d150e55f0d72f584c15dbaf2bd017d19130312b2
Summary: This diff asyncifies build_reporting_handler, and while there also simplifies this function a bit by ignoring cases where log_entries is empty or not specified
Reviewed By: farnz
Differential Revision: D25184396
fbshipit-source-id: 46b5d2f9fb5571e502bcdf5a0fe964fb62426313
Summary:
This diff asyncifies SYNC_LOOP similar to how SYNC_ONCE was asyncified.
The biggest part of SYNC_LOOP is a stream that starts with loop_over_log_entries. Previously it was a very long chain of combinators. This diff makes this chain ~2 times smaller, but unfortunately it can't make it even smaller because of the use of "buffered(...)" method.
Reviewed By: ahornby
Differential Revision: D25123487
fbshipit-source-id: e913bbd1369e4375b5b1d0e6ba462e463a5a44fc
Summary:
The write & read path use different back up state files, which means they can
go out of sync. If that happens, it's a bit annoying because:
- The output of `hg cloud check` is out of sync with that of `hg sl`
- `hg cloud backup -r` says the commit is backedup, and `hg cloud check -r`
says it isn't.
This diff fixes this by just using the `backedup()` revset, which under the
hood reads all state files.
Reviewed By: liubov-dmitrieva
Differential Revision: D25186071
fbshipit-source-id: 0ae2d43227d65a3564b2f796218b55982d5cc467
Summary:
Right now we just get a "deadline exceeded" error, which isn't very amenable to
helping us understand why we timed out. Let's add more logging. Notably,
I'd like to understnad what we've actually received at this point, if anything,
and how long we waited, as I'm starting to suspect this issue doesn't have much
to do with HTTP.
See https://fb.workplace.com/groups/scm/permalink/3361972140519049/ for
more context.
Reviewed By: quark-zju
Differential Revision: D25128159
fbshipit-source-id: b45d3415526fdf21aa80b7eeed98ee9206fbfd12
Summary:
`eden du --clean` currently fails with
```
Fatal Python error: initfsencoding: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
```
Full error: P149352812
It looks like this is because Buck expects to run with a different python, so
here I clear out the PYTHONHOME variable before we start buck.
This reuses out logic used elsewhere to clean up the environment before
calling buck.
Reviewed By: wez
Differential Revision: D24904105
fbshipit-source-id: 73587c52aff3ea00f68339eb44e7042329de2e44
Summary: `lru-disk-cache` depends on an old version of `linked-hash-map` which contains UB in 1.48 (see https://github.com/mozilla/sccache/issues/813). They updated the deps in their repo months ago, but haven't pushed a new version. This diff makes us get `lru-disk-cache` directly from their GitHub instead.
Reviewed By: dtolnay
Differential Revision: D25134582
fbshipit-source-id: 05fd63a76b7095ebeea458645b92a83bbd8c4614
Summary:
It was mistakenly mapping deleted manifest id to a unode id.
I'm surprised the walk.rs code that constructs the DeletedManifestToDeletedManifestChild edge built with this bug. The DeletedManifestId and ManifestUnodeId types presumably coerce to each other.
Reviewed By: markbt
Differential Revision: D25120994
fbshipit-source-id: 1b53037808779c345c163ef32324961938078fc7
Summary:
Soon we are going to use hg sync job for configerator repos, and they might use
Push bookmark move. Let's allow it in sync job
Reviewed By: ikostia
Differential Revision: D25121176
fbshipit-source-id: f6000617b42be8392730ffc56be1947fbdbfff76
Summary:
This adds support for optionally not uploading commits we already have when
they arrive via infinitepush. This can happen if we're replaying bundles.
This works by filtering the commits we have. We still get some futures created
for the underlying uploads, but we never poll them because when we visit
what futures are needed for what commits, we don't select uploads that are
only reachable from commits we filtered out.
Obviously, this isn't super efficient, since the client still has to send us
all this data, but it's better than having them send us all that data then
having us take hours overwriting it all.
Reviewed By: mitrandir77
Differential Revision: D25120844
fbshipit-source-id: f96881d0d98ec622259bedb624fd94b600a2bf1d
Summary: Remove 'static requirement for async methods of Blobstore, propagate this change and fixup low hanging fruits where the code can become 'static free easily.
Reviewed By: ahornby, farnz
Differential Revision: D24839054
fbshipit-source-id: 5d5daa04c23c4c9ae902b669b0a71fe41ee6dee6
Summary:
It feels like invalidating the entry before the directory makes slightly more
sense, so do it in that order.
Reviewed By: chadaustin
Differential Revision: D24800817
fbshipit-source-id: ed053d07bbae6954c276d1ad7a1ff247e5c055d9
Summary:
It turns out the hggit tests weren't passing in Python 3, despite us
having removed them from our py3-broken list. Woops. This fixes them and enables
the tests.
Reviewed By: sfilipco
Differential Revision: D25095189
fbshipit-source-id: acffca34b0d5defa7575ede60621ca2ce0a2afe4
Summary:
Back in March we forced all extras to be strings. It turns out hggit
needs to write binary extras since it stores some binary patches in extras.
To support that, let's encode commit extras using surrogateescaped. That will
allow hggit to put binary data in extras in a later diff.
Reviewed By: sfilipco
Differential Revision: D25095190
fbshipit-source-id: 330bf06b15fc435f70119490557b8c22341b6ed9
Summary:
Apparently shards for some DBs start from 1 (production_filenodes) and for some - from 0 (production_blobstore).
This diff fixed the issue for mysql connections.
Long term we might want to query SMC for the list of shards instead of hardcoding different values in the different places.
Reviewed By: farnz
Differential Revision: D25057136
fbshipit-source-id: 9201a2ec8afe0b66a246a2ee91cc9389630f5ddf
Summary:
Add a TraceBus to HgQueuedBackingStore and allow tracing import events over Thrift.
This powers a new `eden trace hg` command that allows a live view of
tree and blob imports.
Reviewed By: genevievehelsel
Differential Revision: D24732059
fbshipit-source-id: 525152fe39047160a68c1706217a06a00a6dbef1
Summary:
We got a few ubns because one bookmark validator in a single region wasn't able
to connect to mysql and was reporting errors.
This diff fixes by separating logical and infra errors
Reviewed By: ikostia
Differential Revision: D25092364
fbshipit-source-id: 93f4be1a7e0467051b7b8d927eef9b4f5cd6a983
Summary:
This isn't actually being consulted anywhere save for a single test, so let's
just remove it (it's not like the test checks anything important — that field
might not as well exist given we never read it).
Reviewed By: farnz
Differential Revision: D25093494
fbshipit-source-id: 5f4a53f8666fc0e8a89ceade44baa96e71fb813f
Summary:
This is a bit unnecessary as it stands — we roundtrip the path through
execute() just to return it back. This path was used for trace logging, but
given we literally never look at this log, let's just simplify this logging a
little bit.
Reviewed By: StanislavGlebik
Differential Revision: D25089344
fbshipit-source-id: 15b0f1cce8c9b2938429de19ff063e5677794912
Summary:
With segmented changelog, rev can exceed f64 safe range
(Number.MAX_SAFE_INTEGER in Javascript, 9007199254740991, 0x1fffffffffffff).
If rev is used in JSON, then the JSON cannot be parsed with precise rev
information.
This diff adds a compatibility mode so template will map the out-of-range revs
to safe range, and the mapped revs can be detected, and mapped back to be
resolved correctly.
Reviewed By: sfilipco
Differential Revision: D25079001
fbshipit-source-id: 52a5a2c8345a76355b8bf81435034d90c7c31610
Summary: Tests the behaviour of collecting the raw bundles.
Reviewed By: krallin
Differential Revision: D25025255
fbshipit-source-id: 114da273a28d131f5dd24047ed28ea23d076f235
Summary:
The `getdesignatednodes` function returns a boolean indicating whether the requested nodes were actually fetched.
In the case of SSH, this is needed because the server may not support the `designatednodes` capability (only supported by Mononoke). If the fetch fails, Mercurial will fall back to an expensive complete tree fetch.
The HTTP version of this function accidentally omitted `return True`, which meant it implicitly returned `None`, which triggered the fallback path.
Reviewed By: sfilipco
Differential Revision: D25074067
fbshipit-source-id: 089d5382dd566db89ee732cdcb82762c8d43e21a
Summary: The test doesn't compile on Windows, let's just ifdef it.
Reviewed By: genevievehelsel
Differential Revision: D25033804
fbshipit-source-id: 4f312f010f9d0db42cc9ae19df3f668e8e1c4665
Summary:
Converting back and forth between folly::fs::path and AbsolutePath appears to
be problematic on Windows as NUL bytes appears in the paths, causing the tests
to fail. Instead of doing this conversion, let's simply use AbsolutePath everywhere.
Reviewed By: chadaustin
Differential Revision: D25033803
fbshipit-source-id: 6c45c2a20fc4bf18cecc838b219faacfeb8386d8
Summary:
Querying bookmarks in all repos at the exact same time results in us making a
bunch of concurrent queries to MySQL, which in turn results in MyRouter
allocating a bunch of connections.
This is for reporting the age of bookmarks: that stuff can wait... It doesn't
need to be super fast. Let's make it run on repos one at a time to avoid
allocating dozens of connections every 10 seconds (which is more often than
MyRouter seems to close them).
Reviewed By: ikostia
Differential Revision: D25057432
fbshipit-source-id: 8b65ef65752fc9762a26d835ac80f61573003dd7
Summary:
Skip the need to go through SSH/Mercurial servers. Directly connect to Mononoke
via TLS with X509 certificates.
Reviewed By: markbt
Differential Revision: D24756947
fbshipit-source-id: a782c894956d6a6be055eaa78287c040ceb57da7
Summary: Add a test case for walking hg data from non-public changeset
Reviewed By: ikostia
Differential Revision: D25023130
fbshipit-source-id: 34295f77926b32c77095f7c10d6daa8ef59d9550
Summary: Add command line option to exclude derived data types from blobimport so we can use it to create non-public commits without filenode data for tests
Differential Revision: D25023116
fbshipit-source-id: d8e5d6955f11cebec0de2075c22981bf6c6f4af3
Summary: Add DeletedManifest to walker types with edges to linked changeset and its sub-manifests so that they can be scrubbed.
Differential Revision: D25000319
fbshipit-source-id: f146e6132fde0fb13e630d315484cc2c0a964bdc
Summary: Add the mapping from bonsai to deleted manifest to the walker
Reviewed By: ikostia
Differential Revision: D24989424
fbshipit-source-id: 53d622f661629b9b3de91910f4560b641a95a7bf
Summary: Sometimes pretty debug format is too verbose and one per line regular debug format is preferrable.
Reviewed By: ikostia
Differential Revision: D24996432
fbshipit-source-id: 1acda3985658e4c17b57e36734c77b7579e7e28a
Summary: add an output option to walker scrub so we can dump out debug representation of any type the walker can walk
Reviewed By: farnz
Differential Revision: D24996433
fbshipit-source-id: a332d89d65e4d928159155a34bd39b0e2e1131de
