Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary: Let's not heal 10000 blobs in parallel, that's a little too much data.
Reviewed By: farnz
Differential Revision: D22186543
fbshipit-source-id: 939fb5bc83b283090e979ac5fe3efc96191826d3
Summary:
If we're going to iterate through the whole manifest, we should probably
prefetch it. Otherwise, we might end up doing a whole lot of sequential
fetching. We saw this this week when a change landed in sparse profiles that
caused requests to Mononoke to increase 100-fold.
Unfortunately, I don't think we can selectively only fetch the things we are
missing, so this just goes ahead and fetches everything unconditionally. If
there is a better way to do this, I'm all ears.
Reviewed By: StanislavGlebik, xavierd
Differential Revision: D22118926
fbshipit-source-id: f809fa48a7ff7b449866b42b247bf1da30097caa
Summary: This makes it easier to understand why we're fetching so much data.
Reviewed By: quark-zju
Differential Revision: D22114905
fbshipit-source-id: 7cb6d5c9aebcc8c9089891e030b02176e208bd0f
Summary: This got broken in D22115015 — this fixes it.
Reviewed By: farnz
Differential Revision: D22186138
fbshipit-source-id: 54c05466cdbd3be4f6887a852f099351ea5e891e
Summary: This will be used in the following diffs. It just adds commitsync fixtures in a single place, so that we can later play with them in integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D21952665
fbshipit-source-id: 2933a9f7ea8343d5d52e6c3207e7d78a3ef0be25
Summary:
We had to fork OpenNSA and clone from it, see D19437386 for details.
Broadcom has now started hosting OpenNSA as a tarball.
Thus, we no longer need to maintain a fork (yay!)
This patch points opennsa manifest to fetch opennsa from this new location.
Reviewed By: bkoray
Differential Revision: D22175932
fbshipit-source-id: 51cd777ab836e4f191d78fbb2312925e446ca38f
Summary: This bug can be triggered when your computer name contains emoji. getdeps.py will fail to create this file due to Python attempts to write the file as cp1252 (Windows's default encoding)
Reviewed By: wez
Differential Revision: D22171935
fbshipit-source-id: fc3be2d1050c17ddbe05a0fc91d6613865f092ce
Summary:
This test appears to fail when run in hgbuild. It's not clear why it
doesn't repro locally, but it appears to be a locale issue, so let's just for it
to use C.
Reviewed By: singhsrb
Differential Revision: D22178365
fbshipit-source-id: 007c86681ff1bd4845b1feb41b7bbb031edf172b
Summary:
The server response is not strictly typed. Accessing fields might error out.
Protect against those errors.
Reviewed By: chadaustin
Differential Revision: D22169401
fbshipit-source-id: 4e57220d03c322b5b40d81ac63f4391e286525e4
Summary:
On platform the code is expected to run (Windows), the performance difference
is significant (ex. 1.3s -> 0.8s).
The following benchmark emulates truncating the last 4 revisions from changelog:
```
# hg debugshell on Windows
In [2]: clsize = repo.svfs.stat("00changelog.i").st_size
In [3]: def f1():
...: util.tryunlink(repo.svfs.join("new"))
...: shutil.copyfile(repo.svfs.join("00changelog.i"), repo.svfs.join("new"))
...: with repo.svfs.open("new","a") as f:
...: f.truncate(clsize - 256)
In [5]: def g1():
...: util.tryunlink(repo.svfs.join("new"))
...: with repo.svfs.open("00changelog.i") as f1, repo.svfs.open("new", "w") as f2:
...: size = clsize - 256
...: while size > 0:
...: bufsize = min(size, 1 << 24)
...: f2.write(f1.read(bufsize))
...: size -= bufsize
In [6]: %timeit f1()
1 loop, best of 3: 1.13 s per loop
In [7]: %timeit g1()
1 loop, best of 3: 716 ms per loop
In [8]: %timeit f1()
1 loop, best of 3: 1.51 s per loop
In [9]: %timeit g1()
1 loop, best of 3: 853 ms per loop
In [30]: %timeit f1()
1 loop, best of 3: 1.28 s per loop
In [31]: %timeit g1()
1 loop, best of 3: 755 ms per loop
```
I also removed the `vfs.exists` test as the function should raise if the file
does not exist.
Reviewed By: DurhamG
Differential Revision: D21974771
fbshipit-source-id: 358579c057eb694ce6e5b135cf95067dbc66f6f3
Summary: Add the EdenAPI server to the `hg_mononoke_run_tests` test target. This will make it possible to start an EdenAPI server in Mercurial's Buck tests.
Reviewed By: quark-zju
Differential Revision: D22147098
fbshipit-source-id: 1e29b7c06f1dd74f26d821943cd44ef09a7d505f
Summary: `HgPhase` type is redundant and was adding dependency on mercurial in phases crate.
Reviewed By: farnz
Differential Revision: D22162716
fbshipit-source-id: 1c21841d34897d0072ff6fe5e4ac89adddeb3c68
Summary: In advance of peppering async everywhere, allow subcmd's run method to be async.
Reviewed By: genevievehelsel
Differential Revision: D21892187
fbshipit-source-id: f611faacf95649d8bb5588aeefc4546bd5f63984
Summary:
During expensive getbundle request traversing skiplist uses a lot of cpu, and
in fact it's blocking the whole CPU. krallin suggested to yield since it
should avoid blocking cpus, and that's what this diff is doing.
Reviewed By: krallin
Differential Revision: D22160477
fbshipit-source-id: 5bd881d5c50f4d8e64f2cc90389abc8568ec1df6
Summary:
Fix error "AttributeError: module 'concurrent.futures' has no attribute
'_base'" when loading asyncio.base_futures module. There very well may be a more appropriate fix, but this allows ipdb python3 debugging.
Reviewed By: quark-zju
Differential Revision: D22144225
fbshipit-source-id: eb6a96f79c97e3bd40c1efd9fc75660d6fc372ea
Summary:
As per https://github.com/actions/virtual-environments/issues/709 there started to be some issies with Ubuntu envs running out of space. This should fix it.
Also our Cargo builds use a lot of space, changing them to be non-incremental and removing debug symbols keeps the build fast, but greatly reduces the disk space usage leaving us enough space on GitHub Actions virtual machines.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/23
Reviewed By: farnz
Differential Revision: D22160020
Pulled By: lukaspiatkowski
fbshipit-source-id: c23393e310c15ebf5a18b80f0bb5f1f894d24849
Summary: DangerousOverride is moved into a separate crate. Not only it is usually not needed but it was introducing dependencies on mercurial crate.
Reviewed By: StanislavGlebik
Differential Revision: D22115015
fbshipit-source-id: c9646896f906ea54d11aa83a8fbd8490a5b115ea
Summary: This change will ensure that cloning blobrepo is cheap, even if someone adds field that is expensive to clone. Plus it will result in just one arc-clone instead of cloning all the fields one by one.
Reviewed By: mitrandir77
Differential Revision: D22114066
fbshipit-source-id: ca0c3c78033b4c74872da314a32deb37c05b70ca
Summary: Globalrev does not have any dependencies on mercurial so it can be moved to mononoke_types since it is used in BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D22092491
fbshipit-source-id: 1dded88eb2ace08e8c6c3673e2d50ae1fbb9850d
Summary: Move all mercurial changeset generation logic to `blobrepo_hg`. This is preliminary step is required to decouples BlobRepo from mercurial, and in later stages it will be moved to derived data infra once blobrepo is free of mercurial.
Reviewed By: StanislavGlebik
Differential Revision: D22089677
fbshipit-source-id: bca28dedda499f80899e729e4142e373d8bec0b8
Summary: move HgMutationStore to attributes, and all related methods to BlobRepoHg
Reviewed By: StanislavGlebik
Differential Revision: D22089657
fbshipit-source-id: 8fe87418ccb8a7ad43828758844bdbd73dc0573d
Summary: Move `Filenodes` to `BlobRepo::attributes` as it is mercurial specific.
Reviewed By: ikostia
Differential Revision: D21662418
fbshipit-source-id: 87648a3e6fd7382437424df3ee60e1e582b6b958
Summary: This diff introduces `BlobRepoHg` extension trait for `BlobRepo` object. Which contains mercurial specific methods that were previously part of `BlobRepo`. This diff also stars moving some of the methods from BlobRepo to BlobRepoHg.
Reviewed By: ikostia
Differential Revision: D21659867
fbshipit-source-id: 1af992915a776f6f6e49b03e4156151741b2fca2
Summary:
This diff adds additional filed `BlobRepo::attributes` which can store attributes of arbitrary type. This will help store opaque types inside blobrepo without creating dependency on a crate which contains type definition for this attribute. This diff also moves `BonsaiHgMapping` inside attributes set.
- This work will allow to move mercurial changeset generation logic to derive data infrastructure
Reviewed By: ikostia
Differential Revision: D21640438
fbshipit-source-id: 3abd912e7227738a73ea9b17aabdda72a33059aa
Summary: Not all facebook-specific code was moved out of integration_runner_real.py, but removing part of the code that is left would made the code less readable, the rest of it will be removed while the integration_runner_real.py is made usable for OSS
Reviewed By: farnz
Differential Revision: D22114948
fbshipit-source-id: d9c532a6a9ea653de2b12cffc92fbf45826dad37
Summary: Add link support to CountedBlobstore
Reviewed By: StanislavGlebik
Differential Revision: D22090644
fbshipit-source-id: 36dc5454f1ca12c91d0eac6e5059f554ac5cb352
Summary:
Previously perform_move was able to only create a single commit that moves all
files. However this commit can be very large and it can cause problems for
different sync jobs. Let's add a method that all the files in the stack of
commits rather than in a single commit.
Reviewed By: ikostia
Differential Revision: D22067851
fbshipit-source-id: aff4d01cd547eb98dd6f0c45704371724acee9fc
Summary: If a value is above cachelib limit let's try to compress it.
Reviewed By: krallin
Differential Revision: D22139644
fbshipit-source-id: 9eb366e8ec94fe66529d27892a988b035989332a
Summary: This diff adds a PID-fetchCounts map to `ObjectStore` and makes `ObjectStore` update that map after every `didFetch`
Reviewed By: kmancini
Differential Revision: D22100413
fbshipit-source-id: 740342c7b4a453fe482344c2db9542381c3772e4
