Summary:
Right now, ContextConcurrencyBlobstore is instantiated in make_blobstore, which
makes it a lot more effective (3 times more effective, in fact) than we want it
to be, since a ticket is acquired by 3 blobstores in the chain in order to
complete a put:
- The multiplex
- The two underlying blobstores
This also has the potential to deadlock if all tickets are held by the
multiplex, which results in an eventual timeout after 600s of waiting in the
multiplex (this looks like it might be happening at least once or twice per
hour right now on the experimental tier).
In any case, the intention had always been to have one of those per repo, not
one per sub-blobstore, so let's do that. The more natural place to put this
seems to be the RepoBlobstore instantiation.
Since I anticipate I might not be the only one who gets tripped up by this at
some point, I also added a comment about this. I also updated the blobsync
tests to stop re-implementing `RepoBlobstoreArgs::new()` so that adding new
blobstores in RepoBlobstoreArgs will have minimal friction.
Reviewed By: HarveyHunt
Differential Revision: D20467346
fbshipit-source-id: a6ad2d8f04bff1c6fcaa151e947cb8af919eec07
Summary: For now it's identical to LfsParams - it will change in the next diff
Reviewed By: krallin
Differential Revision: D20442530
fbshipit-source-id: 8434610373bb9aefe16702207448283b34676ca2
Summary: Empty flags can be sent by Mercurial, do not error out in that case.
Reviewed By: quark-zju
Differential Revision: D20450124
fbshipit-source-id: c85af42be2afb95b09057583f6fec3a2a13d478a
Summary:
In some rare cases, quickcheck may give us 2 identical paths with different
Key, Mercurial will never do that, therefore reject this.
Reviewed By: quark-zju
Differential Revision: D20451344
fbshipit-source-id: 800b31f1aeea38322052baedb918c4f45a1ec95d
Summary: `buck run edenfsctl -- du --clean` would help reclaim space from buck-out directories.
Reviewed By: chadaustin
Differential Revision: D20203950
fbshipit-source-id: bda045a48ca73b0a1d3e157eff2f92b290a611f5
Summary:
While the sha256 of a blob gives access to its content, it doesn't allow
accessing its metadata, by adding a sha256 index, we can easily get the
metadata of a blob via its content hash.
Reviewed By: quark-zju
Differential Revision: D20445624
fbshipit-source-id: 42c04bd69d3c7380706c6237c5b4f4061c016cca
Summary: This is necessary to properly test LFS stores.
Reviewed By: quark-zju
Differential Revision: D20445625
fbshipit-source-id: 530ddf87249e8d721957806f2d8edef3262f303c
Summary:
The OpenOptions allow for multiple indices to be added, but lookup had no way
to querying these multiple indices.
Reviewed By: quark-zju
Differential Revision: D20445627
fbshipit-source-id: 0cb754ba17b452d892b7bcb56d502d5753ef963a
Summary:
This type can either be a Mercurial type key, or a content hash based key. Both
the prefetch and get_missing now can handle these properly. This is essential
for stores where data can either be fetched in both ways or when the data is
split in 2. For LFS for instance, it is possible to have the LFS pointer (via
getpackv2), but not the actual blob. In which case get_missing will simply
return the content hash version of the StoreKey, to signify what it actually
has missing.
Reviewed By: quark-zju
Differential Revision: D20445631
fbshipit-source-id: 06282f70214966cc96e805e9891f220b438c91a7
Summary:
Similarly to the DataStore trait, this makes it easier to understand that they
deal with a Mercurial type Key.
Reviewed By: quark-zju
Differential Revision: D20445621
fbshipit-source-id: a1143d5f5d6a2c8686d517a6ea3c25b07c0df072
Summary: This makes it clear that these traits are dealing with Mercurial Key.
Reviewed By: quark-zju
Differential Revision: D20445626
fbshipit-source-id: d5acbf442e9407b973e95e40af69b5a61bff0a4d
Summary: Since they are static and immutable, fbwhoami/fbwhatami could be simple structs with public fields.
Reviewed By: eugeneoden
Differential Revision: D20299423
fbshipit-source-id: 492f49c2b3003760517bfc5be06ace07fabbc6b9
Summary: Add a `trees_under_path` method to `HgRepoContext` which is basically just a way to call Mononoke's implementation of `gettreepack`. This is required for EdenAPI's `/trees/prefetch` endpoint, which basically returns the results of a `gettreepack` request via HTTP.
Reviewed By: StanislavGlebik
Differential Revision: D20414860
fbshipit-source-id: 840bd5d19a10625b78758949b290f1947bdb94be
Summary:
The flat git map file can be huge and writing to it can hang for many
seconds or minutes on loaded systems since it calls fsync. Since we actually
rely on the indexedlog map file instead of the flat map, let's just stop writing
the flat map when indexedlog is enabled. This was previously done to enable
falling back, but we've had no issues and never needed to fallback.
Reviewed By: quark-zju
Differential Revision: D20443392
fbshipit-source-id: a3e7dbdb2b37e2e99df774f540b05d06414fb279
Summary:
This makes the test compatible with Python 2, 3 and Windows.
It also looks easier to read.
Reviewed By: xavierd
Differential Revision: D20444919
fbshipit-source-id: c897e9abc8a5d6d98ff1fc526e2484720fb73bb1
Summary:
Since configparser enforces utf-8 config files (because pest wants Rust strings),
let's migrate from Bytes to Text to remove extra encoding conversions.
Previously this was blocked by the lack of ref-counted text (since the "source"
of each config location is the entire config file). Now minibytes provides Text
so we can use it.
This unfortunately requires dependent code to be updated. The pyconfigparser
interface is in theory wrong - it shouldn't return utf-8 bytes but
local-encoded bytes. I think it's cleaner to make pyconfigparser unaware of
HGENCODING, so I changed pyconfigparser to use unicode, and add compatibility
layer in uiconfig.py.
This also fixes non-ascii encoding issues on user name (especially on Windows).
The hgrc config file should be in utf-8 and the config parser returns explicit
unicode types, and Python code round-trip them with local encodings.
Reviewed By: markbt
Differential Revision: D20432938
fbshipit-source-id: b1359429b8f1c133ab2d6b2deea6048377dfeca1
Summary:
This makes it easier to further migrate to `Text` interface.
Dependent crate (`auth`) is updated.
Reviewed By: markbt
Differential Revision: D20432941
fbshipit-source-id: 1dc29d52c9b17ce14676ef0555470c6d36a09c2b
Summary:
Text is a reference-counted shared String.
It's similar to Bytes but works for utf-8 strings.
The motivation is to replace configparser's use of Bytes to Text.
Reviewed By: markbt
Differential Revision: D20432940
fbshipit-source-id: ef990255d269e60d433c6520819f60ccdcbe488f
Summary: This makes it possible to implement "Text". See the next diff.
Reviewed By: markbt
Differential Revision: D20432943
fbshipit-source-id: 94b3810ab205c260d33f57bd637e4accc3ee871d
Summary:
This makes the API easier to use.
Practically this makes it easier for configparser to migrate to minibytes.
Reviewed By: markbt
Differential Revision: D20432942
fbshipit-source-id: ad08eb118d2216054dc24c86b0b129ae82b9d17c
Summary:
Next OpenNSA release with increased stack size (needed for FBOSS) is available.
Start linking with it.
Differential Revision: D20062997
fbshipit-source-id: 9938270c322087ac3990861aa1ddd3b9ea1148ac
Summary: small cleanup - will help remove clone in the later diff
Reviewed By: farnz
Differential Revision: D20441266
fbshipit-source-id: ee15c7bdbfe2f42ccd1d10abbbe0d43e89547df5
Summary:
let's return the generation numbers as part of commit info - they are cheap to
obtain.
Reviewed By: StanislavGlebik
Differential Revision: D20426744
fbshipit-source-id: 50c7017c55aeba04fb9059e2c1db19f2fb0a6e5e
Summary:
Let's return those to the caller. The changeset is already fetched at this
point so this should come at no extra cost.
Reviewed By: markbt
Differential Revision: D20426745
fbshipit-source-id: 7e28f5fd44efdb502b8e37a7afac3ea9113e2c5e
Summary: I'm about to introduce one more usecase of it so let's rename it first.
Reviewed By: farnz
Differential Revision: D20393776
fbshipit-source-id: d74146fa212cdc4989a18c2cbd28307f58994759
Summary:
Currently, x-repo commit validator runs full working copy comparison every time. This is slow, as it requires fetching of full manifests for the commit versions in both a small and a large repo. For those of leafs, which are different, filenodes are also fetched, so that `ContentId`s can be compared. It's slow. Because it's slow, we skip the majority of commits and validate only 1 in ~40 `bookmarks_update_log` entries. Obviously, this is not ideal.
To address this, this diff introduces a new way to run validation: based on comparing full manifest diffs. For each entry of the `bookmarks_update_log` of the large repo, we:
- unfold it into a list of commits it introduces into the repository
- fetch each commit
- see how it rewrites to small repos
- compute manifest differences between the commit and it's parent in each involved repo
- compare those differences
- check that topological order relationship remains sane (i.e. if `a` is a p1 of `b` in a small repo, than `a'` must be a p1 of `b'` in a large repo, where `a'` and `b'` are remappings of `a` and `b`)
In addition, this diff adds some integration tests and gets rid of the skipping logic.
Reviewed By: StanislavGlebik
Differential Revision: D20007320
fbshipit-source-id: 6e4647e9945e1da40f54b7f5ed79651927b7b833
Summary:
Replace ownership moves with borrows here and there, add fetching of `FileType`
as well.
Reviewed By: krallin
Differential Revision: D20387564
fbshipit-source-id: 378511402277d744ddfbb68e01a3f5d1707f6d08
Summary:
Even simplest test took ~30 seconds to finish. Turned out most of the time is
sepnt in creating sqlite shards, and the reason we haven't noticed it before is
because we haven't use sqlblob in integration tests until recently (until
February 24 to be precise - D20001261).
We don't need 100 sqlite shards, let's decrease it downto 5.
Reviewed By: HarveyHunt
Differential Revision: D20438107
fbshipit-source-id: e71fd4cabf908e3d92b446fc518a0e5dd64a00bb
Summary:
For users on slow connections, hg debugnetwork may not complete within 20s.
Reduce the amount of data we use for the download and upload tests, so that we
have a better chance for the tests completing within the 20s limit.
Reviewed By: farnz
Differential Revision: D20439271
fbshipit-source-id: b3cb0d7e96673c8052022c7606cca80db1788370
Summary: At current, there are some specific AOSP branches that are causing problems for the importer. It would be helpful to be able to run convert jobs that only run on a subset of branches for testing. This commit adds a configuration setting for a whitelist that allows us to specify which repo branches we want to convert, and we only convert those branches.
Reviewed By: tchebb
Differential Revision: D18534728
fbshipit-source-id: bf59605d9c6026e26914c4a7cbe9493bdee77ee5