Summary:
The OpenOptions allow for multiple indices to be added, but lookup had no way
to querying these multiple indices.
Reviewed By: quark-zju
Differential Revision: D20445627
fbshipit-source-id: 0cb754ba17b452d892b7bcb56d502d5753ef963a
Summary:
This type can either be a Mercurial type key, or a content hash based key. Both
the prefetch and get_missing now can handle these properly. This is essential
for stores where data can either be fetched in both ways or when the data is
split in 2. For LFS for instance, it is possible to have the LFS pointer (via
getpackv2), but not the actual blob. In which case get_missing will simply
return the content hash version of the StoreKey, to signify what it actually
has missing.
Reviewed By: quark-zju
Differential Revision: D20445631
fbshipit-source-id: 06282f70214966cc96e805e9891f220b438c91a7
Summary:
Similarly to the DataStore trait, this makes it easier to understand that they
deal with a Mercurial type Key.
Reviewed By: quark-zju
Differential Revision: D20445621
fbshipit-source-id: a1143d5f5d6a2c8686d517a6ea3c25b07c0df072
Summary: This makes it clear that these traits are dealing with Mercurial Key.
Reviewed By: quark-zju
Differential Revision: D20445626
fbshipit-source-id: d5acbf442e9407b973e95e40af69b5a61bff0a4d
Summary: Since they are static and immutable, fbwhoami/fbwhatami could be simple structs with public fields.
Reviewed By: eugeneoden
Differential Revision: D20299423
fbshipit-source-id: 492f49c2b3003760517bfc5be06ace07fabbc6b9
Summary: Add a `trees_under_path` method to `HgRepoContext` which is basically just a way to call Mononoke's implementation of `gettreepack`. This is required for EdenAPI's `/trees/prefetch` endpoint, which basically returns the results of a `gettreepack` request via HTTP.
Reviewed By: StanislavGlebik
Differential Revision: D20414860
fbshipit-source-id: 840bd5d19a10625b78758949b290f1947bdb94be
Summary:
The flat git map file can be huge and writing to it can hang for many
seconds or minutes on loaded systems since it calls fsync. Since we actually
rely on the indexedlog map file instead of the flat map, let's just stop writing
the flat map when indexedlog is enabled. This was previously done to enable
falling back, but we've had no issues and never needed to fallback.
Reviewed By: quark-zju
Differential Revision: D20443392
fbshipit-source-id: a3e7dbdb2b37e2e99df774f540b05d06414fb279
Summary:
This makes the test compatible with Python 2, 3 and Windows.
It also looks easier to read.
Reviewed By: xavierd
Differential Revision: D20444919
fbshipit-source-id: c897e9abc8a5d6d98ff1fc526e2484720fb73bb1
Summary:
Since configparser enforces utf-8 config files (because pest wants Rust strings),
let's migrate from Bytes to Text to remove extra encoding conversions.
Previously this was blocked by the lack of ref-counted text (since the "source"
of each config location is the entire config file). Now minibytes provides Text
so we can use it.
This unfortunately requires dependent code to be updated. The pyconfigparser
interface is in theory wrong - it shouldn't return utf-8 bytes but
local-encoded bytes. I think it's cleaner to make pyconfigparser unaware of
HGENCODING, so I changed pyconfigparser to use unicode, and add compatibility
layer in uiconfig.py.
This also fixes non-ascii encoding issues on user name (especially on Windows).
The hgrc config file should be in utf-8 and the config parser returns explicit
unicode types, and Python code round-trip them with local encodings.
Reviewed By: markbt
Differential Revision: D20432938
fbshipit-source-id: b1359429b8f1c133ab2d6b2deea6048377dfeca1
Summary:
This makes it easier to further migrate to `Text` interface.
Dependent crate (`auth`) is updated.
Reviewed By: markbt
Differential Revision: D20432941
fbshipit-source-id: 1dc29d52c9b17ce14676ef0555470c6d36a09c2b
Summary:
Text is a reference-counted shared String.
It's similar to Bytes but works for utf-8 strings.
The motivation is to replace configparser's use of Bytes to Text.
Reviewed By: markbt
Differential Revision: D20432940
fbshipit-source-id: ef990255d269e60d433c6520819f60ccdcbe488f
Summary: This makes it possible to implement "Text". See the next diff.
Reviewed By: markbt
Differential Revision: D20432943
fbshipit-source-id: 94b3810ab205c260d33f57bd637e4accc3ee871d
Summary:
This makes the API easier to use.
Practically this makes it easier for configparser to migrate to minibytes.
Reviewed By: markbt
Differential Revision: D20432942
fbshipit-source-id: ad08eb118d2216054dc24c86b0b129ae82b9d17c
Summary:
Next OpenNSA release with increased stack size (needed for FBOSS) is available.
Start linking with it.
Differential Revision: D20062997
fbshipit-source-id: 9938270c322087ac3990861aa1ddd3b9ea1148ac
Summary: small cleanup - will help remove clone in the later diff
Reviewed By: farnz
Differential Revision: D20441266
fbshipit-source-id: ee15c7bdbfe2f42ccd1d10abbbe0d43e89547df5
Summary:
let's return the generation numbers as part of commit info - they are cheap to
obtain.
Reviewed By: StanislavGlebik
Differential Revision: D20426744
fbshipit-source-id: 50c7017c55aeba04fb9059e2c1db19f2fb0a6e5e
Summary:
Let's return those to the caller. The changeset is already fetched at this
point so this should come at no extra cost.
Reviewed By: markbt
Differential Revision: D20426745
fbshipit-source-id: 7e28f5fd44efdb502b8e37a7afac3ea9113e2c5e
Summary: I'm about to introduce one more usecase of it so let's rename it first.
Reviewed By: farnz
Differential Revision: D20393776
fbshipit-source-id: d74146fa212cdc4989a18c2cbd28307f58994759
Summary:
Currently, x-repo commit validator runs full working copy comparison every time. This is slow, as it requires fetching of full manifests for the commit versions in both a small and a large repo. For those of leafs, which are different, filenodes are also fetched, so that `ContentId`s can be compared. It's slow. Because it's slow, we skip the majority of commits and validate only 1 in ~40 `bookmarks_update_log` entries. Obviously, this is not ideal.
To address this, this diff introduces a new way to run validation: based on comparing full manifest diffs. For each entry of the `bookmarks_update_log` of the large repo, we:
- unfold it into a list of commits it introduces into the repository
- fetch each commit
- see how it rewrites to small repos
- compute manifest differences between the commit and it's parent in each involved repo
- compare those differences
- check that topological order relationship remains sane (i.e. if `a` is a p1 of `b` in a small repo, than `a'` must be a p1 of `b'` in a large repo, where `a'` and `b'` are remappings of `a` and `b`)
In addition, this diff adds some integration tests and gets rid of the skipping logic.
Reviewed By: StanislavGlebik
Differential Revision: D20007320
fbshipit-source-id: 6e4647e9945e1da40f54b7f5ed79651927b7b833
Summary:
Replace ownership moves with borrows here and there, add fetching of `FileType`
as well.
Reviewed By: krallin
Differential Revision: D20387564
fbshipit-source-id: 378511402277d744ddfbb68e01a3f5d1707f6d08
Summary:
Even simplest test took ~30 seconds to finish. Turned out most of the time is
sepnt in creating sqlite shards, and the reason we haven't noticed it before is
because we haven't use sqlblob in integration tests until recently (until
February 24 to be precise - D20001261).
We don't need 100 sqlite shards, let's decrease it downto 5.
Reviewed By: HarveyHunt
Differential Revision: D20438107
fbshipit-source-id: e71fd4cabf908e3d92b446fc518a0e5dd64a00bb
Summary:
For users on slow connections, hg debugnetwork may not complete within 20s.
Reduce the amount of data we use for the download and upload tests, so that we
have a better chance for the tests completing within the 20s limit.
Reviewed By: farnz
Differential Revision: D20439271
fbshipit-source-id: b3cb0d7e96673c8052022c7606cca80db1788370
Summary: At current, there are some specific AOSP branches that are causing problems for the importer. It would be helpful to be able to run convert jobs that only run on a subset of branches for testing. This commit adds a configuration setting for a whitelist that allows us to specify which repo branches we want to convert, and we only convert those branches.
Reviewed By: tchebb
Differential Revision: D18534728
fbshipit-source-id: bf59605d9c6026e26914c4a7cbe9493bdee77ee5
Summary:
The input for getcommitdata is a list of HgChangesetIds. For every entry, the
endpoint retrieves the commit, formats it like the revlog then returns it back
to the client.
The format for the returned entries is:
```
Hash " " Length LF
RevlogContent LF
```
Looking for recommendations for how to structure the code better.
Looking for recommendations on implementation requirements:
metrics, throttling, veriftying hash for returned bytes.
Reviewed By: krallin
Differential Revision: D20376665
fbshipit-source-id: 5d9eb0d581fd2b352cf3ce44f4777ad45076c8f4
Summary:
The data portion of the changelog revlog from local checkouts is going away.
Instead of fetching data from disk, data will be fetched from the server.
For migration purposes we want the data from the server to match the data that
would have been available locally.
Note. I don't understand the difference between `RevlogChangeset` and
`HgChangesetContent`. I want to eventually call
`mercurial_revlog::changeset::serialize_cs` and this change reduces
the number of transformations required. The whole process is more direct and
thus easier to follow.
Reviewed By: StanislavGlebik
Differential Revision: D20373568
fbshipit-source-id: f2f4eccdf6a35b322e13c5a4b3fa18d5d289848e
Summary:
The diff only contains HgCommand signatures. No implementation yet.
The purpose of the getcommitdata command is to return the serialized contents
of a commit. Given a Mercurial Changelog Id, the endpoint returns the same
contents that the Revlog would return on a Mercurial server.
At this point I am looking for suggestions regarding the protocol and the
implementation. My assumption is that both request and response can be fully
kept in memory. I think that we may decide that the request is going to be
streamed to the client so the initial protocol allows for that.
Requirements:
Input: HgChangelogId
Output: Changelog entry contents
Protocol Summary:
```
Request: "getcommitdata" LF "nodes " Length LF Length*(HEXDIG|" ")
Response: *(40HEXDIG Length LF Length*(%x00-79) LF)
```
A bit of a silly protocol. Let me know what recommendations you have.
The Request is modelled after the "known" command. This allows for efficient
batching compared to a batch command model. It's a bit awkward that we don't
pass in the number of HgChangelogId entries that we have in the request but
that is the existing protocol.
For every HgChangelogId in the request the response will first have a line
with the HgChangelogId that was requested and the length of the contents.
The next line will contain the contents followed by line feed.
Reviewed By: krallin
Differential Revision: D20345367
fbshipit-source-id: 50dffff4f6c60396f564f2f1f519744ce730bf96
Summary:
The metalog message is just for display purpose so it does not have to be
byte-to-byte accurate. This solves potential crashes on Python 2 Windows.
File "c:\Tools\hg\python27.zip\edenscm\mercurial\transaction.py", line 587, in _writemetalog
" ".join(map(util.shellquote, pycompat.sysargv[1:])), int(util.timer())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 1: invalid utf-8
The logic will become more "correct" when we migrate to Python 3.
Reported by: markbt
Reviewed By: markbt
Differential Revision: D20422747
fbshipit-source-id: 41123d132a1e545db77d7321099da611668174f4
Summary:
rust-crypto hasn't been updated since 2016, we are replacing it according to https://github.com/RustSec/advisory-db/blob/master/crates/rust-crypto/RUSTSEC-2016-0005.toml
Specifically, we replace crypto::Digest with digest::Digest, crypto::Sha1 with sha1::Sha1.
Reviewed By: krallin
Differential Revision: D20424524
fbshipit-source-id: aa5d268cb5d3c4b1b4e9f7c6f1f3c41d559f121a
Summary:
hg makes sure commit messages are utf-8. So there is no need to check it.
The check logic is also incorrect if the local encoding is not utf-8:
`ctx.description()` returns local-encoding-encoded bytes, which should not
be treated as utf-8 bytes.
Reviewed By: markbt
Differential Revision: D20424241
fbshipit-source-id: 05480e37335c3f2c7971c28c2e23c115dfc6fba7
Summary: This makes it possible to hide draft branches using `hg hide`.
Reviewed By: markbt
Differential Revision: D20403505
fbshipit-source-id: d316e02c24ce636fdc6593f95a5d974b1ba0fc85
Summary:
During cloud sync, the `_applycloudchanges` step will pull all the new commits
from commit cloud, and then record that it has updated to the new workspace
version in the cloud sync state.
If the transaction subsequently fails and gets rolled back, we will roll back
the pull (so the new commits vanish) but we will not roll back the recording of
having updated to the new cloud workspace version.
On the *next* cloud sync, commit cloud will think the user has deliberately
hidden all of those missing commits, and remove them from the cloud workspace.
Write the cloud sync state within the transaction, so if the transaction is
rolled back, we also roll back the recording of the new workspace version.
Reviewed By: quark-zju
Differential Revision: D20420240
fbshipit-source-id: f3d201f299e491cbff47fc6b703a4e1af4015b18
Summary:
Commit cloud sync works at the store granularity, so the status should be
stored in the store vfs. Move it there.
Improve the output of `hg cloud status` by logging more information about the workspace.
Reviewed By: quark-zju
Differential Revision: D20419223
fbshipit-source-id: 4f0d6b9aab55d2bbeaf89b489606f0bc25400de5
Summary:
Avoid boilerplate in unit tests by consistently using `fbinit::compat_test`.
This is a very mechanical change; most of the modified lines are due to indentation changes.
Reviewed By: krallin
Differential Revision: D20414564
fbshipit-source-id: 11c6bd702de28c8a21978f0345d2b0ff8a574e72
Summary:
Expose the "TreeSpans" type from tracing-collector so filtering can be done on
them.
Reviewed By: DurhamG
Differential Revision: D19797707
fbshipit-source-id: 56b63c8fb79865666ce923612dbd5a9cc512dce6
Summary:
Previously Rust str was serialized into bytes. To be Python 3 friendly, let's
serialize it into `str`.
Reviewed By: markbt
Differential Revision: D19797706
fbshipit-source-id: 388eb044dc7e25cdc438f0c3d6fa5a5740f22e3d
Summary:
Let's use functionality added in D20389226 so that we can generate
diffs even for large files.
I've contemplated between two approaches: either "silently" generate
placeholder diff for files that are over the limit or add a new option
where client can request these placeholders. I choose the latter for a few
reasons:
1) To me option #1 might cause surprises e.g. diffing a single large file
doesn't fail, but diffing two files whose size is (LIMIT / 2 + 1) will fail.
Option #2 let's the client be very explicit about what it needs - and it also
won't return a placeholder when the actual content is expected!
2) Option #2 makes the client think about what it wants to be returned,
and it seems in line with source control thrift API design in general i.e.
commit_file_diffs is not trivial to use by design to because force client to
think about edge cases. And it seems that forcing client to think about additional
important edge case (i.e. large file) makes sense.
3) The code change is simpler with option #2
I also thought about adding generate_placeholder_diff parameter to CommitFileDiffsParams,
but that makes integration with scs_helper.py harder.
Reviewed By: markbt
Differential Revision: D20417787
fbshipit-source-id: ab9b32fd7a4768043414ed7d8bf39e3c6f4eb53e
Summary:
This adds a test demonstrating that we can perform BFS fetching over SSH. The
test should demonstrate that the fetches are:
- Done in a BFS fashion (we don't fetch the entire trees before comparing them,
instead we do a fetch at each layer in the tree). In particular, note that
the cc tree, which is unchanged, doesn't get explored at all.
- Done using the right paths / filenodeids, and therefore the right linknodes
are located.
Reviewed By: farnz
Differential Revision: D20387124
fbshipit-source-id: b014812b0e6e85a5cdf6abefe3fe4f47b004461e
Summary:
This is convenient because it makes it possible to tell what is going on within
Mononoke from the outside (e.g. introspect perf counters).
I'll land this after D20387115.
Reviewed By: farnz
Differential Revision: D20387125
fbshipit-source-id: ee070c4d658a0ec8f232152fe8b34bd0b56e6888
