Summary:
add an option to pass some metadata in the token
This will be used for content tokens, for example. We would like to guarantee that the specific content has been uploaded and it had the specific length. This will be used for hg filenodes upload.
Reviewed By: markbt
Differential Revision: D29136295
fbshipit-source-id: 2fbd3917ee0a55f43216351fdbc1a6686eb80176
Summary:
use `commitknown` edenapi api for checking the existing commits
it uses the same `lookup_commits` under the hood but a bit shorter to use
we won't need the tokens for existing changesets, so can use a simpler api
also, make `lookupfilenodes` function a bit shorter
Reviewed By: markbt
Differential Revision: D29134677
fbshipit-source-id: 257624d64480102c34761560b2bd768049cbfa83
Summary: Obsolescence markers have been deprecated in favour of mutation and visibility for some time. Remove syncing of obsmarkers via the commit cloud service.
Reviewed By: liubov-dmitrieva
Differential Revision: D29159443
fbshipit-source-id: 33e1d526a9df5c02059931c880529d695036c616
Summary:
Non-buck build cannot succeed if Python.h is missing.
Let's check it explicitly so we don't pick a bad Python.
Reviewed By: DurhamG
Differential Revision: D29179295
fbshipit-source-id: 421c824053d066914a6611f05815527768f257ee
Summary:
It turns out that `python3` might be a symlink to a fbcode Python that cannot
perform the build in some (not all) environment. Let's try more names like
`python3.8`, `python3.7`, etc.
Reviewed By: DurhamG
Differential Revision: D29178933
fbshipit-source-id: da6cae351f25a90ab8a9da85282d09f79505c5e7
Summary:
IPython 7 has compatibility issues when bundling into zip:
- The dependency parso does not work when embedded in zip. It expects
`python/grammar38.txt` to exist on the filesystem (not in a zip).
- It stacktraces after exiting the IPython shell.
- TAB completion does not really work.
Downgrading to IPython 5 and avoid the bundling the parso dep to resolve the
issues.
This should fix `test-autofix.t` and `test-argspans.py` failures.
Reviewed By: DurhamG
Differential Revision: D29170653
fbshipit-source-id: 14a3d16deaca72fbfb7b3acc0a4246a771c4d0aa
Summary: They will be reused in import_pull_data
Reviewed By: quark-zju
Differential Revision: D29147950
fbshipit-source-id: 192bf33c30067f43c4fcaaf3054741b39efb4e25
Summary: This is an interface for importing pull data into dag
Reviewed By: quark-zju
Differential Revision: D29142979
fbshipit-source-id: b40b94403a044c0b74d1574528aa374ec309a0cf
Summary: This will be used to import pull data into segmented changelog
Reviewed By: quark-zju
Differential Revision: D29142981
fbshipit-source-id: 2d19a035ee0b6cefef8fc0547a5dfb79f284a1de
Summary:
`cl.userust()` is always True after D29020191 (3765f8bd76) so `cl.userust() is False` code
is now dead.
Reviewed By: andll
Differential Revision: D29142464
fbshipit-source-id: f9a7e5c56641218758f12bad3de43d1cd1a71716
Summary: Pack files are no longer supported, yet we still have many tests which exercise them. In preparation for landing `scmstore` as a drop-in replacement for ContentStore, I'm removing our tests which only exist to test datapack-specific functionality.
Reviewed By: DurhamG
Differential Revision: D29099012
fbshipit-source-id: 635a913ee0d93ed8d536e71f8fa6a600b823b343
Summary:
The only real change here is: https://github.com/BurntSushi/ripgrep/pull/1756
This is a patch release but fixes a very glaring bug that others have
depended on. This diff fixes the uses to match the old behavior.
Although it's billed as a "fix", it's actually a huge perf improvement
for Linttool, which uses predominantly recursive suffix globs. The fact
that we don't have to compile ~5,000 regexps at Linttool startup anymore
makes such a huge difference that I am going to do write up soon.
Reviewed By: ndmitchell
Differential Revision: D29085977
fbshipit-source-id: 304470e5fa8cb986738aa0d9dd941641684a9194
Summary:
Python 2 is no longer built anywhere. Let's make the various py3
options the default, like renaming 'make local3' to 'make local' and let's get
rid of the dead setup.py and rpm spec.
Reviewed By: quark-zju
Differential Revision: D29077093
fbshipit-source-id: 0c50c2296fe10ff1db9ac8f9b0df2a4836c0ea5b
Summary:
For a while we had two methods in async runtime: block_on_future and block_on_exclusive, due to historic reasons
Recently those methods were calling same code, and now it is time to replace both of them and rename to block_on
Reviewed By: quark-zju
Differential Revision: D29121107
fbshipit-source-id: 5faa76ae181e491b55d799c23c9de1b4e80298f3
Summary:
If this opens a transaction and fails, then we start a never-ending cycle of
starting hg cloud backup and failing.
This command normally doesn't open transactions but it might via other
extensions, here is one example: P423007962.
This ends up killing people's machines:
https://fb.workplace.com/groups/scm/permalink/3957513017631622/
This fixes that by not kicking off background backups in this case.
Reviewed By: liubov-dmitrieva
Differential Revision: D29136077
fbshipit-source-id: 1c4e6de6147571dd6d728f761324506b1804f303
Summary:
The added command gives access to read (execute) dynamicconfig without
using an on-disk repo.
It can be used by the clone script to stage rollout lazy changelog, or just to
verify dynamic config changes without using a repo.
Reviewed By: DurhamG
Differential Revision: D29123072
fbshipit-source-id: e8856d816a636fa860bfcc9694306a4a37552523
Summary:
implement uploading file content via Eden API
* in this diff I aim to upload file content for the given set of filenodes
* also, the code would check with Mononoke using the lookup request what is already there and skip those
* also, this diff introduces calculation of blake2 hash (called ContentId) for file contents (we would probably need to store/cache those and the mapping from hg filenode id to the canonical Mononoke content_id)
* for every uploaded content EdenApi returns a token that we would also need to store later
Reviewed By: markbt
Differential Revision: D29063229
fbshipit-source-id: 739a44bc3ff904cb04a39514ba5efd01c80ba6d0
Summary: This will be used in eager repo integration tests
Reviewed By: quark-zju
Differential Revision: D29113218
fbshipit-source-id: a24232bd6c19010d8ac90d1305f57f1094b06323
Summary:
Similar to D29111710. Let's avoid asking servers for unknown nodes.
In theory this might affect correctness. Practically, this should only affect "landed as" markers,
because all drafts should be non-lazy. If the "landed as" correctness is an issue, we can fix
forward "landed as" later (ex. by writing down the public commit hash explicitly in
debugmarklanded).
Reviewed By: andll
Differential Revision: D29114049
fbshipit-source-id: db8dc34244feb66919cdff9433b6f18967c9ea9b
Summary: This will be used by upcoming changes.
Reviewed By: andll
Differential Revision: D29114048
fbshipit-source-id: ce963a42ebc5722ca9c61ab0a4e7a7377bbf7abb
Summary:
The mutationstore might contain lots of names (commit hashes) that are unknown
to the repo. When using a lazy changelog, by default, those names will be
looked up remotely, and most of the time the server just returns "name is outside
the graph".
We don't have negative cache to speed it up, because cache invalidation is
tricky - every lazy pull would in theory invalidate them.
To make things faster, let's just skip names in the mutationstore that are unknown
_locally_ without asking the server.
In theory this might affect correctness. Practically, this should only affect "landed as" markers,
because all drafts should be non-lazy. If the "landed as" correctness is an issue, we can fix
forward "landed as" later (ex. by writing down the public commit hash explicitly in
debugmarklanded).
Reviewed By: andll
Differential Revision: D29111710
fbshipit-source-id: 2c1b16a8140ca4f7195bf1206ffe1db4750185b9
Summary: This allows BTreeMap to be used as a provide of config for testing.
Reviewed By: andll
Differential Revision: D28019023
fbshipit-source-id: 94a1585139d6c6612fa163f2ff1aecc909db3a72
Summary: Bumping the crossbeam version so we can use `recv_deadline`. This also necessitates updating the lsp and lsp-types crates.
Reviewed By: alunyov, dtolnay
Differential Revision: D29056473
fbshipit-source-id: 9434e9e0895d82482f4c70afa01a2f77702b965f
Summary:
This functionality is exercised in `test-lfs-copytracing.t` and a few other tests, though I'm not sure if it's necessary outside of that. I've added a `TODO` to investigate and try to eliminate such cases.
Because we'd rather not support this, I implemented it on top of ContentStore - it looks like we might have some lingering datapack-related tests where I think this comes up, too.
Reviewed By: DurhamG
Differential Revision: D29056647
fbshipit-source-id: e6bd5ec31dde06d8c509665f738cfe19270ddc76
Summary:
Modify the `ExtractInnerRef` implementations for `filescmstore` and `treescmstore` to return the new scmstore objects rather than the deprecated ones.
Modify `pycheckout` to use a new methods, `get_oldscmstore`, to access the deprecated scmstore instead (native checkout will be updated to use the new scmstore later on).
Reviewed By: DurhamG
Differential Revision: D29056498
fbshipit-source-id: 09b4655941bca048b01e6d365fcca1ccfb1cd700
Summary: Temporarily implement `upload` on `FileStore` by forwarding to the existing `upload` implementation using `LfsRemote`. Eventually we'll want to re-implement this entirely inside `FileStore`.
Reviewed By: DurhamG
Differential Revision: D29045264
fbshipit-source-id: 8ebbaead4c1032949e5bdd1dcf9a733d0086a77c
Summary: Add a `get_shared_mutable` method to `TreeStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).
Reviewed By: DurhamG
Differential Revision: D29042716
fbshipit-source-id: f2236ff8b47ef213b2ffc61501ca301da02dc492
Summary: Add a `get_shared_mutable` method to `FileStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).
Reviewed By: DurhamG
Differential Revision: D29042485
fbshipit-source-id: e28af0bafac5eba87523b0ef522f32355106b467
Summary: Add `FetchLogger` to `FileStore` to track remote fetches which match `remotefilelog.undesiredfileregex`.
Reviewed By: DurhamG
Differential Revision: D29042243
fbshipit-source-id: 08ec69d979d66ef93e8bbe8659171a412ade7e22
Summary: `ReportingRemoteDataStore` wraps a store and records all fetches which match `remotefilelog.undesiredfileregex`. Since scmstore doesn't use a hierarchical store composition, this change extracts out the matching/reporting logic into `FetchLogger` so that it can be re-used by both `ReportingRemoteDataStore` and scmstore.
Reviewed By: DurhamG
Differential Revision: D29041488
fbshipit-source-id: db0c4b545886ffebcf2a2841a506d301b2f2f230
Summary: Straightforward update of `debugscmstore` from old scmstore to new scmstore. I'll want to improve this command a bit more in the future, but this at least enables you to easily test arbitrary fetches.
Reviewed By: DurhamG
Differential Revision: D29047527
fbshipit-source-id: 4e10cb88cba4b572d3e413640ca3d800940d675d
Summary:
Our log deletion is a two step process. First it deletes the meta file,
then it deletes the log. There was a bug where if the meta file deletion
succeeded but the log deletion failed (often because Eden has the log open) then
future attempts to delete the log will fail because it fails to delete the meta
file.
Reviewed By: quark-zju
Differential Revision: D29035913
fbshipit-source-id: 33ab9a4e46b285819c4bf14031426983d2b4342f
Summary:
Since all the errors are eaten, we don't have good insight into the
rotatelog cleanup process. Windows users were seeing large growth in their
hgcache and this logging would've been useful, so let's add it.
Reviewed By: quark-zju
Differential Revision: D29035914
fbshipit-source-id: 54f0f05aa8bed1911b6b95438cd2d3ccc45e8ec9
Summary:
The crecord width calculation used a function that requires a string,
so we tried to force the rows from bytes to columns, which crashed if the row
wasn't utf8. Let's just use the error="replace" utf8 error handler since we
don't care if the row computation is slightly off.
Reviewed By: quark-zju
Differential Revision: D29043385
fbshipit-source-id: a00d867bca9c62007ae4f68b6198d159deeb6016
Summary: This method will be used to get all Ids that needs to be included into CloneData::id_map during fast forward pull.
Reviewed By: quark-zju
Differential Revision: D29045538
fbshipit-source-id: f6eb04f537aa5365f7588391ea4c60b3dc010a53
Summary: revert the zstd crates back to previous version
Reviewed By: johansglock
Differential Revision: D29038514
fbshipit-source-id: 3cbc31203052034bca428441d5514557311b86ae
Summary:
implement `known` method based on the existing `lookup` API that has been already implemented
this might be useful for eden api pull project / wireproto deprecation, as the pull algorithm queries the server for whether commits are known by the server
Differential Revision: D29030429
fbshipit-source-id: 4d78c8b1e7983d89de917951d9765b1bd1d90fb0
Summary: This will be used in fastpath pull
Reviewed By: quark-zju
Differential Revision: D29010304
fbshipit-source-id: 9a7464fbc42544f8c5cc52f6fa25db5f7b5d6bd9
Summary: idset_to_flat_segments allows to return intersection between flat segments and span set
Reviewed By: quark-zju
Differential Revision: D28980521
fbshipit-source-id: da90c66a021a1e8bb95edbbc2e06cd2a850d7dcd
Summary:
This diff introduces Subspan trait for objects that contains spans and allows to take a 'subspan' of an object
Implementing Subspan trait allows to intersect arbitrary objects and spans
Reviewed By: quark-zju
Differential Revision: D28980523
fbshipit-source-id: 29a2e58ec3c79a838db751d8a37227b4cdaaa372
Summary:
It is only useful for Python revlog-based changelog, which was removed by the
previous diff.
Reviewed By: DurhamG
Differential Revision: D29020190
fbshipit-source-id: 3332f1532a510dc72f21d68801e093b5eba10145
Summary:
Modify the status command to prefer comparing content sha256 when comparing files over comparing content directly, using cached aux data when possible and writing newly computed hashes to cache.
A new config option, `scmstore.status`, gates this new behavior.
When `scmstore.status` is enabled, the `status` output should be more accurate, not assuming files with non-matching `HgId`s in an arbitrary pair of revisions are different. As written, this will cause fetching of data that wouldn't otherwise be fetched, but remote fetching of aux data is planned.
This change also doesn't introduce batch fetching, this is planned.
Reviewed By: DurhamG
Differential Revision: D28826219
fbshipit-source-id: dcd253f169723983738148559d61ba655dd453a4
Summary: Introduce a config for enabling separate indexedlogs for storing "aux data" in scmstore. The storage format is not yet finalized, and isn't used in production yet, so we gate the constructing it on a cofig.
Reviewed By: DurhamG
Differential Revision: D28826209
fbshipit-source-id: 1ed8c20329f21b3ced598eea16d9fac0f40112e6
Summary:
Make `doctor` aware of non-revlog changelog and do not hide commits
incorrectly.
Reviewed By: DurhamG
Differential Revision: D29020189
fbshipit-source-id: af7fab75a877997e3f11dd21e506c965da078f83
Summary: The doctor command assumes revlog changelog, which does not cover all setups.
Reviewed By: DurhamG
Differential Revision: D29020188
fbshipit-source-id: 68bb4e20efb49482df84a71ab7d86a20a6fa93ce
Summary: For all clients and tests we're using revlog changelog based on Rust.
Reviewed By: DurhamG
Differential Revision: D29020191
fbshipit-source-id: 44ab603ea07b8891831692285e94ea0e4df849f5
Summary:
Verify is tightly coupled with revlog details, and assumes a non-lazy repo.
Revlog is going away and our repo is lazy typically, so let's just make verify
a no-op.
Reviewed By: DurhamG
Differential Revision: D28974546
fbshipit-source-id: 487dc65ceb9a1e445b58d03a53353a8382aefc08
Summary:
Practically, our client-side revlog changelogs should be non-inline now for a
long time. There is no need to keep the migration logic.
The revlog is being deprecated too so its implementation details (inline) is
going to be irreverent. The related test is then removed.
Reviewed By: DurhamG
Differential Revision: D28974551
fbshipit-source-id: ea456c46dac11d6a8b225c269b49598ab34c2548
Summary: It was only useful server-side and will be incompatible with upcoming changes.
Reviewed By: DurhamG
Differential Revision: D28974549
fbshipit-source-id: 70a715ce170baa78adb8b1fcf7d29e3d1479c05e
Summary: hgsql is irrelevant. Remove more tests that will be broken by upcoming changes.
Reviewed By: DurhamG
Differential Revision: D29019287
fbshipit-source-id: 6fd04d2eb088a0ca9c975b25a4f28a5772f0e088
Summary:
This test uses bundle2 details that are hard to maintain.
Let's just remove the test.
Reviewed By: DurhamG
Differential Revision: D29019286
fbshipit-source-id: a64918736039331bf2fc3cd23e9c67dd77510c22
Summary:
This test is too tricky to maintain with modern setups. Namely, we don't
support strip and are dropping revlog usage. Let's just remove the test.
Reviewed By: DurhamG
Differential Revision: D28974550
fbshipit-source-id: e8d30e726735432820ceaf4ef27d1b83753122a4
Summary:
We no longer uses this code base for hg server logic.
The test requires non-Rust commit backend which will be removed.
Reviewed By: DurhamG
Differential Revision: D28974547
fbshipit-source-id: 433a6697f6cbf08450c43ce810490fcdb53cf718
Summary: It tests revlog details that are going to be irrelevant.
Reviewed By: DurhamG
Differential Revision: D28974552
fbshipit-source-id: 3ff08473be236849442c3c30d5cf1e1c2a1b628d
Summary: It tests revlog details that are going to be irrelevant.
Reviewed By: DurhamG
Differential Revision: D28974548
fbshipit-source-id: a44e97daa24aece446d899e7711a59cb4a133398
Summary: Revise the help text so it matches the latest implementation.
Reviewed By: DurhamG
Differential Revision: D28971683
fbshipit-source-id: 3f8fb7ccc42a71fdb65b87e2b99d06fb347983f3
Summary: This option allows to exclude ignored files, this way if user specifices wildcard patterns in --include, they will still not include ignored files
Reviewed By: mrkmndz
Differential Revision: D29007062
fbshipit-source-id: a8458811b4c16e11a91abdc31967b53c3cdf2ed7
Summary:
`df` doesn't exist on Windows, so this part of `rage` isn't populated.
The closest equivalent on Windows is `wmic LogicalDisk`. Use this to query the
free space and size of the disks.
Reviewed By: quark-zju
Differential Revision: D28997337
fbshipit-source-id: 08b3b74d70928f2e9801061f049359a58108f4bf
Summary: Update to latest version. This includes a patch to async-compression crate from [my PR updating it](https://github.com/Nemo157/async-compression/pull/125), I will remove once the crate is released.
Reviewed By: mitrandir77
Differential Revision: D28897019
fbshipit-source-id: 07c72f2880e7f8b85097837d084178c6625e77be
Summary:
Now that we don't publish any Python 2 packages, let's drop make local.
Once we've confirmed that nothing was using make local, we can rename make
local3 to be make local.
Reviewed By: kulshrax
Differential Revision: D28647154
fbshipit-source-id: de277887e93a6dbc0324a30f592198ef7c83f818
Summary:
More things needing fixing to be python 3 compatible. Caught when
trying to remove the Python 2 build.
Reviewed By: quark-zju
Differential Revision: D28880028
fbshipit-source-id: d162c78237f330f1f931c3581b25ead24e3ea375
Summary:
The combination of metalog and the new clone pattern of first creating
the repo causes local copy clones to fail on Windows because the initial metalog
files are held open and the copy can't overwrite them.
Let's drop the destrepo before we do the local copy.
Reviewed By: quark-zju
Differential Revision: D28880029
fbshipit-source-id: 2a4ef52675eebf16afa528e645acd927a6110cb4
Summary: This will be used for rate limiting decisions. Also, could be logged to scuba tables to get more info about clients.
Reviewed By: quark-zju
Differential Revision: D28750197
fbshipit-source-id: 83f54e38f998c9dd824ef2d3834c777a44d0ffed
Summary: Let clients connect to lfs with HTTP through unix socket so we don't have to worry about certificates presence.
Reviewed By: johansglock
Differential Revision: D28683392
fbshipit-source-id: f6228b4099ef04fe584e320cb1892e6cb513e355
Summary:
create end to end intergation for the lookup API on the client
Start prototyping of `hg cloud upload` command.
Currently, it just performs lookup for existing heads.
This way we can end to end test the new APIs.
Reviewed By: markbt
Differential Revision: D28848205
fbshipit-source-id: 730c1ed4a21c1559d5d9b54d533b0cf551c41b9c
Summary:
Files upload will be executed in 2 stages:
* check if content is already present
* upload missing files
The check api is generic, it could be used for any id type. Called 'lookup' API.
Reviewed By: markbt
Differential Revision: D28708934
fbshipit-source-id: 654c73b054790d5a4c6e76f7dac6c97091a4311f
Summary:
Previously we set this in the rpm spec, but we need to set it in make
local as well since sometimes hgbuild invokes make local directly.
Ideally we'd put this in setup.py, since make and rpmspecs go through that, but
we need this environment also set for the dulwich build, which we don't really
control the setup.py for.
Reviewed By: singhsrb
Differential Revision: D28902015
fbshipit-source-id: bfc170c3027cc43b24c6a517512a63a71f433d23
Summary:
The recent change to make run-tests work with Python 3 broke the
allow/deny list functionality because it started testing the full test name
instead of the base. This fixes that.
Reviewed By: quark-zju
Differential Revision: D28885125
fbshipit-source-id: 586a71e66e0f094b79e6a3e07e27813db6f662d3
Summary: create `uncopy` command to unmark files that were copied using `copy`.
Reviewed By: quark-zju
Differential Revision: D28821574
fbshipit-source-id: c1c15f6fb2837cec529860aba70b516ddd794f10
Summary:
Time 0.2 is current, and 0.1 is long obsolete. Unfortunately there's a
large 0.1 -> 0.2 API change, so I preserved 0.1 and updated the targets of its
users. Also unfortunate that `chrono` has `oldtime` as a default feature, which
makes it use `time-0.1`'s `Duration` type. Excluding it from the features
doesn't help because every other user is specifying it by default.
Reviewed By: dtolnay
Differential Revision: D28854148
fbshipit-source-id: 0c41ac6b998dfbdcddc85a22178aadb05e2b2f2b
Summary:
They are breaking and hgsql is not relevant (hg server repo was forked). So
let's just remove the tests.
Reviewed By: andll
Differential Revision: D28852159
fbshipit-source-id: 04a47ea489b3f190cffe7f714a9f4161847a2c86
Summary:
Fix remaining issues like encoding and `bname` vs `name` difference
(bname was deleted by a previous change, but it's not just encoding
difference from name, bname does not have " (case x)" suffix).
Differential Revision: D28852092
fbshipit-source-id: df013b284414600deb6f20a5c0883f09906bf976
Summary:
Instrument file scmstore with tracing logging. There's more we should add here, but this will be a good starting place - I've already discovered some issues from looking at the log output. (Why does drop run twice? How does it run twice?)
It'd also probably be nice to support formatting the output like https://crates.io/crates/tracing-tree, which will be a lot less cluttered by the logged fields (like `attrs` on `fetch`).
Reviewed By: DurhamG
Differential Revision: D28750954
fbshipit-source-id: 63baa602f7147d24ac3e34defa969a70a92f96a4
Summary:
Now that EdenFS is using EdenAPI more, let's let it take advantage of
EdenAPI's better batching. We alread have a batch API for files, let's copy the
pattern for trees as well. This adds the C++ bindings. The next diff consumes
this from EdenFS
This is largely just a copy of how batch blob fetching does this. But I'm a C++
noob, so feel free to tear this apart with nits.
Reviewed By: chadaustin
Differential Revision: D28426789
fbshipit-source-id: 88d359985e849018fb3c2b4ef9e52d07c96bf31a
Summary:
Now that EdenFS is using EdenAPI more, let's let it take advantage of
EdenAPI's better batching. We alread have a batch API for files, let's copy the
pattern for trees as well. This first diff just produces the Rust code. Future
diffs will add the C++ bindings then integrate it into EdenFS.
This is largely just a copy of how batch blob fetching does it.
Reviewed By: chadaustin
Differential Revision: D28426790
fbshipit-source-id: 822ef6e7b3458df5dba7a007657e85351162b9ff
Summary: Windows has an issue where subprocess don't inherit stdout/stderr correctly. util.system() has a workaround for this, so let's use that instead of subprocess when executing shell aliases. This fixes 'hg pull --rebase' which is a shell alias.
Reviewed By: kulshrax
Differential Revision: D28815381
fbshipit-source-id: 7521c17166a2b2c0e4ee872dacfd09d2d97e00ce
Summary: These are breaking buck test runs
Reviewed By: quark-zju
Differential Revision: D28802741
fbshipit-source-id: a30c7b64d72356df05676ffab87291a246033d49
Summary:
Previously it requires migrating to doublewrite first. There is no reason why
the doublewrite migration cannot be done automatically. So let's do it.
Reviewed By: DurhamG
Differential Revision: D28757734
fbshipit-source-id: ba2533b5506309610b87865a838d7efe22bccfac
Summary:
Add the `fetch_contentsha256` python method to `filescmstore`, which accepts a list of keys and returns a list of (key, sha256).
This is intended to be used by the modified `status` command implementation, which will prefer comparing content hashes to directly comparing file content.
Reviewed By: DurhamG
Differential Revision: D28696618
fbshipit-source-id: a0304319b0a19d4f09d07bec02dc41964aec7255
Summary:
Merge `found_file` and `found_aux_indexedlog` into a new `found_attributes` method, which simply "or"s the newly found attributes into the `found` map.
Replaces the `satisfies` concept with a new `pending` check, used the same way by each `pending_*` method, which considers a key pending if fetching from a store which returns a given set of attributes would allow us to resolve any requested by missing attributes, optionally taking into account attributes that can be computed from those already found. This will still need to be adjusted to support preferring remote fetching of attributes to local computation, but it is no longer as brittle as the previous implementation: there's no requirement that aux data be computed as content is fetched in order to avoid redundantly fetching content.
Move attribute computation to a separate phase, and filter out un-requested attributes in the `finish` function.
Reviewed By: DurhamG
Differential Revision: D28694192
fbshipit-source-id: 9b096c056736cadc0f97ff09243ed09d5266504d
Summary: Use associated constants instead of methods for `FileAttributes` bit masks.
Reviewed By: DurhamG
Differential Revision: D28724729
fbshipit-source-id: 441c0d2361166824c4ee7cfd5ad0b6f21ee1ac26
Summary:
Previously, the `found_error` required `&mut self`, even though it only ever interacted with the error fields. This prevents Rust's type checker from validating the safety of logging errors while iterating over the `found` map, for instance.
Replacing the `&mut self` method call with a field access into an existing `&mut self` resolves this problem, and allows logging errors while mutating other fetch state.
Reviewed By: DurhamG
Differential Revision: D28722547
fbshipit-source-id: 59c6a530cbf331282d6f654a56e492d47cafcd2f
Summary:
Don't try to fetch from a store if we don't have any pending keys.
Handle missing content when writing to cache after fetching from remote stores. Currently, `found_in_*` will be populated even if we don't store the content, having just used it for aux data computation. This change won't be necessary, but won't cause any problems either, after the next change which only prunes overfetching in the `finish` method, allowing remote blobs to be written to local cache even if we only fetched them to compute their attributes. I might revert this portion of the change, or warn if content is unexpectedly unavailable.
Reviewed By: DurhamG
Differential Revision: D28694964
fbshipit-source-id: 465211c9257cbf49b1cb68856473323fc940f10b
Summary: Extends the previous change to add support for computing aux data (currently only Content Sha256) and caching it locally. Introduces a `FetchState` config option, `compute_aux_data`, which controls if content will be fetched in order to compute aux data, or if unavailable aux data will be treated as "not found".
Reviewed By: DurhamG
Differential Revision: D28528456
fbshipit-source-id: 26189d18c8e453040f3c1f6e22a34d623a5aa40d
Summary:
The migration to Python 3 broke the unified diff code because difflib
expected the paths to also be bytes.
Reviewed By: quark-zju
Differential Revision: D28758876
fbshipit-source-id: 367ef237594d2908377cd8b81def364b77ee02e2
Summary:
`rm -A` means removing files that are "deleted" (`rm`-ed but not `hg rm`-ed).
It does not need to list clean files. Listing clean files can be very slow
in a large repo.
Avoid listing clean files so `rm -A` can be faster.
This has a side effect that we no longer maintain the exit value (0: repo
becomes empty, 1: repo is not empty) like before. But I guess nobody really
cares the 1 exit value (and it does not really make sense in the `rm -A`
case).
Reviewed By: DurhamG
Differential Revision: D28622558
fbshipit-source-id: 2087d6508932905564a8307e9438895538ecede9
Summary:
The usage of bytes for paths and environment variables makes this entire file hacky and makes it not work on Windows. Let's remove all of that.
We still use bytes for test output and other file content type cases.
Reviewed By: andll
Differential Revision: D28227825
fbshipit-source-id: b15993601db501160c9fa4eb2463678cde1fa554
Summary:
Previously, migrating to lazy means repo requirement changes. This diff uses
the new API to actually make the changelog lazy.
Reviewed By: DurhamG
Differential Revision: D28700896
fbshipit-source-id: 82cfd70645230cd67223195e25ef07ae5abe7df6
Summary:
Switch debugrebuildchangelog from using revlog stream clone to lazy segment clone.
This removes the revlog techdebt and can be used as a way to repair
repos with broken segmented changelog. As we migrate off double-write backend we
can no longer migrate down to revlog then migrate up, and a full reclone can be
slow. So a partial reclone command that just recreates the segmented changelog
seems useful.
This command is one of the two commands that handle emergency situations
when segmented changelog related logic goes wrong. The other command
is the emergency clone mode, added by D27897892 (d1413bbbad), which assumes everything
related to segmented changelog is broken on Mononoke side and we still
need to commit and push. This command relies on segmented changelog
related features, such as hash<->location lookup, and clone on Mononoke
to work properly and the server having a compatible IdMap. So it might
not be able to address all issues if Mononoke goes wrong.
Reviewed By: DurhamG
Differential Revision: D28430885
fbshipit-source-id: 17357a33f6fda4a67d46e2c7e7be6653b530f499
Summary:
Use the interruptible block_on API so the Python methods can be interrupted by Ctrl+C.
This is especially useful if some operation triggers lots of expensive network fetches.
Reviewed By: DurhamG
Differential Revision: D28723008
fbshipit-source-id: b6c692de6290a49852eabcd960ebd9b2fb68685a
Summary:
This will be used by the next change to test migrating from a non-lazy
changelog to a lazy changelog actually makes commits lazy.
More commits were added to the graph to test laziness. The old graph
does not have commits that will be made lazy by the current standard
(parents of merges are not lazy).
Reviewed By: DurhamG
Differential Revision: D28700897
fbshipit-source-id: 527c3ce672327ed5e2398c0d89a8e9e92e5b244f
Summary:
This will be used by the next change to migrate from a non-lazy changelog to a
lazy changelog.
Reviewed By: DurhamG
Differential Revision: D28700898
fbshipit-source-id: ff12754f224586b9d0d62f73b41bbb07fc7a6eea
Summary:
If a patch declared the length of it's last hunk as N lines, but it
only contained N-1 lines, the Rust code would enter into an infinite loop. This
could happen if a text editor remove the trailing spaces from a patch file.
Let's fix it and add a test
Reviewed By: kulshrax
Differential Revision: D28683977
fbshipit-source-id: 0a999ae108676531a2cf18e77a3b426ba4647164
Summary: Sometimes things take longer, make sure we are able to distinguish whether that's due to networking, tls handshake, http parsing, or Mononoke wireproto handling.
Reviewed By: markbt
Differential Revision: D28705508
fbshipit-source-id: 1bafda7fc447f2e429690f47fe7ab81cec511494
Summary:
Extends the `FileScmStoreBuilder` to construct two new indexedlog stores for caching aux data. The stores will be created in a directory adjacent to the normal non-LFS indexedlog stores.
Currently aux data stores will not be constructed for production users, a configuration option will be introduced to gate this before `.store_aux_data()` is called in the `filescmstore` constructor bindings.
Reviewed By: DurhamG
Differential Revision: D28689693
fbshipit-source-id: e3ad1594e5beee00b1a8b9fe489e3b6af3a2e93e
Summary:
Modify `FileStore` to introduce basic aux data fetching. Aux data is currently read from a separate IndexedLog store, serialized with `serde_json` (chosen for expediency / ease of debugging, I intend to optimize the storage format before releasing this, at the very least to avoid unnecessarily serializing the key path).
Currently aux data fetching will never succeed, as aux data fetching is not supported in the EdenApi "files" API and nothing else exists to populate the local aux data stores. Later in this stack, computing aux data (currently only content sha256) to populate the aux data storage is implemented.
Reviewed By: DurhamG
Differential Revision: D28526788
fbshipit-source-id: c8e21a1377689d7913a68426a3a480d53148da66
Summary:
Simplify tracking of incomplete fetches in preparation for attributes support in the next change.
Now, all keys which have not been completely and successfully fetched are recorded in `pending`, and are removed only when the complete fetch is recorded in `found`. Keys are now removed from `lfs_pointers` and `pointer_origin` as they are completed, as they aren't needed for anything other than fetching from local LFS and remote LFS respectively.
Reviewed By: DurhamG
Differential Revision: D28546515
fbshipit-source-id: c657e5c6350cadc8da970f57bb7694ed71022efb
Summary:
Now metalog can no longer be `None`. Let's just remove logic handling its
`None` case.
This changes the commitcloud-sync-race test because the metalog itself has
internal locking and changes are atomic.
Reviewed By: DurhamG
Differential Revision: D28595292
fbshipit-source-id: bd9851f5f3bb25f28f15d673f608af2863953c46
Summary:
fncache and store have been default on for years. Enable them unconditionally.
This also makes sure that metalog is always available.
Practically, the only place that does not use fncache is hgsql server repos and
they are irrelevant now.
Reviewed By: DurhamG
Differential Revision: D28595289
fbshipit-source-id: 32b9906c179518acdb17a206b54f98a3dc994921
Summary: I have modified the places where most of the errors were raised that users reported and were resolved by renewal of certificates.
Reviewed By: krallin
Differential Revision: D28568561
fbshipit-source-id: 44fb127a49bde83efee1c934e0435b31f8602a8d
Summary: Upcoming changes will force enable metalog so there will be no way to migrate down.
Reviewed By: DurhamG
Differential Revision: D28595290
fbshipit-source-id: a130b3c60c5b553d024868f28a28e48c50d44783
Summary:
It was added by D8527475 (72c3d8afc1) to workaround hgsql with no-fncache and long file
names synced from svn. Upcoming changes will force fncache to simplify
configuration and the hgsql server code was forked. So let's just delete
the test.
Reviewed By: DurhamG
Differential Revision: D28595291
fbshipit-source-id: 60d2449cca7af46b8b5b3c3b557a36507ff1576e
Summary: This will be used by fbclone to ship lazy commit hash backend.
Reviewed By: DurhamG
Differential Revision: D28554445
fbshipit-source-id: a263ae7683124b3b86f4025b02c7de20dcb9813e
Summary: This makes it possible to use non-debugshell to compact the metalog.
Reviewed By: DurhamG
Differential Revision: D28550902
fbshipit-source-id: 789830ba35243d248397e6a52ee343584c1e01a9
Summary:
The "compact" API rebuilds the metalog by removing older history. It could be
useful to reduce the size overhead of the metalog.
This is also useful if we're doing other "rebuild" work, such as rebuilding the
changelog.
Reviewed By: DurhamG
Differential Revision: D28550903
fbshipit-source-id: 56f875bd955247181236a976dcce6163d126a4b6
Summary:
The zipimport logic requires the pyc mtime to match its source. However, the
Windows system time zone can invalidate it and cause slow startups.
Workaround it by making the zipimport mtime function return a fallback value so
the mtime check is then bypassed.
# zipimport.py, _unmarshal_code
source_mtime, source_size = \
_get_mtime_and_size_of_source(self, fullpath)
if source_mtime: # if source_mtime is false, then the check is bypassed.
# We don't use _bootstrap_external._validate_timestamp_pyc
# to allow for a more lenient timestamp check.
if (not _eq_mtime(_unpack_uint32(data[8:12]), source_mtime) or
_unpack_uint32(data[12:16]) != source_size):
_bootstrap._verbose_message(
f'bytecode is stale for {fullname!r}')
return None
Change my Windows time zone from GMT-7 to GMT-4. Set PYTHONVERBOSE and
PYTHONDEBUG to 1. Ran `hg init -h` and check its stderr. It prints:
# bytecode is stale for 'edenscm.traceimport'
and alike before this change, and no longer after replacing the `__init__.py`
in the zip with the new version.
Reviewed By: DurhamG
Differential Revision: D28622287
fbshipit-source-id: bb3e8e378ea168e4f83f4b6aa9713103b2c90ef8
Summary:
don't apply an old public bookmark if the commit is older than max_sync_age
there is a complicated logic because we need to make sure if we later run it with different commitcloud.max_sync_age value or hg cloud sync --full the bookmarks will be appear back.
So, the changes required in both:
* checkomission
* _mergebookmarks
but both cases covered in the tests
also, if you run with max_sync_age=1000 and later max_sync_age=0, the bookmarks will not disappear, which is expected.
Reviewed By: markbt
Differential Revision: D28572875
fbshipit-source-id: 317e897a2b81c3371dbea7eb39b8925570c1d40a
Summary:
this output is not noisy for big workspaces
if a head is omitted, don't warn about the bookmark because it is expected
Reviewed By: markbt
Differential Revision: D28568919
fbshipit-source-id: eb19e1d155f65de411c1dd41a8be6d83ca71c264
Summary:
The output is a bit too noisy for large workspaces.
We could skip older commits, we know the list coming ordered from commit cloud service.
All hashes are anyway available via `hg cloud sl`.
Also I fixed several look-ups in the list. Omittedheads are heavily used to check if something is present there.
Reviewed By: markbt
Differential Revision: D28568421
fbshipit-source-id: bcf62522798fed92df7ca546c73aa14da95f1567
Summary: Add config pull.httpbookmarks to use the edenapi http protocol to fetch bookmarks in the central local repo pull method. This impacts the pull command, as well as other commands that pull bookmarks.
Reviewed By: quark-zju
Differential Revision: D27479112
fbshipit-source-id: 2b9821f458ec0af2579143fb2c2ed7d3ff41878a
Summary:
Support decompression for mononoke connections. When we request it, Mononoke
can support compression our stream, saving bandwith on low throughput
connections.
Reviewed By: StanislavGlebik
Differential Revision: D28535058
fbshipit-source-id: 7594f72978093a474efd168bb87b41c415310d6c
Summary: It can be used by `cloud ssl` template after D28000088 (b506eeea0c).
Reviewed By: liubov-dmitrieva
Differential Revision: D28561180
fbshipit-source-id: fb4bf3de85f7c320c13a2a53c6a103e85ebb5425
Summary:
Like it says in the title. The API between Bytes 1.x has changed a little bit,
but the concepts are basically the same, so we just need to change the
callsites that were calling `bytes()` and have them ask for `chunk()` instead.
This diff attempts to be as small as it can (and it's already quite big). I
didn't attempt to update *everything*: I only updated whatever was needed to
keep `common/rust/tools/scripts/check_all.sh` passing.
However, there are a few changes that fall out of this. I'll outline them here:
## `BufExt`
One little caveat is the `copy_to_bytes` we had on `BufExt`. This was
introduced into Bytes 1.x (under that name), but we can't use it here directly.
The reason we can't is because the instance we have is a `Cursor<Bytes>`, which
receives an implementation of `copy_from_bytes` via:
```
impl<T: AsRef<[u8]>> Buf for std::io::Cursor<T>
```
This means that implementation isn't capable of using the optimized
`Bytes::copy_from_bytes` which doesn't do a copy at all. So, instead, we need
to use a dedicated method on `Cursor<Bytes>`: `copy_or_reuse_bytes`.
## Calls to `Buf::to_bytes()`
This method is gone in Bytes 1.x, and replaced by the idiom
`x.copy_to_bytes(x.remaining())`, so I updated callsites of `to_bytes()`
accordingly.
## `fbthrift_ext`
This set of crates provides transports for Thrift calls that rely on Tokio 0.2
for I/O. Unfortunately, Tokio 0.2 uses Bytes 0.5, so that doesn't work well.
For now, I included a copy here (there was only one required, when reading from
the socket). This can be removed if we update the whole `fbthrift_ext` stack to
Bytes 1.x. fanzeyi had been wanting to update this to Tokio 1.x, but was blocked on `thrift/lib/rust` using Bytes 0.5, and confirmed that the overhead of a copy here is fine (besides, this code can now be updated to Tokio 1.x to remove the copy).
## Crates using both Bytes 0.5 & Bytes 1.x
This was mostly the case in Mononoke. That's no coincidence: this is why I'm
working on this. There, I had to make changes that consist of removing Bytes
0.5 to Bytes 1.x copies.
## Misuse of `Buf::bytes()`
Some places use `bytes()` when they probably mean to use `copy_to_bytes()`. For
now, I updated those to use `chunk()`, which keeps the behavior the same but
keeps the code buggy. I filed T91156115 to track fixing those (in all
likelihood I will file tasks for the relevant teams).
Reviewed By: dtolnay
Differential Revision: D28537964
fbshipit-source-id: ca42a614036bc3cb08b21a572166c4add72520ad
Summary:
This allows us to do staged rollout where some users are using "lazy" backend
and they won't be migrating down to "doublewrite" backend.
Reviewed By: liubov-dmitrieva
Differential Revision: D28554381
fbshipit-source-id: ebe2e25c96fd3b086a451c3909643d19c64a186c
Summary: Migrating from the lazy backend to lazy backend should be a no-op.
Reviewed By: liubov-dmitrieva
Differential Revision: D28554382
fbshipit-source-id: 71c06584f6f7a89096ce4a94843c88cbea542475
Summary: Modifies `treescmstore` and `filescmstore` to also construct `TreeStore` and `FileStore` respectively. Currently these newly constructed stores are not used anywhere, no application code behavior should change as a result of this.
Reviewed By: DurhamG
Differential Revision: D28237680
fbshipit-source-id: 2bf3fd4b96be8c26e5c1e55cfd2e865f98e6ba91
Summary:
Implement `HgIdDataStore`, `RemoteDataStore`, `LocalStore`, `HgIdMutableDeltaStore`, and `ContentDataStore` for `FileStore`.
Currently I've left `RemoteDataStore::upload` unimplemented, as it's a little more complicated than the other functionality (with lots of private field accesses), and is probably worth building a good API for first. As a temporary workaround, I can store an `LfsRemote` (which requires an associated `LfsStore` for cache) and just call upload on that for now, but that's pretty ugly with the current design. I could also construct one on the fly, but it currently stores a bare `LfsRemoteInner`, not an `Arc<LfsRemoteInner>`. I'll take one of these three approaches after getting the integration tests running with the new `TreeStore` and `FileStore`.
Reviewed By: DurhamG
Differential Revision: D28235602
fbshipit-source-id: 13c72cd9379cba70a2ca7038dad419346fe0b14a
Summary:
Implement `HgIdDataStore`, `RemoteDataStore`, `LocalStore`, `HgIdMutableDeltaStore`, and `ContentDataStore` for `TreeStore`.
Also add a `Drop` impl that flushes the local stores, which matches the behavior of `ContentStore` (such as impl does not exist for the underlying stores, but it might be more appropriate there).
Reviewed By: DurhamG
Differential Revision: D28235060
fbshipit-source-id: 5a12d8c2ecff9fcc204cf437bf6f2a98f08645b4
Summary:
Introduce a new, flat, FileStore implementation. This `FileStore`, like the previously submitted `TreeStore`, directly handles all the fallback, local caching, etc, necessary to implement our storage system.
The API supports fetching batches of `Key`s, writing batches of entries (currently only in the "hg file blob" format, with copy header embedded), and querying only the local subset of underlying stores (to allow implementing `get_missing`). Other store subsets and write features will be added in the future.
Reviewed By: DurhamG
Differential Revision: D28138800
fbshipit-source-id: ca5bb91c66fa078019a19180235dd632ea73a0b3
Summary:
Introduce `from_hg_file_blob` and `from_content` LfsPointersEntry constructors, which are used for creating the correct `LfsPointersEntry` for a `Delta` (HgId + file content).
Add `sha256` accessor to `LfsPointersEntry`. Comments on `LfsPointersEntry` and looking at the construction logic suggest there should always be an associated Sha256 content hash. We use it often, so an accessor is useful to avoid the cumbersome HashMap access + match.
Add `fetch_available` to `LfsStore`, which is used by scmstore for handling cases where either only the pointer, or both the pointer and data are available. Existing LFS code directly accesses the underlying blob and pointer store.
Reviewed By: kulshrax
Differential Revision: D28231747
fbshipit-source-id: e6b1f210605d821f542fcb8e87aea366a0864d44
Summary:
Convert client certificates (which are expected to be supplied as PEM files) into an in-memory PKCS#12 archive to pass into libcurl. This is necessary on certain platforms (such as Windows) whose native crypto APIs do not support loading PEM files.
This was previously landed as D27637069 (5b759a2b52), which unconditionally converted the certificates under the assumption that all major TLS backends support PKCS#12. That assumption is still true, but it did not account for the fact that libcurl itself is dynamically linked on some platforms (such as MacOS), and the system libcurl may be too old to support support in-memory certs (via `CURLOPT_SSLCERT_BLOB` added in libcurl version 7.71.0). This diff gates this feature behind the `http.convert-cert` config option, which we can selectively set on platforms where it is needed.
Reviewed By: mzr
Differential Revision: D28524444
fbshipit-source-id: 4af9cdd60b8ef3977ad81abdb8e406c63795e628
Summary:
I forget to add fbclone build rule for the Python 3 build and that's blocking Mercurial release. This diff fixes that.
(Note: this ignores all push blocking failures!)
Reviewed By: DurhamG
Differential Revision: D28541340
fbshipit-source-id: 2c12583b97ccd18e3a4717b63a4680e8a5c3de46
Summary:
add a config option to show all bookmarks in output of `hg cloud sl`
by default local bookmarks pointing to public commits are not returned unless it's a public root of some draft stack
Reviewed By: markbt
Differential Revision: D28537657
fbshipit-source-id: 0287c18b1b6c79b271f8a67f604024086a37ffcf
Summary: If you have checked out a shared workspace or other user workspace, this part of hg doctor could hide incorrectly, so, should be skipped.
Reviewed By: markbt
Differential Revision: D28505928
fbshipit-source-id: 65e1b3978a916fad2a33bb4f81ff1b75cd657567
Summary: Fetch bookmarks via the http edenapi protocol in the bookmark command with the --list-remote option when all bookmark patterns are full bookmark names (not prefixes).
Reviewed By: kulshrax
Differential Revision: D27331526
fbshipit-source-id: 4f4eda255c551c9b55c6966569755f493335b458
Summary:
The --workspace-version option is currently ignored by interactive history.
Allow it to be used to specify the initial version. This makes jumping back to
a much older version easier.
Reviewed By: liubov-dmitrieva
Differential Revision: D28478194
fbshipit-source-id: f4f121d919e89c298677256f227f2e96d63ef644
Summary: if this option is enabled, the server will be asked to add them
Reviewed By: markbt
Differential Revision: D28412810
fbshipit-source-id: d1531ecf97615cdb5e32d72c8c31598e6a406956
Summary:
This was broken by my recent change to have mergetools respect HGPLAIN
instead of ui.formatted.
Reviewed By: andll
Differential Revision: D28423783
fbshipit-source-id: 00831a6cc47acc11574fcf67462a1dccdde21fda
