Summary: Log changelog backend types so we can filter commands by type.
Reviewed By: DurhamG
Differential Revision: D24022284
fbshipit-source-id: c402aea0ce3bd20d0f310fea167f24cb1b7a3ae6
Summary: Add Python bindings to the Rust progress wrappers. This may seem pointless since the Rust code just calls right back into Python, but this is a useful step to get the Rust and Python code to use a common interface for progress. (Which, in turn, will allow switching to a Rust progress implementation down the line.)
Reviewed By: markbt
Differential Revision: D23999816
fbshipit-source-id: 9bca0f23170d3ca474a1cb5d547840e63572ec71
Summary:
With the new tracing-core (0.1.10 -> 0.1.16), it's no longer effective
to set EDENSCM_TRACE_LEVEL to more verbose within the test.
Let's set it in run-tests.py which spawns the test process instead.
Reviewed By: kulshrax
Differential Revision: D24004327
fbshipit-source-id: db2cadc7334eb59b25a1e0517e3d1a513e31e0fe
Summary:
These tests rely on sleep to simulate race conditions. They are the
largest source of hgbuild breakages. Let's just disable them. Most of this code
is going away in a couple months anyway when Mononoke takes over and revlogs go
away.
Also fixes a failure in test-bookmarks-strip.t that showed up.
Reviewed By: singhsrb
Differential Revision: D23922660
fbshipit-source-id: 35cacc19de03c031c84c1ebf2124884f4b4248e6
Summary:
The remotefilelog.indexedlog{data,history}store configs are no longer in use,
so we shouldn't use them as a way to test whether the indexedlog are present.
Reviewed By: quark-zju
Differential Revision: D23918133
fbshipit-source-id: 902a369f5e1774b8e2fa1e4b3280fbb7cc377775
Summary:
This test relies on precise timing of racing processes. This is flakey
in our automated tests. Since this is mainly about revlog based repos, and we
only have that on the servers now, and we're going to delete them soon, let's
delete this test to make our tests more stable.
Reviewed By: kulshrax
Differential Revision: D23908316
fbshipit-source-id: 3fd110a8267d3bc16bbcb4545b9ff921274f7588
Summary: Linkrevs are on their way out. Let's switch to linknodes.
Reviewed By: quark-zju
Differential Revision: D23765176
fbshipit-source-id: 0dc1e0db11d732ce1edd24d863f32f08a5a5ce42
Summary:
The rust contentstore doesn't allow runtime manipulation of the list of
stores, which is required in order to insert the bundle store into the store.
Let's continue using the old python union store in these cases. This still let's
us delete the python pack code later, but we'll have to keep around the python
union store until we come up with a better solution.
Reviewed By: quark-zju
Differential Revision: D23689630
fbshipit-source-id: 0c60e6f268e81804149afa24409f64b5303e1e34
Summary:
The remaining test failures are mostly around bundle support, which
I'll fix in a later diff.
Reviewed By: quark-zju
Differential Revision: D23664037
fbshipit-source-id: 2bdde3cb4fcded6e0cf3afdc23269662544821df
Summary:
test-commitcloud-sync.t is a new change and just needs to be made cross
platform.
I have no idea how test-common-commands-fb.t ever worked. When HGRCPATH is set,
I expect the system hgrc to not be loaded, and therefore we can't run hg-clone.
Let's just unset it, since this is meant to test if the new Mercurial can
execute a clone. Ideally we'd redirect the system hgrc to the in-repo
staticfiles, but that's more effort.
Reviewed By: singhsrb
Differential Revision: D23869645
fbshipit-source-id: 66669d9fd9c3a23b01bc43b365723185b7b2ed33
Summary:
Move some commit cloud operations under infinitepush read path:
those are:
* `hg cloud check` command
* `hg cloud sync` command when the local repo is clean
* `hg cloud switch` command will normally use the read path for the dest workspace because we clean up the repo before performing the switch
* `hg cloud rejoin` command we use in fbclone will normally go through the read path as it runs in a fresh repo
If something is broken, there is always a way to rerun any of these command with '--dest' flag pointing it to the write path.
```
./hg cloud check -r 0c9596fd1 --remote --dest infinitepush-write
./hg cloud sync --dest infinitepush-write
./hg cloud switch -w other --dest infinitepush-write
```
Those use cases are limited and the lag of forward filler shouldn't be noticeable for them but we will be able to collect more signal how Mononoke performs with Commit Cloud.
Sitevar to control the routing of read traffic:
https://www.internalfb.com/intern/sv/HG_SSH_WRAPPER_MONONOKE_ROLLOUT/#revisions_list
Reviewed By: mitrandir77
Differential Revision: D23840914
fbshipit-source-id: 40fbe2e72756e7a4cf8bc5be6a0b94f6cf4906b4
Summary:
On Ubuntu the output is a bit different:
```
$ hg cloud sync --use-bgssh
commitcloud: synchronizing 'server' with 'user/test/default'
- remote: /bin/sh: trashssh: command not found
- abort: no suitable response from remote hg!
+ remote: /bin/sh: 1: trashssh: not found
+ abort: no suitable response from remote hg: '[Errno 32] Broken pipe'!
```
Glob them out to make the test pass.
Reviewed By: DurhamG
Differential Revision: D23824735
fbshipit-source-id: 7f96149ee16daff31fd0a1c68975b5edfa27cc46
Summary:
It seems OSX python2 has SIGINT handler set to SIG_IGN by default when running
inside tests. Detect that and reset SIGINT handler to raise KeyboardInterrupt.
This fixes test-ctrl-c.t on OSX.
As we're here, improve test-ctrl-c.t so it checks a bit more things and run
quicker.
Reviewed By: DurhamG
Differential Revision: D23853455
fbshipit-source-id: 05c47650bc80f9880f724828d307c32786265e2c
Summary:
This provides a way to fix the local cache of backed up heads if it is in an
invalid state.
The most important, it will allow early dogfooding of write traffic from Mononoke
without the reverse filler in place for developers or for the team.
You could just run `hg cloud backup -f` assuming the repo is backfilled to fix
any inconsistency when switch between the two backends
Reviewed By: markbt
Differential Revision: D23840162
fbshipit-source-id: bbd331162d65ba193c4774e37324f15ed0635f82
Summary:
D23759711 (be51116cf4) changed the way signal handlers work, which apparently causes
this test to fail. The SIGCHLD signal of the child changing state is received
during os.waitpid, which apparently counts as a signal during a system call,
which throws an OSError.
I'm not sure what the real fix should be. Sleeping gets us past the issue, since
presumably the signal is handled before the system call.
Reviewed By: quark-zju
Differential Revision: D23832606
fbshipit-source-id: 70fca19e419da55bbf546b8530406c9b3a9a6d77
Summary:
Automigration gets messed up with `hg cloud rejoin` command in fbclone code because it triggered by the pull command.
As a result fbclone ends up to join a hostname workspace instead of the default for some cases.
* make sure that the migration never runs if background commit cloud operations are disabled
* also, add skip the migration in the pull command in fbclone
Once of those would be enough to fix the issue but I prefer to make both
changes.
Reviewed By: markbt
Differential Revision: D23813184
fbshipit-source-id: 3b49a3f079e889634e3c4f98b51557ca0679090b
Summary:
This extends the Ctrl+C special handling from edenapi to the entire Python
command so Ctrl+C should be able to exit the program even if it's running
some blocking Rust functions.
`edenapi` no longer needs to spawn threads for fetching.
Reviewed By: singhsrb
Differential Revision: D23759710
fbshipit-source-id: cbaaa8e5f93d8d74a8692117a00d9de20646d232
Summary:
Move bunch of code into a separate file (scm daemon related options). Move them
out of cloud sync.
Also introduce additional check that the `hg cloud sync` command scm daemon
runs is intended for the current connected workspace
In theory when we switch a subscription, the SCM daemon gets notified but races possible and it is better to have this additional check, so SCM daemon triggers cloud sync where it is supposed to.
Reviewed By: markbt
Differential Revision: D23783616
fbshipit-source-id: b91a8b79189b7810538c15f8e61080b41abde386
Summary:
The config is not actually used any more (with rust-commits, it is forced on, without rust-commits,
there is no point to keep it on). Therefore removed.
Reviewed By: singhsrb
Differential Revision: D23771570
fbshipit-source-id: ad3e89619ac5e193ef552c25fc064ca9eddba0c6
Summary: It is handy to test edenapi methods directly.
Reviewed By: kulshrax
Differential Revision: D23750709
fbshipit-source-id: 33c15cecaa0372ba9e4688502e7d8f3fdda7c3b8
Summary:
Add a command to rebuild the changelog without recloning other parts of the
repo. This can be used as a way to recover from corrupted changelog. It
currently uses revlog because revlog is still the only supported format during
streamclone.
In the future this can be used for defragmentation.
Reviewed By: DurhamG
Differential Revision: D23720215
fbshipit-source-id: 6db0453d18dbf553660d55d528f990a4029d9da4
Summary: The command will be provided as hint if username changes has been detected in configuration.
Reviewed By: markbt
Differential Revision: D23769942
fbshipit-source-id: 3e84ecef6dd68267022b92bf10f5e68dfc07f270
Summary:
`hg cloud join --merge` doesn't really solve rename problem because it doesn't
preserve:
1. old heads
2. history
I added a proper API in Commit Cloud Service for renaming workspaces and now we
can use it to provide a rename command and 'rehost' command which is a version
of renaming to bind the current workspace to the current devserver.
Rehost command is meant to be used after dev server migration. I am plannig to
add this to the dev server migration wiki.
Next diff will cover how we can use the rename command to fix a username in workspaces names after username has been changed.
Reviewed By: markbt
Differential Revision: D23757722
fbshipit-source-id: dc11cb226eb76d347cdab70b3c72566448dcd098
Summary:
Some tests run `hg init` right inside the test directory, turning the
entire $TESTTMP into a repo. In future diffs we'll start to rely more on hgcache
being present during tests, which creates a directory in $TESTTMP. Let's make
sure all repos are created as sub-directories of $TESTTMP.
Reviewed By: kulshrax
Differential Revision: D23662077
fbshipit-source-id: 2b2b974ebfd1bd19ad6acd1ebe3e68dd03a09869
Summary:
When sending trees and files we try to avoid sending trees that are
available from the main server. To do so, we currently check to see if the
tree/file is from the local store (i.e. .hg/store instead of $HGCACHE).
In a future diff we'll be moving trees to use the Rust store, which doesn't
expose the difference between shared and local stores. So we need to stop
depending on logic to test the local store.
Instead we can test if the commit is public or not, and only send the tree/file
is the commit is not public. This is technically a revert of the 2018 D7992502 (5e95b0e32e)
diff, which stopped depending on phases because we'd receive public commits from
svn there were not public on the server yet. Since svn is gone, I think it's
safe go back to that way.
This code was usually to help when the client was further ahead than another
client and in some commit cloud edge cases, but 1) we don't do much/any p2p
exchange anymore, and 2) we did some work this year to ensure clients have more
up-to-date remote bookmarks during exchange (as a way of making phases and
discovery more reliable), so hopefully we can rely on phases more now.
Reviewed By: quark-zju
Differential Revision: D23639017
fbshipit-source-id: 34c13aa2b5ef728ea53ffe692081ef443e7e57b8
Summary:
Some tools, like ShipIt, close stdin before they launch the subprocess.
This causes sys.stdin to be None, which breaks our pycompat buffer read. Let's
handle that.
Reviewed By: quark-zju
Differential Revision: D23734233
fbshipit-source-id: 0adc23cd5a8040716321f6ede0157bc8362d56e0
Summary: filter returns a generator in Python 3, but we need a list.
Reviewed By: singhsrb
Differential Revision: D23720661
fbshipit-source-id: 8de3f5844bfe8b85b37c44423733fd2a09967397
Summary:
When I added the surrogateescape patch for the email parser decoder
used during patches, I incorrectly added a corresponding encoder on the other
end when we get the data out of the parser. It turns out the parser is
smart/dumb. When using get_payload() it attempts a few different decodings of
the data and ends up replacing all the non-ascii characters with replacement
bits (question marks). Instead we should use get_payload(decode=True), which
bizarrely actually encodes the data into bytes, correctly detecting the presence
of surrogates and using the correct ascii+surrogateescape encoding.
Reviewed By: singhsrb
Differential Revision: D23720111
fbshipit-source-id: ed40a15056c39730c91067b830f194fbe41e5788
Summary:
This test is flaky due to `hg up` not always reading data from the stores, and
thus not always failing to reading the LFS blob. A better way to force read
from the store is to simply use `hg log -p` to read from the stores.
Reviewed By: DurhamG, singhsrb
Differential Revision: D23718823
fbshipit-source-id: 98bc37a76e93a67d031ba7bfa124b1db816983a1
Summary:
In preparation of moving away from SSH as an intermediate entry point for
Mononoke, let Mononoke work with newly introduced Metadata. This removes any
assumptions we now make about how certain data is presented to us, making the
current "ssh preamble" no longer central.
Metadata is primarily based around identities and provides some
backwards-compatible entry points to make sure we can satisfy downstream
consumers of commits like hooks and logs.
Simarly we now do our own reverse DNS resolving instead of relying on what's
been provided by the client. This is done in an async matter and we don't rely
on the result, so Mononoke can keep functioning in case DNS is offline.
Reviewed By: farnz
Differential Revision: D23596262
fbshipit-source-id: 3a4e97a429b13bae76ae1cdf428de0246e684a27
Summary: That might be used to pass more data to the server
Reviewed By: markbt
Differential Revision: D23704722
fbshipit-source-id: a6e41d615f6548f2f8fd036814c59573a45f93bc
Summary:
`hg cloud rejoin` is used in fbclone
By providing a bit more information about the workspaces available we can improve user
experience and try to eliminate the confusion multiple workspaces cause.
Reviewed By: mitrandir77
Differential Revision: D23623063
fbshipit-source-id: 7598c1b58597032c9cfcef0b44b0ec1b00510ffa
Summary:
For repositories that have the old-style LFS extension enabled, the pointers
are stored in packfiles/indexedlog alongside with a flag that signify to the
upper layers that the blob is externally stored. With the new way of doing LFS,
pointers are stored separately.
When both are enabled, we are observing some interesting behavior where
different get and get_meta calls may return different blobs/metadata for the
same filenode. This may happen if a filenode is stored in both a packfile as an
LFS pointers, and in the LFS store. Guaranteeing that the revisionstore code is
deterministic in this situation is unfortunately way too costly (a get_meta
call would for instance have to fully validate the sha256 of the blob, and this
wouldn't guarantee that it wouldn't become corrupted on disk before calling
get).
The solution take here is to simply ignore all the lfs pointers from
packfiles/indexedlog when remotefilelog.lfs is enabled. This way, there is no
risk of reading the metadata from the packfiles, and the blob from the
LFSStore. This brings however another complication for the user created blobs:
these are stored in packfiles and would thus become unreadable, the solution is
to simply perform a one-time full repack of the local store to make sure that
all the pointers are moved from the packfiles to to LFSStore.
In the code, the Python bindings are using ExtStoredPolicy::Ignore directly as
these are only used in the treemanifest code where no LFS pointers should be
present, the repack code uses ExtStoredPolicy::Use to be able to read the
pointers, it wouldn't be able to otherwise.
Reviewed By: DurhamG
Differential Revision: D22951598
fbshipit-source-id: 0e929708ba5a3bb2a02c0891fd62dae1ccf18204
Summary:
The command is easier to use than `hg cloud join --switch`.
Also highlight the workspace name in the output of `hg cloud status`
Reviewed By: mitrandir77
Differential Revision: D23601507
fbshipit-source-id: 74eb17c9366a9dbe96881c8e3e0705619fadb3d6
Summary:
In order to keep the hgcache size bounded we need to keep track of pack
file size even during normal operations and delete excess packs.
This has the negative side effect of deleting necessary data if the operation is
legitimately huge, but we'd rather have extra downloading time than fill up the
entire disk.
Reviewed By: quark-zju
Differential Revision: D23486922
fbshipit-source-id: d21be095a8671d2bfc794c85918f796358dc4834
Summary:
As the repository grows the opportunity for large downloads increases.
Today all writes to data packs get sent straight to disk, but we have no way to
prevent this from eating all the disk.
Let's automatically flush datapacks when they reach a certain size (default
4GB). In a future diff this will let us automatically garbage collect data packs
to bound the maximum size of packs.
Rotatelog already have this behavior.
Reviewed By: quark-zju
Differential Revision: D23478780
fbshipit-source-id: 14f9f707e8bffc59260c2d04c18b1e4f6bdb2f90
Summary:
This makes it easy for `metaedit` to be used by automation. Provided
with a simple JSON file with hash->{user, message} mapping metaedit will
do all of its work without any prompts.
Reviewed By: quark-zju
Differential Revision: D23545527
fbshipit-source-id: 18763ecacff9143b9ad492faf654b176b0f86d1f
Summary:
The "meaningfulparents" concept is coupled with rev numbers.
Remove it. This changes default templates to not show parents, and `{parents}`
template to show parents.
Reviewed By: DurhamG
Differential Revision: D23408970
fbshipit-source-id: f1a8060122ee6655d9f64147b35a321af839266e
Summary:
Now that the Rust revisionstore records undesired filename fetches,
let's log those results to Scuba in Python.
Reviewed By: StanislavGlebik
Differential Revision: D23462572
fbshipit-source-id: b55f2290e30e3a5c3b67d9f612b24bc3aad403a8
Summary: This hopefully makes it more obvious so it looks less like an hg crash.
Reviewed By: kulshrax
Differential Revision: D23509569
fbshipit-source-id: 7174780bc7e9841e3f89a482280c49427b62fb74
Summary:
The default archive behavior archives the entire working copy. That is
undesirable and easy to accidentally trigger in a large repository. Let's
prevent it and require users to specify what they want archived.
Reviewed By: quark-zju
Differential Revision: D23464818
fbshipit-source-id: c39a631d618c2007e442e691cda542400cf8f4c3
Summary:
See the test change. Partially successful auto restack should have bookmarks
moved.
Reviewed By: DurhamG
Differential Revision: D23441932
fbshipit-source-id: 07e509a70bcc5cf81f702d40ec1b8dc4a5a781ff
Summary:
As a follow up to the previous diff, let's also warn if dirstate includes
marker files that should not be included in any sparse profiles.
Reviewed By: DurhamG
Differential Revision: D23414361
fbshipit-source-id: 3d171328bf0ba5754e5bacde85f09abb4fed8603
Summary:
With segments backend, the revision numbers will be longer than commit hashes
and are confusing.
Reviewed By: DurhamG
Differential Revision: D23408971
fbshipit-source-id: e2057fa644fc7b6be4291f879eee3235bb4e687b
Summary: It is about legacy revlog formats that are no longer relevant.
Reviewed By: DurhamG
Differential Revision: D23390436
fbshipit-source-id: 58c2c432804181bcc6517d6c988777b843fc9ba4
Summary:
We have a few safeguards against creating full checkouts. However we have
sparse profiles that are not full, but that include very large directories
which normally should not be included.
This diff adds a logic that checks if a new sparse profile has any of the "marker"
files i.e. some files from a folder that should not be included. Operation
aborts if that the case, however there's always a way to workaround that.
Reviewed By: DurhamG
Differential Revision: D23414200
fbshipit-source-id: 626f392319eb1be8b35f39cadafb61f3c1dfefe3
Summary:
Mergedrivers can call dirstate.add directly and are adding paths with
"." and "..". Let's block those paths.
Reviewed By: quark-zju
Differential Revision: D23375469
fbshipit-source-id: 64e9f20169cfd50325ecd8ebcc1dd3be7a5cb202
Summary:
extdiff uses shutil.rmtree which calls os.rmdir with new python 3
options. Since we pathc os.rmdir, we need to support those options.
Reviewed By: quark-zju
Differential Revision: D23350968
fbshipit-source-id: 081d179dcd67b51ffdeb6b85899adf4e574a8d0f
Summary: Similar to D18528858 so module names do not need to be spelled twice.
Reviewed By: markbt
Differential Revision: D23091380
fbshipit-source-id: a2a261abc9c78c8805cea62b38498ba65398796d
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D23124169
fbshipit-source-id: 438296002eed14db599d6ec225183bf824096940
Summary:
The debugshell command can be long running and contains uninteresting stuff.
Do not profile it.
Practically this hides showing the background statprof thread when using `%trace`.
Reviewed By: DurhamG
Differential Revision: D23278597
fbshipit-source-id: bad97de22e1be2be8b866bee705ea3a6755aa54b
Summary:
The `--trace` flag enables tracing Python modules.
For compatibility reasons, it also enables `--traceback`.
It can be used with debugshell to make `%trace` more useful.
Reviewed By: sfilipco
Differential Revision: D23278600
fbshipit-source-id: d6d0b34bd5c48111f8cd33d7df115f349b0e95b6
Summary:
I found this when I aborted an rebase Dxxx and trying rebasing again and it
complained about "nothing to rebase". It was caused by Dxxx resolving into
a hidden commit.
Reviewed By: sfilipco
Differential Revision: D23307794
fbshipit-source-id: f7a956b5300240089b6a4648f28cf4a152ee2433
Summary:
The Python 3 email library internally stores the message as text, even
though our input and requested output is bytes. Let's make our own wrapper
around the parser to use ascii surrogateescape encoding so we can get the
actual bytes out later and not get universal newlines.
Based off the upstream 7b12a2d2eedc995405187cdf9a35736a14d60706,
which is basically a copy of the BytesParser implementation (https://github.com/python/cpython/blob/3.8/Lib/email/parser.py) with
newline=chr(10) added.
Reviewed By: quark-zju
Differential Revision: D23363965
fbshipit-source-id: 880f0642cce96edfdd22da5908c0b573887bed12
Summary:
There are users who prefer run `hg cloud leave` if they notice they are
connected to commit cloud sync.
Proving more information and add a prompt might help them to change their mind.
For some users who left new fbclone will connect them back. So on next leave they can learn more information about Commit Cloud Workspaces.
Reviewed By: markbt
Differential Revision: D23346091
fbshipit-source-id: 72f170f7133cd64b772ec75ae29a85dc8809e351
Summary: Provide a way to see segments.
Reviewed By: sfilipco
Differential Revision: D23196408
fbshipit-source-id: b1418f945a5a3364ac73b0f97466d973dd4b6300
Summary:
Windows doesn't seem to follow the same write permission rules, so
let's just skip that part of the test for now.
Reviewed By: xavierd
Differential Revision: D23348515
fbshipit-source-id: bfcfa4f8adf94413047b045987e570ba98f9078d
Summary:
It will be clearer to provide both names because all other commands (like `cloud
sl`, `cloud join`) use short names and if you would like to use the long one you would need to provide
'--raw-workspace' option and this is an advanced option.
Reviewed By: markbt
Differential Revision: D23274288
fbshipit-source-id: d751346a2e49cab21d1054eb196f7977c72c8764
Summary:
This was just a dumb issue. I added a test that depended on the
previous logic which didn't run on OSX.
Reviewed By: kulshrax, singhsrb
Differential Revision: D23324601
fbshipit-source-id: 259175c91c9eb54e8350f196770ff967b53ee841
Summary:
Dynamicconfigs are now part of our critical path. Let's remove the
option to not load them. This also let's us get rid of a circularl dependency
where loading dynamicconfigs required having already loaded some configs. This
will let us move dynammicconfig loading to be before system rc loading in a
later diff.
Reviewed By: sfilipco
Differential Revision: D23309090
fbshipit-source-id: 5138059a8ed944c3616007e7c1289b6a57be0e65
Summary:
An earlier diff moved all dynamicconfig loadding into Rust, but it lost
some of the config validation along the way. This allowed dynamicconfig values
to take precedence over system rcs when they shouldn't yet. Most notably the
remotefilelog.cachepath value.
Let's ensure validation is run on all load paths. A future diff will go a step
farther and move dynamicconfigs to be loaded before system configs so we can
ensure system configs always take precedence, until we can remove them entirely.
Reviewed By: quark-zju
Differential Revision: D23305712
fbshipit-source-id: 33a6b4c56d97fa2e2e8f3acc343a8a8868b797ef
Summary:
Dynamicconfig was throwing errors if hgrc.dynamic wasn't writable.
Let's eat those errors for normal read operations. We still treat it as an error
for straight hg debugdynamicconfig invocations.
Reviewed By: quark-zju
Differential Revision: D23301100
fbshipit-source-id: ed0bd1282d2c7ee747f0909c238a5fa07b7bc9bc
Summary:
We've seen user reports of this error. Let's add a test to demonstrate
it. The next diff will fix it.
Reviewed By: sfilipco
Differential Revision: D23309612
fbshipit-source-id: 6fb9e4e65d3351fa29812fc75095d054465cfe13
Summary:
This could help simplify the graph a lot for repos with lots of merges. For
example, logging tags on linux.git looks like:
o fb893de3 Yesterday at 17:28 master
├─┬─┬─┬─┬─┬─┬─┬─┬─┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o bcf87687 Aug 02 at 14:21 v5.8
╷ ╷ ╷ ╭─────┬─┬───┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o 92ed3019 Jul 26 at 14:14 v5.8-rc7
╷ ╷ ╷ ╭─────┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ba47d845 Jul 19 at 15:41 v5.8-rc6
╷ ╷ ╷ ╭─┬─┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ o 11ba4688 Jul 12 at 16:34 v5.8-rc5
╷ ╷ ╷ ╭─┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ o dcb7fd82 Jul 05 at 16:20 v5.8-rc4
╷ ╷ ╷ ╭─┬─┬─┤
╷ ╷ ╷ ╷ ╷ o ╷ 9ebcfadb Jun 28 at 15:00 v5.8-rc3
╷ ╷ ╭─┬─┬─╯ ╷
╷ ╷ ╷ ╷ o ╷ 48778464 Jun 21 at 15:45 v5.8-rc2
╷ ╷ ╷ ╭─╯ ╷
╷ ╷ ╷ o ╷ b3a9e3b9 Jun 14 at 12:45 v5.8-rc1
╭─┬─┬─┼─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─╮
╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 3d77e6a8 May 31 at 16:49 v5.7
╭─┬─┴───────┬───────────┬─┬───┬─╮
╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 9cb1fd0e May 24 at 15:32 v5.7-rc7
╷ ╰─────────┬─────────────┬─┬─┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o b9bbe6ed May 17 at 16:48 v5.7-rc6
╭───────────┬─────────────┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o 2ef96a5b May 10 at 15:16 v5.7-rc5
╭───────────┬─────────────┬─┬─╯
╷ ╷ ╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 0e698dfa May 03 at 14:56 v5.7-rc4
╭───────────┴───────────┬─┬─╮
o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 6a8b55ed Apr 26 at 13:51 v5.7-rc3
╰─────────────────┬───────┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ae83d0b4 Apr 19 at 14:35 v5.7-rc2
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╭─┤
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ 8f3d9f35 Apr 12 at 12:35 v5.7-rc1
╭─┬─┬───────┬─────┬─┬─┬─┬─┼─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ ╷ 7111951b Mar 29 at 15:25 v5.6
╷ ╭─────────┬─────┬─┬─┬─┴───╮
╷ ╷ ╷ ╷ ╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ 16fbf79b Mar 22 at 18:31 v5.6-rc7
╷ ╷ ╭───────┴─────┬─┬─┬─────╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o fb33c651 Mar 15 at 15:01 v5.6-rc6
╷ ╭─┬─────────────┬─┬─┬─────╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ 2c523b34 Mar 08 at 17:44 v5.6-rc5
╷ ╭─┬─────────────┬─┬─╯ ╷
╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 98d54f81 Mar 01 at 14:38 v5.6-rc4
╷ ╭─┴─────────────┬─╮ ╷
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ f8788d86 Feb 23 at 16:17 v5.6-rc3
....
And with simplification turned on, it looks like:
o fb893de3 Yesterday at 17:28 master
├─╮
o ╷ bcf87687 Aug 02 at 14:21 v5.8
╷ ╷
o ╷ 92ed3019 Jul 26 at 14:14 v5.8-rc7
╷ ╷
o ╷ ba47d845 Jul 19 at 15:41 v5.8-rc6
╷ ╷
o ╷ 11ba4688 Jul 12 at 16:34 v5.8-rc5
╷ ╷
o ╷ dcb7fd82 Jul 05 at 16:20 v5.8-rc4
╷ ╷
o ╷ 9ebcfadb Jun 28 at 15:00 v5.8-rc3
╷ ╷
o ╷ 48778464 Jun 21 at 15:45 v5.8-rc2
├─╯
o b3a9e3b9 Jun 14 at 12:45 v5.8-rc1
╷
o 3d77e6a8 May 31 at 16:49 v5.7
╷
o 9cb1fd0e May 24 at 15:32 v5.7-rc7
╷
o b9bbe6ed May 17 at 16:48 v5.7-rc6
╷
o 2ef96a5b May 10 at 15:16 v5.7-rc5
╷
o 0e698dfa May 03 at 14:56 v5.7-rc4
╷
o 6a8b55ed Apr 26 at 13:51 v5.7-rc3
╷
o ae83d0b4 Apr 19 at 14:35 v5.7-rc2
╷
o 8f3d9f35 Apr 12 at 12:35 v5.7-rc1
╷
o 7111951b Mar 29 at 15:25 v5.6
╷
o 16fbf79b Mar 22 at 18:31 v5.6-rc7
╷
o fb33c651 Mar 15 at 15:01 v5.6-rc6
╷
o 2c523b34 Mar 08 at 17:44 v5.6-rc5
╷
o 98d54f81 Mar 01 at 14:38 v5.6-rc4
╷
o f8788d86 Feb 23 at 16:17 v5.6-rc3
....
Under the hood, the difference is how `reachableroots` gets calculated.
See also D22657197 (a5c36fd0b1) and D22368827 (da42f2c17e).
Since the old behavior almost always seems confusing to human. The new
config is turned on by default (but only takes effect if the "segments"
backend is used).
Reviewed By: sfilipco
Differential Revision: D23095468
fbshipit-source-id: f0fc631d2d9a00e3b36744e4236b43d230d10687
Summary:
Use `git config core.autocrlf false` to silent warnings like:
```
$ git add alpha
+ warning: LF will be replaced by CRLF in alpha.
+ The file will have its original line endings in your working directory
```
Reviewed By: sfilipco
Differential Revision: D23270146
fbshipit-source-id: af3bf241edb9f615bcc285b51cc491385f208039
Summary: The command is needed to restore a deleted workspace
Reviewed By: markbt
Differential Revision: D23250376
fbshipit-source-id: e24a7cbc0aad004291853b4c34d7474789aa9c2b
Summary:
The immediate goal is to run benchmarks on a commit graph provided by a git
repo without converting a whole (large) repo from git to hg. Note git repos can
be cloned in a shallow way so it only contains the commit graph. For example:
git clone https://github.com/torvalds/linux --filter=tree:0 -n
Note: The above command writes repositoryformat=1 in `.git/config`
which is not supported by libgit2. Manually editing it to repositoryformat=0
would enable libgit2 to read it for this crate's use-case.
In the longer term we might want to extend the support so refs/trees/files can
be read/written directly via the git repo based on this work. However that's
currently beyond scope.
Reviewed By: DurhamG
Differential Revision: D23095467
fbshipit-source-id: 868beb0c7de60453b47962639863eb8f7e3f5753
Summary:
Add alternative paths will be faster if changelog2 is used, since they are
backed by native paths.
Add a config option to disable the fast paths if they cause issues.
Reviewed By: DurhamG
Differential Revision: D23036074
fbshipit-source-id: 489b6eac64148867c209d595623d0b9c21ad1d5a
Summary:
OSX doesn't support touch -d. Let's just skip that part of the test on
that platform. This fixes the OSX build.
Reviewed By: singhsrb
Differential Revision: D23253475
fbshipit-source-id: 0eccb884cbdd4bf0a4068fbf943ba7dac9df4e04
Summary:
Detect the "segments" backend and calculate the revset differently.
Practically, with collapse-obsolete disabled, the time of related revset
calculation drops from 0.14s to 0.03s in my fbsource repo.
The `obsolete()` set calculation is expensive (0.4-0.6s) and a bit more
expensive with the new DAG APIs, which will be addressed in upcoming
changes. EDIT: Addressed by D23036063.
Reviewed By: DurhamG
Differential Revision: D23036055
fbshipit-source-id: 71140a88599cc68bfa90d564c786da89b3ebd38b
Summary:
The `compact` template is rarely used and is coupled with rev numbers (ex. rev
number decides what "parents" to show). Use explicit templates. This makes the
test change easier to check.
Reviewed By: DurhamG
Differential Revision: D23036076
fbshipit-source-id: f2cc0f25191711fa7d846a8ad38aee8fb9171273
Summary: This allows migrating between a few changelog formats we have.
Reviewed By: DurhamG
Differential Revision: D22970571
fbshipit-source-id: d6b577ae5beb72a43fff999c26c35fcdc33e8f83
Summary: For now it just prints some details about the changelog backend.
Reviewed By: DurhamG, sfilipco
Differential Revision: D22970573
fbshipit-source-id: 719a5e5bb6f3856df3c9357e47daa9e7c8584952
Summary:
Dynamicconfigs compares the timestamp of config files with the current
timestamp to determine when to regenerate. If the timestamp of the config file
is newer than the current timestamp, Rust throws an exception. Let's handle that
case and treat it as if the file was just created instead of crashing.
Reviewed By: quark-zju
Differential Revision: D23230216
fbshipit-source-id: ca185de7dfca46953e04ec08c84668eda6d749bd
Summary:
If a cloud sync requires both fetching from the cloud and uploading new state to the cloud,
commit the transaction between the two steps, so that a successful cloud fetch is not
rolled back by failure to send to the cloud.
While we're here, limit the number of sync attempts to 3 in one go.
Reviewed By: farnz
Differential Revision: D23211846
fbshipit-source-id: fa97165d94eee973284ff3d00466387b3041306c
Summary: The value of linkrev is invalid with segmented changelog. Do not resolve it.
Reviewed By: sfilipco
Differential Revision: D23036079
fbshipit-source-id: 8f8b097458bc694327db6ba4e2dc4107bdf44157
Summary: Rather than using a Python program, simulate having no editor during tests using `internal:none`.
Reviewed By: quark-zju, ikostia
Differential Revision: D23152446
fbshipit-source-id: 5560f58885ee5959c62f0ac8bcf0483b9c3072f6
Summary:
hg `cloud join -w myworkspace --switch/--merge` should require --create option for all not default
workspaces if the workspace doesn't already exist.
I also enabled reusing of http connection, so it makes `hg cloud switch` much faster.
It is less than 2 sec now.
Reviewed By: markbt
Differential Revision: D23188491
fbshipit-source-id: 9a2ba2df4ba91e9dba9484cd49fc886f70b09880
Summary: The python code stripped new lines but the Rust code did not.
Reviewed By: singhsrb
Differential Revision: D23167515
fbshipit-source-id: add33ec6e4cfd9169e6fef8208490e0aeede38bd
Summary:
The tests weren't windows compatible. Let's get rid of the path
separators.
Reviewed By: quark-zju, singhsrb
Differential Revision: D23170573
fbshipit-source-id: f934691cd2891205442885a12debe3a28d275fc5
Summary:
This new disallowlist will let us specify config section.key's which
should not be accepted from old rc files. This will let us incrementally disable
loading of those configs from the old files, which will then let us delete them
from the old rc's and eventually delete the old rc's entirely.
This diff also removes hgrc.local and hgrc.od from the list of configs we
verify, since those are not on the list of configs that need to be removed in
this initiative.
Reviewed By: quark-zju
Differential Revision: D23065595
fbshipit-source-id: 5cd742d099efd651174cab5e87bb7cdc4bae8054
Summary:
We had two spots that loaded system and user configs, one in the
pyconfigparser layer, and one in the pure rust config layer. In an upcoming diff
I'd like to move dynamicconfig loading down into the pure rust layer, so let's
unify these.
Reviewed By: quark-zju
Differential Revision: D22585554
fbshipit-source-id: 0cea7801ae1d5a3a3c12b80ee23b37f9e690e2bc
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.
Reviewed By: quark-zju
Differential Revision: D23089541
fbshipit-source-id: 5010e417a83a2611283322f1dbb7023f4286f503
Summary: Use filesets so that we don't have to worry about exceeding command line
Reviewed By: quark-zju
Differential Revision: D23008329
fbshipit-source-id: bbfef9ecf23b810f4fa6ca8a6de1bbeb9330ae09
Summary: This diff has a unit test to make sure on the fly sparse profiles work. We also allow for more than one sparse profile, and for multiple exclusion sparse profiles.
Reviewed By: quark-zju
Differential Revision: D23004062
fbshipit-source-id: 15f00401084c89a77d8e7bdd8c7eb688e20d131b
Summary:
Biggrep recently moved to a sharded system where fbsource is spread
over three shards and can have three different corpus revs. Let's update hg's
biggrep integration to handle this new logic.
We basically just take the earliest corpus commit and diff that against the
users commit. There's room for some incorrectness here, like if the users
commit is based off a commit before the earliest corpus commit then it will
receive changes from the alter corpus commits. Long term biggrep will work on
unifying the various shards to use a consistent corpus rev, but for now this is
probably acceptable.
Reviewed By: quark-zju
Differential Revision: D23057055
fbshipit-source-id: 745cafe772cea44bbe2d8eb1eee70dd317f255fe
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D22992103
fbshipit-source-id: b5503e498d5216d4ba19701ecd5582387e4f45f5
Summary:
This makes the main command table cleaner.
I dropped the `indexedlogrepair` command as it cannot rebuild indexes. `hg
doctor` is a better replacement. Some debug commands are renamed so they
no longer have `-` in the command name.
Reviewed By: DurhamG
Differential Revision: D22992107
fbshipit-source-id: f65d74e36fb971e592ad0cc8be9a94e245c39662
Summary:
When running `edenfsctl prefetch **/BUCK` with an empty hgcache, EdenFS ends up
asking mercurial for every manifest one by one. Unfortunately, every manifest
fetched also causes the packfile to be flushed to disk, which then leads EdenFS
to rescan the filesystem for the new packfile. Once too many packfiles are
present on disk, Mercurial triggers a repack. Effectively, that means we have a
quadratic complexity both on Mercurial, and on EdenFS's side.
While this has been a long standing issue, we've so far avoided falling into
this complexity for a number of reason. The main one being that the hgcache is
very rarely empty, and thus the quadratic complexity is usually on low number
of files. Users also rarely run a prefetch of all the files for the entire
repo. However, on repositories with long standing branches, the hgcache is
effectively cold and thus any prefetch would trigger the pathological behavior.
To solve this, we take the same approach taken for files: sending the raw
manifest to EdenFS, which will then take care of deserializing it properly.
Reviewed By: DurhamG
Differential Revision: D23035335
fbshipit-source-id: 855e6fb4fabf81c427fad6c9f17d05f95c47e9ae
Summary:
On macOS, it appears that ssh has a ~1% chance of never being able to connect
too the server and just hang. This caused mactest to be completely unhealthy
for a couple of days and a similar hotfix was applied to mitigate the issue.
Since it proved to be working, let's now backport this hotfix in the actual
code.
Reviewed By: DurhamG
Differential Revision: D22953230
fbshipit-source-id: ead7662ea6d0a33efaa5c4044c9391b2835ee421
Summary:
There are two different magic background syncing that can be enabled. The first
is triggered by commit or any other local changes. The second is triggered by
SCM Daemon by any remote change in this workspace.
I would like to explain it a bit better in `hg cloud status` command.
This will also offer some reassurance to clients.
For example, assume they run `hg cloud disable` command that should disable all background Commit Cloud traffic for some time, so then they can run `hg cloud status` and verify that neither local changes, nor remote changes trigger any commit cloud traffic on this machine.
I also provide full log path to Scm Daemon logs if it is enabled.
Reviewed By: markbt
Differential Revision: D23038954
fbshipit-source-id: c3a5b8f58df729ee3f1c7f15da44ad6e6e0b98f6
Summary:
All of these are already valid utf-8 characters, no need to dance to
decode/encode them again.
Reviewed By: DurhamG
Differential Revision: D22978828
fbshipit-source-id: c5f6e25e71cdcaa1c0558d4a1181b667ffe379fb
Summary:
A commit doesn't show up after `hg pull -r` command if it's known locally.
This is a bug that the test demonstrates.
Reviewed By: quark-zju
Differential Revision: D22977182
fbshipit-source-id: 428094568140892fc8a13004f3395371d8b55ebf
Summary:
This provides Ctrl+C/SIGKILL safety. It's needed because we no longer use the
Python transaction framework. If the write is incomplete, the revlog index
logical length will ensure new processes won't see incomplete data.
The length of revlog data is not tracked, as some "unused" in it does not
really matter. Reading the revlog should be still fine.
Reviewed By: sfilipco
Differential Revision: D22914423
fbshipit-source-id: f2f446cde79c7270cbd1ef165f8707368a0a2990
Summary:
This is more complex than previous libraries, mainly because `dag` defines APIs
(traits) used by other code, which might raise error type not interested
by `dag` itself. `BackendError::Other(anyhow::Error)` is currently used to
capture types that do not fit in `dag`'s predefined error types.
Reviewed By: sfilipco
Differential Revision: D22883865
fbshipit-source-id: 3699e14775f335620eec28faa9a05c3cc750e1d1
Summary:
Add a way to actually try out segmented changelog by setting
`format.use-segmented-changelog`.
The main compatibility issues are assumptions that `0 .. len(repo)` are valid
revision numbers. Segmented changelog uses 2 spans `0 .. len(master_group)`
and `<a large 64-bit int> ..` for the non-master group. Some assumptions
about `len(repo)` were removed.
Segmented changelog also does not support linkrevs since revs can be
re-assigned. Some logic (ex. in treemanifest) that relies on linkrev being
`len(repo)` is changed.
There might be other issues, which will be fixed once we discover them.
Reviewed By: sfilipco
Differential Revision: D22657187
fbshipit-source-id: 688ad718741de0ca15e6aeaf84aced24a20c6b09
Summary:
For example, `master~1000000` should not trigger auto pull of the name
`1000000`. `x~y` is parsed as `(ancestor x y)`.
Reviewed By: singhsrb
Differential Revision: D22905438
fbshipit-source-id: 757ae8856f28126bc6e988d9989a894f83d83eb4
Summary:
This feature should be similar to `hg cloud deletebackup` but for cloud sync
It is needed to deprecate old backups.
Reviewed By: markbt
Differential Revision: D22876332
fbshipit-source-id: 90527bac4f352dc14fadf8b04f6c2df01045f5ce
Summary:
An earlier diff, D21772132 (713fbeec24), add an option to default hgcache data store
writes to indexedlog but it only did it for data, not history. Let's also do it
for history.
Reviewed By: quark-zju
Differential Revision: D22870952
fbshipit-source-id: 649361b2d946359b9fbdd038867e1058077bd101
Summary:
There have been lots of issues with user experience related to authentication
and its help messages.
Just one of it:
certs are configured to be used for authentication and they are invalid but the `hg cloud auth`
command will provide help message about the certs but then ask to copy and
paste a token from the code about interactive token obtaining.
Another thing, is certs are configired to use, it was not hard to
set up a token for Scm Daemon that can be still on tokens even if cloud
sync uses certs.
Now it is possible with `hg auth -t <token>` command
Now it should be more cleaner and all the messages should be cleaner as well.
Also certs related help message has been improved.
Also all tests were cleaned up from the authentication except for the main
test. This is to simplify the tests.
Reviewed By: mitrandir77
Differential Revision: D22866731
fbshipit-source-id: 61dd4bffa6fcba39107be743fb155be0970c4266
Summary:
We shouldn't add any tls related configs to the default configuration.
Tls is not used by default. Tokens are currently the default, and tls is another
option. It is cleaner to cover the defaults in the code itself, rather than add
complexity to the configuration here.
Reviewed By: mitrandir77
Differential Revision: D22864541
fbshipit-source-id: 0c0723c77c2a961a0915617d636b83bc65ac8541
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/41
As of D22098359 (7f1588131b) the default locale used by integration tests is en_US.UTF-8, but as the comment in code mentiones:
```
The en_US.UTF-8 locale doesn't behave the same on all systems and trying to run
commands like "sed" or "tr" on non-utf8 data will result in "Illegal byte
sequence" error.
That is why we are forcing the "C" locale.
```
Additionally I've changed the test-walker-throttle.t test to use "/bin/date" directly. Previously it was using "/usr/bin/date", but the "/bin/date" is a more standard path as it works on MacOS.
Reviewed By: krallin
Differential Revision: D22865007
fbshipit-source-id: afd1346e1753df84bcfc4cf88651813c06933f79
Summary:
Restore the behavior to before D22368827 (da42f2c17e). This also significantly speeds up
graph log like `smartlog` because the fast native path of `reachableroots`
can be used.
Reviewed By: DurhamG
Differential Revision: D22657197
fbshipit-source-id: e3236938d8acfd0935ec45e761763bf0477f2152
Summary:
The `changelog2.changelog` type does not inherit from `revlog`.
It is basically taking implementation from `changelog` with `userust` branches
returning true.
Reviewed By: DurhamG
Differential Revision: D22657195
fbshipit-source-id: dc718d180c7ef3d64f822c3a8c968ef6027047d5
Summary: This will help us verify that the C index is no longer necessary.
Reviewed By: DurhamG
Differential Revision: D22657196
fbshipit-source-id: 16ed74acc5400661572880adf3d8d3267c8b53e2
Summary:
The debug print abuses the `linkmapper`. The Rust commit add logic does not
use `linkmapper`. So let's remove the debug message to be consistent with
the Rust logic.
Reviewed By: DurhamG
Differential Revision: D22657189
fbshipit-source-id: 2e92087dbb5bfce2f00711dcd62881aba64b0279
Summary:
Those tests are going to break with the latest changelog. We're moving away
from revlog so let's just remove the tests.
Reviewed By: DurhamG
Differential Revision: D22657198
fbshipit-source-id: 6d1540050d70c58636577fa3325daca511273a2b
Summary:
The zstore-commit-data code paths are in Python. We want to move them to behind
the Rust HgCommits abstractions. So stop making Python interact with the
low-level details.
Reviewed By: DurhamG
Differential Revision: D22638457
fbshipit-source-id: 435db8425a29ce4eae24a6202ad928f85a5f5ee2
Summary:
Hard link adds complexity for revlog writes. It's not that useful in production
setup. The Rust revlog `flush` API does not break hardlinked files. So let's
just avoid using hard links during local repo clone.
Reviewed By: DurhamG
Differential Revision: D22638460
fbshipit-source-id: 038f4d5c48e9972b14c9e59a9d7ef72b6bc5308d
Summary:
Re-implement the `findcommonheads` logic using `changelog` APIs that are going
to have native support from Rust.
This decouples from revlog-based Python DAG logic, namely `dagutil.revlogdag`,
and `ancestor.incrementalmissingancestors`, unblocking Rust DAG progress, and
cleans up the algorithm to not use revision numbers.
The core algorithm is unchanged. The sampling logic is simplified and tweaked
a bit (ex. no 'initial' / 'quick initial' special cases). The debug and
progress messages are more verbose, and variable names are chosen to match
the docstrings.
I improved the doc a bit, and added some TODO notes about where I think can be
improved.
Reviewed By: sfilipco
Differential Revision: D22519582
fbshipit-source-id: ac8cc8bebad91b4045d69f402e69b7ca28146414
Summary:
It has been long replaced by setdiscovery. This removes another dependency on
`dagutil.revlogdag`.
Reviewed By: DurhamG
Differential Revision: D22519585
fbshipit-source-id: ee261173ba584ffcb3371ec640b233609aafcf77
Summary: This avoids depending on the C index if the Rust DAG is available.
Reviewed By: DurhamG
Differential Revision: D22519587
fbshipit-source-id: a89d91184feaeef6641d2b04353601297bf5d4d5
Summary:
Replace the Python spanset with the Rust-backed idset.
The idset can represent multiple ranges and works better with Rust code.
The `idset` fast paths do not preserve order for the `or` operation, as
demonstrated in the test changes.
Reviewed By: DurhamG, kulshrax
Differential Revision: D22519584
fbshipit-source-id: 5d976a937e372a87e7f087d862e4b56d673f81d6
Summary: It looks nicer if we highlight the current workspace in the list.
Reviewed By: mitrandir77
Differential Revision: D22826619
fbshipit-source-id: 416b77fb57d8dfe19057e248e12d411dfc5f9412
Summary:
Some users on macs and dev servers are connected to their default workspace, other
users after D22802064 will be connected to their machine workspace. Assume a
user decides to reclone the repo. Currently, as a result of rejoin, the second batch of the users will
automatically see all their commits from the default workspace and will be a bit
surprised. It makes sense to adapt rejoin logic of choosing the default
workspace if workspace name is not given.
Reviewed By: markbt
Differential Revision: D22817941
fbshipit-source-id: 764034c9f2d774051c5523cb2db093af525f27d7
Summary: Make the output more informative
Reviewed By: markbt
Differential Revision: D22803543
fbshipit-source-id: 35dd4ff0a1f1003690b250d5284e48e6abb4f4b1
Summary:
If the LFS server is down, we are going to retry fetching filenode from the
Mercurial server directly, who is expected to not return a pointer.
Previously, this was achieved by adding a hack into `get_missing`, but since
the function is no longer called during prefetch calls, we cannot rely on it
anymore. Instead, we can wrap the regular remote store and translate all the
StoreKey::Content onto their corresponding hgid keys.
Reviewed By: DurhamG
Differential Revision: D22565604
fbshipit-source-id: 2532a1fc3dfd9ba5600957ed5cf905255cb5b3fd
Summary:
The simplest fix so far is to erase accessed bookmarks state before switching
New cloud join will
Reviewed By: markbt
Differential Revision: D22791409
fbshipit-source-id: 9675ec03c275e42e640d3a95dd5eda2ae084b92b
Summary:
Switching workspaces is good to have before depracating old style backups.
Otherwise it will be too confusing for users who would like to keep their work
on different hosts separate.
Reviewed By: markbt
Differential Revision: D22692393
fbshipit-source-id: abae7667ce24465e69613f3cdd4cd01471fc7704
Summary:
When using LFS, it's possible that a pointer may be present in the local
LfsStore, but the blob would only be in the shared one. Such scenario can
happen after an upload, when the blob is moved to the shared store for
instance. In this case, during a `get` call, the local LFS store won't be able
to find the blob and thus would return Ok(None), the shared LFS store woud not
be able to find the pointer itself and would thus return Ok(None) too. If the
server is not aware of the file node itself, the `ContentStore::get` would also
return Ok(None), even though all the information is present locally.
The main reason why this is happening is due to the `get` call operating
primarily on file node based keys, and for content-based stores (like LFS),
this means that the translation layer needs to be present in the same store,
which in some case may not be the case. By allowing stores to return a
`StoreKey` when progress was made in finding the key we can effectively solve
the problem described above, the local store would translate the file node key
onto a content key, and the shared store would read the blob properly.
Reviewed By: DurhamG
Differential Revision: D22565607
fbshipit-source-id: 94dd74a462526778f7a7e232a97b21211f95239f
Summary:
In some rare situation, it is possible to have an LFS pointer in both the
packfile and the LFS store. In this case, the flags for this filenode may or
may not be empty, making the assert too strong.
Reviewed By: DurhamG
Differential Revision: D22565605
fbshipit-source-id: b82282b3f47af2a9e607f09a7a7d271ecc4e521a
Summary:
I'm not sure how these landed, but they need fixes to work correctly in
python.
Reviewed By: sfilipco
Differential Revision: D22699596
fbshipit-source-id: fb6d237f4de92c5efa3b422ddb86117fc256460f
Summary:
It now passes
This also caught an issue with our wireprotocol serialization for non-ascii
bookmark names.
Reviewed By: quark-zju
Differential Revision: D22655896
fbshipit-source-id: 3312ff2fd13ebaea7988b18ace9a83137f40daa3
Summary: This change introduces a bail macro that allows tagging errors using the syntax `bail!(fault=Fault::Request, "my normal {}", bail_args)` or `bail!(Fault::Request, "my normal {}", bail_args)`.
Reviewed By: DurhamG
Differential Revision: D22646428
fbshipit-source-id: a6ec2940001b26db8ddc3a6d3620a1e17406c867
Summary:
When we change `CommitSyncConfig`, we want to not have to restart `scs` servers, and instead have them pick up the new config by using `LiveCommitSyncConfig`.
This diff turned out larger than I expected, mainly due to the need to introduce various things around `TestLiveCommitSyncConfig`:
- `TestLiveCommitSyncConfig`: the trait implementer to use in `mononoke_api::Repo`
- `TestLiveCommitSyncConfigSource`: the helper struct to keep around for new values injection (very similar to how our `ConfigStore` has an inner `ConfigSource`, which can also be `TestSource`, but here injected values can be `CommitSyncConfig` instead of JSON).
- all the places in integration tests, where `setup_configerator_configs` is now needed (I plan to start setting up configerator configs always in a separate diff, as it is cheap)
Here are the explanations for a few things I think may be not immediately obvious:
- I removed the `Clone` bound from `LiveCommitSyncConfig` trait, as `Clone` traits [cannot be made into trait objects](https://doc.rust-lang.org/book/ch17-02-trait-objects.html#object-safety-is-required-for-trait-objects)
- `TestLiveCommitSyncConfigSource` vs `TestLiveCommitSyncConfigSourceInner` discrepancy is to ensure nobody should instantiate `TestLiveCommitSyncConfigSourceInner` outside of `live_commit_sync_config/src`
- I am aware of the ugly discrepancy between the main `--mononoke-config-path`, which is used to load initial configuration and can be both a file-based and a configerator-based config; and `--local-configerator-path`, used to override config sources for `Tunables` and `LiveCommitSyncConfig`. Ideally these two should just be unified somehow, but that is a little out of scope of this diff (I've already added it to the dirt doc though).
- in `mononoke_api::Repo` there are methods `new_test_xrepo` and `new_test_common`, which changed from maybe accepting just a `CommitSyncConfig` to now maybe accepting both `CommitSyncConfig` and `LiveCommitSyncConfig`. It can be made a bit cleaner: I can just query `CommitSyncConfig` from `LiveCommitSyncConfig` in `new_test_common` and avoid having two args. I was too lazy to do this, lmk if you feel strongly about it.
Reviewed By: StanislavGlebik
Differential Revision: D22443623
fbshipit-source-id: 0d6bbda2223e77b89cc59863b703db5081bcd601
Summary: This makes tests depend less on revision numbers.
Reviewed By: DurhamG
Differential Revision: D22468669
fbshipit-source-id: 74a06930faa3e6ee9d246ecc718c2a3740f57a54
Summary:
The Rust strip logic works for both revlog and segmented changelog. Use it in
tests for easier segmented changelog migration.
The old revlog strip has a lot of complexities on linkrev. Since this is only
used for testing, I avoided the complexities by just saying all linkrevs are
invalidated so we can skip the complexities of dealing with filelog or
manifest, or generating a temporary bundle and unbundling them later.
Most test changes are because we no longer strip filelogs. So unbundling
them will not add new file revisions.
`test-repair-strip.t` is removed because it is only about "Permission Denied"
cases. With the new code path file logs are not stripped and the test is not
relevant.
This does not affect hgsql as it has the Rust logic totally disabled.
Reviewed By: DurhamG
Differential Revision: D22402198
fbshipit-source-id: 1c207393f59a8a3a601cfbdf7f07534b8acf46bf
Summary:
Extend `_log_exception` to inspect exceptions for metadata, and log any that is found to Scuba. Eventually I'd like to merge at least some of this information from the `perfpipe_hg_errors` table into the `perfpipe_dev_command_timers`.
Introduce `Tagged` mixin and `Fault` enum Python types for tagging Python exceptions.
Introduce `debugthrowexception` for testing Python error handling flow.
Reviewed By: DurhamG
Differential Revision: D22585764
fbshipit-source-id: f6300ae25db25d5e75fab9c2a03fb4f9beb06c1f
Summary: This makes it more compatible if the strip algorithm gets changed.
Reviewed By: DurhamG
Differential Revision: D22402194
fbshipit-source-id: 0d296117c6959c0559bb33dcbe1b2de1624dc3b7
Summary:
I dropped the special case of wdir handling. With the hope that we will handle
the virtual commits differently eventually (ex. drop special cases, insert real
commits to Rust DAG but do not flush them to disk, support multiple wdir
virtual commits, null is no longer an ancestor of every commit).
`test-listkeyspatterns.t` is changed because `0` no longer resolves to `null`.
Reviewed By: DurhamG
Differential Revision: D22368836
fbshipit-source-id: 14b9914506ef59bb69363b602d646ec89ce0d89a
Summary:
This can pick a different common ancestor in criss-cross case.
The test is updated to reflect that.
Reviewed By: DurhamG
Differential Revision: D22368837
fbshipit-source-id: 9f897142b3a4bacd0fc90a4a054b7806f1bde826
Summary:
`unittestify.py` has logic to discover executables. It is hardcode as Python 2
hg for now. Change it to be aware of the Python 3 versions.
Reviewed By: DurhamG
Differential Revision: D22562875
fbshipit-source-id: 2a19ac6ee805e22fbcc2651ada5f4eaa1bc85f51
Summary:
Implements based Rust-Python binding layer for error metadata propagation.
We introduce a new type, `TaggedExceptionData`, which carries CommonMetadata and the original (without metadata) error message for a Rust Anyhow error. This class is passed to RustError and can be accessed in Python (somewhat awkwardly) via indexing:
```
except error.RustError as e:
fault = e.args[0].fault()
typename = e.args[0].typename()
message = e.args[0].message()
```
As far as I can tell, due to limitations in cpython-rs, this can't be made more ergonomic without introducing a Python shim around the Rust binding layer, which could adapt the cpython-rs classes to use whatever API we'd like.
Currently, anyhow errors that are not otherwise special-cased will be converted into RustError, with both the original error message and any attached metadata printed as shown below
```
abort: intentional error for debugging with message 'intentional_error'
error has type name taggederror::IntentionalError and fault None
```
We can of course re-raise the error if desired to maintain the previous behavior for handling a RustError.
If we'd like other, specialized Rust Python Exception types to carry metadata (such as `IndexedLogError`), we'll need to modify them to accept a `TaggedExceptionData` like `RustError`.
Renamed the "cause an error in pure rust command" function to `debugcauserusterror`, and instead used the name `debugthrowrustexception` for a command which causes an error in rust which is converted to a Python exception across the binding layer.
Introduced a simple integration test which exercises `debugthrowrustexception`.
Added a basic handler for RustError to scmutil.py
Reviewed By: DurhamG
Differential Revision: D22517796
fbshipit-source-id: 0409489243fe739a26958aad48f608890eb93aa0
Summary:
This fix was originally meant to prevent doing checkunknown files when
it was clear there was no conflict. Unfortunatley, the data doesn't appear to
show it helped, and in some cases it definitely hurt it. Let's back it out for
now until we can do more investigation.
Reviewed By: StanislavGlebik
Differential Revision: D22574000
fbshipit-source-id: aeb644ecd6da046df17e6d10418a72363c1ee532
Summary:
D22526230 (7ab2c54f08) adds extra output.
The test seems to be skipped in buck test due to the hgsql requirement.
Reviewed By: kulshrax
Differential Revision: D22555707
fbshipit-source-id: 8659de3cda517a49296c747e65bfbd0d372c031a
Summary:
Introduce new rust library, taggederror, which contains utilities for attaching metadata to errors. The library provides two main methods for attaching metadata to an error, the TaggedError wrapper type, and the AnyhowExt trait methods. Provides a struct, CommonMetadata, which contains all the metadata types introduced by taggederror (fault, transience, category, and typename), which can also be attached individually (and the same pattern can be used to attach other metadata).
Introduce a new native rust command, debugthrowrustexception, which causes the command to return an error, with some attached metadata.
Modify hg rust native command dispatch error handling to use debug formatter to print anyhow::Error errors. This will print out the source chain, contexts, and backtrace if available, which will cause the metadata we attach as a wrapper error or context to be printed.
Reviewed By: DurhamG
Differential Revision: D22420941
fbshipit-source-id: d38c5a10b686d86b69a2c0a19f5bcbf4ca24dff6
Summary:
Previously you could only canary locally on a devserver by setting an
environment variable. Let's add a --canary flag to debugdynamicconfig that
accepts a host. Hg will ssh to that host and run the configerator cli to grab
the canaried config from that host.
Reviewed By: quark-zju
Differential Revision: D22535509
fbshipit-source-id: af1c21d8402c4e729769e50388d913bf52b66b89
Summary:
The `helpers-testrepo.sh` runs something like
hg files --cwd /path/to/fbsource/fbcode/eden/scm/tests/
to check if `hg` works or not. Previously it fails fast because of the
lz4revlog requirement. The next diff will enable lz4revlog that makes
`hg files` work.
That `hg files` command (if does not fail) is quite expensive because it has no
patterns and lists every file.
It can also be undesirable if the in-repo hg does some kind of auto migration
that makes the repo unusable by the system hg. Therefore just unconditionally
use the system hg.
Reviewed By: DurhamG
Differential Revision: D22419967
fbshipit-source-id: 2d5c75696efecce1ec38a2e4fa23aff101219545
Summary: `hg revert -i` now works and test-revert-interactive.t now passes
Reviewed By: quark-zju
Differential Revision: D22517586
fbshipit-source-id: 76e3c76ea1687fecbd7d66fff42312dac1300c17
Summary:
Previously we would audit all configs and report them if the
dynamicconfig did not match the rc-file config. Now that dynamicconfigs are
widely deployed, let's switch this around to auditing only configs we know have
had issues. This will let us start adding new configs via dynamicconfigs instead
of via the legacy staticfiles and chef, before we've finished migrating all the
legacy configs over.
Reviewed By: quark-zju
Differential Revision: D22401865
fbshipit-source-id: 5c41c674d39c8113b2a40da61e020e8a33c39312
Summary:
We're seeing cases were cloning can take 10's of GB of memory because
we pend all the history information in memory. Let's flush the history info
every 10 million adds to bound the memory usage.
10 million was chosen somewhat arbitrarily, but it results in pack files that
are 800MB, which corresponds roughly with 8GB of memory usage.
This requires updating repack to be aware that a single flush could produce
multiple packs. Note, since repack writes via this same path, it may result in
repack producing multiple pack files. In the degenerate case repack could
produce the same number (or more) of pack files than was inputted. If we set the
threshold high enough I think we'll be fine though. 800MB is probably
sufficient.
Reviewed By: xavierd
Differential Revision: D22438569
fbshipit-source-id: 425d5d3b7999b81e44d1dbe1f2a4ea453ab6ca4f
Summary: Add a new `EdenApiBlocking` trait that exposes blocking versions of the `EdenApi` trait's methods, for use in non-async code.
Reviewed By: quark-zju
Differential Revision: D22305396
fbshipit-source-id: d0f3a73cad1a23a4f0892a17f18267374e63108e
Summary:
Bypass truncation-based transaction if narrow-heads is on.
The transaction abort still works logically because commit references stay
unchanged on abort.
Related EdenFS and Mononoke tests are updated. Mononoke tests probably
shouldn't rely on revlog / fncache implementation details in hg.
Reviewed By: DurhamG
Differential Revision: D22240186
fbshipit-source-id: f97efd60855467b52c9fb83e7c794ded269e9617
Summary:
With modern configs, repo is unfiltered and `ctx.children()` returns unfiltered
commits. Use the revset function `children` instead so invisible children won't
trigger auto restack.
Reviewed By: DurhamG
Differential Revision: D22421689
fbshipit-source-id: 3ec8f616c17254ee9ccfcad96673d209b9163da6
Summary: The test demostrates an issue with the current auto restack logic.
Reviewed By: DurhamG
Differential Revision: D22421690
fbshipit-source-id: e035cd3212357f24322f8eb9ec5941767ad780d9
Summary:
Make zsh_completion complete standard aliases like `checkout`.
This restores the behavior before D18463299 (54451585ce) stack.
Reviewed By: farnz
Differential Revision: D22396737
fbshipit-source-id: 745761041d6d1dec6adba2efb102e2021a01b36b
Summary:
checkunknown is quite expensive since it has to read the contents of
every untracked file, which can be 10's of thousands of non-parallel stats and
reads. For files that don't exist in the working copy, it's just wasted work to
stat for the files at all. Status can efficiently tell us what files are
unknown, so let's use that to triage most "unknown" files to normal writes
before we even get to checkunknown.
The downside of this approach is that it makes an additional call to status,
which is not cached (only non-unknown+non-ignore+non-clean status calls are
cached). We could add more caching if this is a problem.
This doesn't help the case where a user might have 10k+ untracked files due to a
ctrl+c'd checkout, but we'll improve that in a future diff.
Reviewed By: quark-zju
Differential Revision: D22366758
fbshipit-source-id: b54fec113dc162f97a35e705ed083ddd14babe55
Summary:
The test relies on Python revlog implementation details which
do not exist in the Rust revlog implementation.
Reviewed By: DurhamG
Differential Revision: D22240183
fbshipit-source-id: b245b35e561c3364618a0e199244df030cc47942
Summary:
The original test is unmaintainable. Rewrite it to test key features.
I dropped detailed tests about merge conflict / content handling.
In the future we probalby will have a clean Rust implementation of "applying
diff between X and Y to Z" which can replace various unmaintainable patch
application logic in Python. We can test that Rust library extensively and
commands will just use the clean library (ex. revert, backout)
Reviewed By: sfilipco
Differential Revision: D22240184
fbshipit-source-id: 4d6c65fe02ccc92e64c62a48f702187678973086
Summary:
`debugstrip` is an operation that depends on multiple legacy components (revlog
strip, truncate-based transaction). They are incompatible with modern configs
(no truncation, heads-based visibility, metalog-based transaction).
Avoid using it in the test.
Reviewed By: DurhamG
Differential Revision: D22240187
fbshipit-source-id: ec215d75fb766957a3d6f58e491ef815f5bedbdc
Summary:
Change by `fix-revnum.py`. Part of the tests using `hg debugstrip`,
which I'm trying to avoid.
Reviewed By: DurhamG
Differential Revision: D22240181
fbshipit-source-id: a569b712fe4b985378e5c61c000deecccefbc488
Summary:
Since tests do not use it, and we have long disabled it in production.
Let's just disable the command unconditionally.
Reviewed By: DurhamG
Differential Revision: D22368834
fbshipit-source-id: 7ebc5b07c4044b6809defc06437cda7256cb2ebf
Summary:
`hg rollback` was long disabled in production setup. It has weird behavior and
is likely incompatible with modern transaction frameworks. Remove its usage in
tests.
Reviewed By: DurhamG
Differential Revision: D22240180
fbshipit-source-id: 453684ebbc77132e09b1b717b6ad1e106dcad214
Summary:
End-users have been using visibleheads + narrowheads for a while, and hgsql
does not require any filtering, and most tests are migrated to modern configs
(visibility + narrow heads). Now it's time to consider removing the repoview
layer.
This removes complexities around `changelog.filteredrevs` and various different
`repoview` objects with caching problems (ex. I have seen that `repo` and `unfi`
have inconsistent phasecache causing they calculate phases differently and it's
quite hard to reason about confidently).
This will also make it easier to migrate to segmented changelog.
Reviewed By: DurhamG
Differential Revision: D22201084
fbshipit-source-id: 3661c26dd72a64b5005d86e164af4da5a6895649
Summary:
The test case assumed that clone would return data in order from the server.
That is not a valid assumption and Mononoke doesn't return data in order.
Reviewed By: xavierd
Differential Revision: D22364636
fbshipit-source-id: abfcbe0074a08c9a76c42d351ce5c792eb65e24f
Summary:
Enable mutation, remotenames and narrow-heads (for clients).
Some `x` commits are shown because the server repo cannot hide them.
Reviewed By: DurhamG
Differential Revision: D22200501
fbshipit-source-id: 35abe5198025598f4f5ef8807c2eaa6f3b2f2318
Summary:
D21894320 (64585a5895) tries to disable features from hgsql. However tests can
run hg commands with hgsql extension turned on or off, and that
messes up things.
Let's test the hgsql requirement and disable related features unconditionally
so we have more confident that the related features won't be turned on
accidentally.
Previously, the `initclient` helper function will create repos with the `hgsql`
requirement. Since we're using the repo requirement to disable features, disable
`hgsql` temporarily during `initclient` so that the client repos can still have
modern features like visibility and narrow-heads.
The globalrevs test is affected and temporarily disabled. It will be fixed in
follow-ups.
Reviewed By: DurhamG
Differential Revision: D22200507
fbshipit-source-id: 3715464430a9115bb00122a8dfa03b1abf9d03ab
Summary: This helps debugging some hgsql issues especially in test-globalrevs.
Reviewed By: DurhamG
Differential Revision: D22200509
fbshipit-source-id: cddf79a00951d135dee20a5e9fb3a486abad2ff7
