Summary:
The goal is to improve backup experience, make repo after recovery to behave as it was for the most popular commands.
This is only if backup is backed by cloud sync because we have more meta information in the repo.
the commands:
`hg diff --since-last-submit`
`hg log -r 'lastsubmitted(.)'`
`hg unamend`
for example, are not working for fresh repos that are recloned because the repo will only contains the latests snapshot of commits and also all obsmarker history but not obsoleted commits.
those are popular commands, so let's make them work.
the standard error there is 'unknown revision' and it is very confusing for users
there is no reason not to pull a remote obsoleted revision into the repo if needed, and there is no reason to ask user to do that manually.
the obstore is still lazy loaded, so shouldn't be perf problems
this diff supports good range of commands, so should make experience for our backup use case users significantly better.
pulling the hidden revisions is also relatively fast and doesn't affect commit cloud state of the repo in any way
Reviewed By: quark-zju
Differential Revision: D9437518
fbshipit-source-id: 01065c642aa9a194f2d321f03c1bd747f57c74b1
Summary:
Update the `remotenames()` revset to check if the repository in question
actually supports remotenames. Otherwise the code will crash trying to access
`repo._remotenames` if it does not exist.
The `_findrecenttree()` function in the `treemanifest` extension evaluates the
`remotenames()` revset if the remotenames extension is loaded. This can cause
it to evaluate this revset even if the repository in question was initialized
before the remotenames extension was loaded, and therefore does not support
remotenames.
Reviewed By: quark-zju, strager
Differential Revision: D9486916
fbshipit-source-id: 4a656f5f56330b4038afdb2f88e35b7c97bbc81e
Summary: This was a review comment on D8003628. bolinfest is on PTO, so I fix it.
Reviewed By: wez
Differential Revision: D9444972
fbshipit-source-id: ecbaa6b7f8867e76274e5bd9700dd77f73d9545c
Summary:
as it states in the comments, we ideally wouldn't get
in this state, but if we do, let's try to recover and continue
a bit more gracefully.
Reviewed By: quark-zju
Differential Revision: D9445209
fbshipit-source-id: a08e605f37f85a9296f71d595f950bbf110e204d
Summary:
Sync up with pywatchman changes; some of these are rather
old now :-/
Reviewed By: quark-zju
Differential Revision: D9440040
fbshipit-source-id: a19b53bb21ffcb8e2566bfa7461a636bf1b739aa
Summary:
We'd like to enable `lfs.url` on hg servers to make blob verification work
without downloading actual blobs. Previously, the LFS local blob store is
always backed by the local filesystem. D6699099 added a simple in-memory store
for another purpose. That in-memory local store fits the
"avoid downloading to disk" purpose. So let's add a config option to use it.
Reviewed By: zhh95
Differential Revision: D9478744
fbshipit-source-id: 69b04cd719cedb77f01b90b4666676c8f38584a1
Summary:
There is a user report that a command was strangely forked into 2 processes:
The blackbox looks like:
.... (no other "commit" command around this time)
2018/08/21 10:57:03 c00b (33973)> commit
2018/08/21 10:58:59 c00b (34191)> sl
2018/08/21 10:58:59 c00b (34159)> pythonhook-pretxncommit: hgext.checkmessagehook.checkcommitmessage finished in 0.00 seconds
2018/08/21 10:58:59 7b7a (34159)> commit exited 0 after 116.00 seconds
2018/08/21 10:58:59 c00b (34191)> sl exited 0 after 0.39 seconds
2018/08/21 10:58:59 7b7a (33973)> commit exited -1 after 116.17 seconds
It appears that pid 34159 was a fork of pid 33973. The fork and the original
process both end and mess up with the transaction framework.
The user did pressed Ctrl+C. Looking at the code, I suspect extutil suffers
a same issue that I fixed for upstream worker.py [1]. So let's fix extutil
too.
[1]: 3176378509
Reviewed By: singhsrb
Differential Revision: D9438093
fbshipit-source-id: 66d72d5d1d2be89f078f391eac9bd7b21b4c176c
Summary:
the smartlog message notification about "cloud sync is running" was not working because srcrepo should be used instead of repo.
commit cloud is srcrepo based feature
Reviewed By: ikostia
Differential Revision: D9479775
fbshipit-source-id: a41852b86a8dea7c9d01ea0f31cecdcb0a2bc229
Summary:
The verification also runs client-side and that can cause issues with complex
treemanifest setups.
Therefore turn the verification off by default. It would be enabled on the
server-side.
Reviewed By: phillco
Differential Revision: D9478745
fbshipit-source-id: 299b9180c661023d8e2358eedeae7b5c7d36ab79
Summary:
The output is still long. So let's use a pager.
This is better than `hg rage -p --pager=1` because the progress bar is
preserved.
Reviewed By: markbt
Differential Revision: D9443603
fbshipit-source-id: e2dbf9fa4de378a150e51bb2f3022463d7a344fd
Summary:
The filesystem type field is useful to tell whether it's an eden repo or not
quickly.
Reviewed By: markbt
Differential Revision: D9443608
fbshipit-source-id: 60ca660ee297ca00e36a46d69c8823687fd2c576
Summary:
All configs could be hundreds of lines. We're only interested in things that
are overridden. So let's limit the config to only include those chef overrides.
Reviewed By: markbt
Differential Revision: D9443604
fbshipit-source-id: e9903e79568bae99d03e19b5045b37336bc02095
Summary:
Show progress bar for each step so people know what information takes long to
collect.
Avoid using the "paste" word since with `-p` there is no paste.
As we're here, uncapitalize the message so it's consistent with the rest of the
code.
Reviewed By: markbt
Differential Revision: D9443605
fbshipit-source-id: 74672a1cf52d44289e438abeb1ca10019e37b531
Summary:
In the later diffs I'll add some more functionality there, not strictly
related to encoding paths.
Reviewed By: quark-zju
Differential Revision: D9441427
fbshipit-source-id: 069ab30a24761038fa2c1a4f180bbc0699d38ef9
Summary:
If a user uses `hg undo` to undo a `commit` or `amend` operation, this discards
the changes. If they wanted to keep them in the working copy, they should have
used `hg uncommit` or `hg unamend`.
Provide a hint telling the user how to get back their changes, and letting them
know that `hg uncommit` and `hg unamend` exist.
Reviewed By: quark-zju
Differential Revision: D9346061
fbshipit-source-id: 300b1ac851a12ccf2432ba2c11fceac619667701
Summary:
In these threads, our users are working around what is
either an invalid state transition on the hg side and/or slow
processing of the state transitions on the watchman side.
Since the state updates are purely advisory, we do not want to
block users in this case.
This doesn't address the root cause of this problem, it just stops
it from breaking our users.
Reviewed By: quark-zju
Differential Revision: D9437674
fbshipit-source-id: a2b0514a71e31ee7c92df1c24031b12a349822b5
Summary:
this is an option to take into consideration only specified heads that run cloud sync
this is helpful to debug issues with individual stacks on push part of cloud sync
Reviewed By: quark-zju
Differential Revision: D9349821
fbshipit-source-id: c5220be3b30cfc300bf62bdcfc472368ce74104e
Summary:
about 3-6 seconds speed up for cloud sync after hg hide (no ssh any more)
also to sync hg hide on another machine
Reviewed By: quark-zju
Differential Revision: D9403921
fbshipit-source-id: 56c1475dc104f6b13483db4bdb40dabd09a6400d
Summary:
backup token in the secrets tool to easier recover it when move to another dev server
secrets tool is a distributed keychain
this will allow `cloud sync` and `cloud rejoin` to work even if .commitcloudrc file doesn't exists
we don't move to storing it in the secret tool only and read it from there each time because of overhead
Reviewed By: quark-zju
Differential Revision: D9399916
fbshipit-source-id: 571f31b323f09259f7e955780f662aad2b4a5c3e
Summary:
Similar to rebase, do not use unsafe strip for histedit.
This makes `histedit --abort` safer, faster, undo-able, and also solves
potential crashes using real strip with clindex.
Note `test-histedit-obsolete.t` shows a case that is suboptimal. It'll be fixed
by the next commit.
Reviewed By: phillco
Differential Revision: D9236662
fbshipit-source-id: e436b0aa9b4a1b155dcfb111c5618371b68232eb
Summary:
per @[1070541862:quark]'s suggestion, I attempt to add a verification hook that checks LFS blobs were uploaded before push.
The code execution path looks like this:
`command.py: push()` -->
`exchange.py: push()` -->
......
`exchange.py: _pushbundle2()`
`exchange.py: _pushcheckoutgoing()`
......
And I insert the hook in `exchange.py: _pushcheckoutgoing()`
Reviewed By: quark-zju
Differential Revision: D9212199
fbshipit-source-id: 98e016da81f9df85731771b7d486d03848624bcb
Summary:
This change fixes two problems in pushrebase recording, that will be used to test mononoke pushrebase:
- record correct `repo_id`, it used to be always 0
- `ontorev` truncation, as we store rev as hex string we need 40 bytes instead of 20
Reviewed By: ikostia
Differential Revision: D9398009
fbshipit-source-id: 91add2d75fbb9acf32b99dc952c467d44c2ccbc9
Summary: Packing loose files for local data into `.hg/store/packs`. Existing pack files for local data can also be repacked.
Reviewed By: DurhamG
Differential Revision: D9181106
fbshipit-source-id: 4d044e97178b4263b87404357445b43eb99017b3
Summary:
The real strip was used on purpose [1] as a workaround of the old obsstore
design where commits cannot be revived.
Since reviving commits is now possible. Let's avoid dangerous strip here.
This makes `rebase --abort` safer, faster, undo-able, and also solves an
issue where clindex crashes with real strip reported by @[839419353:jeroenv] at
https://fburl.com/uqfglmu5.
[1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-March/095572.html
Reviewed By: DurhamG
Differential Revision: D9236658
fbshipit-source-id: 9ae7089a8991d8d21ddc5e107c15b8374c7d7339
Summary: No longer need to check inhibit.
Reviewed By: DurhamG
Differential Revision: D9236656
fbshipit-source-id: d05ec2ed2b2314ef7608c54df4f1653ca8b1b53d
Summary:
Reviving commits is an essential feature. So move it to core.
Most test changes are caused by the auto-date-bump behavior, which is needed to
revive commits automatically.
Note the existing code is not fully ready for the change. For example,
`precursors.get(node)`, `successors.get(node)` do not filter out markers that
are suppressed. For example, some code paths treat node as obsoleted if
`precursors.get(node)` is non-empty. That's no longer true. markbt's
planned visibility change might clean up this area a bit.
Regarding on tests, most changes are because of the "auto bump date" feature.
The graphlog change in `test-obsmarker-template.t` is because it creates a
cycle, which behaves differently in the new code. Half of obsmarker exchange
tests break. Given the fact that we do not use and will probably rewrite the
exchange algorithm, related tests are deleted, including
`test-obsolete-distributed.t`.
Reviewed By: DurhamG
Differential Revision: D9236661
fbshipit-source-id: 85b983f8bd46dece908c05f56bea2abbc8ffbaf6
Summary: This avoids histedit to enter an infinite loop with a later change.
Reviewed By: DurhamG
Differential Revision: D9236659
fbshipit-source-id: cdfc52e1ab263ee6c8d1753ab9bda217510a1dd3
Summary:
The obsmarker does not make much sense on the server repo since the precursor
does not exist. The client can still receive them.
This makes sure even if we turn obsstore on by default, the server won't create
markers.
Reviewed By: DurhamG
Differential Revision: D9236657
fbshipit-source-id: 32cce8c6ce6bd53ce92e17d1e7b048e089b43fe6
Summary: This step could take a long time. Therefore add a progress bar.
Reviewed By: DurhamG
Differential Revision: D9375200
fbshipit-source-id: f7eb3d3bb750b873df90eb950098a743e24c6440
Summary:
We already have the config set to make version one used in all our main
repositories, but it wasn't being used in all of our tests. Let's change
mutablehistorypack to default to creating version 1 historypacks now.
Reviewed By: quark-zju
Differential Revision: D9367017
fbshipit-source-id: 7d4055455cd8f8e6b2378c5f3a9993b2a40221a8
Summary:
Adds a format.userusthistorypack config option for using the rust code
path instead of the python one.
Reviewed By: singhsrb
Differential Revision: D9367478
fbshipit-source-id: 72c8ba56daf863d2d02e1aa3b4aeb55b09dfa6d4
Summary:
The python getancestors implementation expects a known optional
argument. Unfortunately I can't get rust cpython to support optional arguments,
so for now we make it required.
Reviewed By: singhsrb
Differential Revision: D9367480
fbshipit-source-id: 47dd74e6cbd4fe5a774ebf0949e2114bae15844d
Summary:
We can have situations where using the repository name from the
`hgsql.reponame` configuration as key for `globalrevs` data in the database can
lead to issues. As an example, for some repo `x`, we usually have a
`x-from-svn` Mercurial repository which gets the commits from svn and then
repository `x` pulls that data from `x-from-svn`. In such a setup, the
`hgsql.reponame` is usually set to `x-from-svn` for the corresponding
repository and similarly for the repository `x`. For the purpose of getting rid
of the backing Subversion repository, we will need to use the `globalrevs`
extension in the `x-from-svn` repository and it makes sense for us to have the
repository name as `x` from the beginning instead of `x-from-svn` from a
`globalrevs` perspective.
This commit just allows specifying a custom repository name to help with
aforementioned situation.
Reviewed By: quark-zju
Differential Revision: D9372695
fbshipit-source-id: 080e08a4a3ab3a4ba7a213e5177e40deb5dd5d88
Summary: This is helpful to check whether a host is recently restarted or not.
Reviewed By: singhsrb
Differential Revision: D9368886
fbshipit-source-id: 8529219a5e4958cd673420db09b3ac7c714df279
Summary:
For some arcrc formats, hg commands prefer a deprecated blob in the .arcrc file which does not contain an oauth token, and only contains a phabricator cert (which is deprecated in favor of oauth)
This updates the logic to look for the phabricator.intern.facebook.com blob in the .arcrc to get an oauth token.
Reviewed By: mitrandir77
Differential Revision: D9351851
fbshipit-source-id: 11efe30b077480caa7b147c45a8d0eb78189277e
Summary:
if there is a lot to push, it is worth to check with the server first
currently there is no any checks inside hg push implementation for commit cloud commits
hopefully it will be implemented in mononoke (heads and known wire proto request) and this logic will not be needed. Client and server will 'discuss' and figure out the delta what should be pushed and therefore any filtration in cloud sync will not be needed.
Reviewed By: quark-zju
Differential Revision: D9345385
fbshipit-source-id: 01a345905597af215dbf9b7609272491518abdf8
Summary:
We didn't expect output like `Binary file some/path/to/file matches`, so we'd emit an ugly stack trace.
This diff checks for that case specifically, and also adds a reasonable fallback: any lines that fail to parse
are simply passed through.
Reviewed By: quark-zju
Differential Revision: D9351302
fbshipit-source-id: 412a4c21a7c941d24a0d26d9c89cdf62a42c52f1
Summary:
Adds the cpython bindings that will let us call HistoryPack from rust
code.
Reviewed By: markbt
Differential Revision: D9273267
fbshipit-source-id: a29c97607d428255784c6508773cf14e283f287c
Summary:
Implements the python get_node_info function on top of an existing
HistoryStore.
Reviewed By: markbt
Differential Revision: D9273266
fbshipit-source-id: c29ce2636394a6f46fc470508d8c789873850f74
Summary:
Implements the python get_missing function on top of an existing
HistoryStore.
Reviewed By: markbt
Differential Revision: D9273262
fbshipit-source-id: ca6e123e670a57e740df83fbcc0a49582a6be3e2
Summary:
Implements the python get_ancestors function on top of an existing
HistoryStore.
Reviewed By: markbt
Differential Revision: D9273263
fbshipit-source-id: 7c369a7681161cc93ffdda860081ae306ce3e06b
Summary:
Declares the HistoryStorePyExt trait and implements it for any class
that implements HistoryStore. In future diffs we'll implement each of the
functions, then call the functions from a rust cpython declared historypack
class.
Reviewed By: markbt
Differential Revision: D9273265
fbshipit-source-id: 8f2dabc54072076600a5cc22a814485df9035211
Summary:
In an upcoming diff we'll be adding HistoryStore python extension
methods, so let's move all the existing ones to their own files to keep things
clean.
Reviewed By: markbt
Differential Revision: D9273268
fbshipit-source-id: 56be3e3130c2c95d80510915d0d1a14e2fddc783
Summary: This information has to be checked at repo init time.
Reviewed By: DurhamG
Differential Revision: D9328451
fbshipit-source-id: 042e95affb66f902b6afa4e452ebc5156ff06a33
Summary:
While I cannot find anything obviously wrong, let's do extra sanity checks to
make sure we won't return bogus results:
- In `gitlookup`, do not fallback to original lookup function if the key starts
with special `_gitlookup_` prefix.
- In `gitrevset`, do not consider a lookup successful if the result has a
length other than 20.
Also correct the naming "hex" -> "bin" in gitrevset.py.
Reviewed By: DurhamG
Differential Revision: D9328277
fbshipit-source-id: a65c5a474922649aad8ec6da5e79f4a8b3125d9e
Summary:
This was added by D7067121 by @[1093711665:mitrandir] before @[1070541862:quark] landed the copying of obsmarkers in D7121487. The copied obsmarker means we can rebase obsolete commits correctly now, so don't exclude them.
This fixes the restack test added in the previous commit.
Reviewed By: quark-zju
Differential Revision: D8908156
fbshipit-source-id: 229c8099cbd3e7559e106fd6abb653b387f12469
