Summary:
We're seeing an issue on the hg servers where the filecache assertion
that if a value is in obj.__dict__ it's also in obj._filecache is broken. This
occurred about 10% of the time in sandcastle jobs. The diff that caused this
went in in April (D21148446 (73c5cb89de)), so it's unclear why it's only cropping up now.
This is caused by the following steps:
1. repo._bookmarks is accessed while _bookmarks is in the _filecache but not in
the __dict__
2. This causes construction of _bookmarks, before it can set it to __dict__
3. Construction of _bookmarks calls repo.invalidate(clearfilecache=True), which
deletes _bookmarks from _filecache.
4. _bookmarks construction completes, and gets set to __dict__ (but now it's
missing from _filecache, so the invariant will fail next time someone checks).
5. Someone accesses _bookmarks later, and the assertion fires.
The fix is to just not clear the filecache during bookmark construction. The
main purpose of this invalidate was to let the changelog be reloaded, and I
think that will still happen since, if there are any new commits in the
changelog, the file size and time will change, triggering a reload next time the
_filecache entry is checked.
Reviewed By: quark-zju
Differential Revision: D24182914
fbshipit-source-id: fb49137e28d9224c6617d9c84faaf2f9de363aaf
Summary:
Our stdin/stdout bytes/str manipulations caused input() to print
warnings about buffered not being supported. The only reason we do those
manipulations to handle the case in tests where the prompt answer doesn't come
from stdin, so let's just handle that case via readline instead of prompt.
This is what upstream Mercurial does.
Reviewed By: quark-zju
Differential Revision: D24122909
fbshipit-source-id: ab9d989a66d39990b688c65a1fae80bd48b0f42e
Summary:
Initial support for a backend using edeanpi.
Note this is just the first step. Most code paths are not updated to the
streaming API to get commit data, so they will error out with commit not found
errors.
Confirm that commit data can be fetched via edenapi:
$ RUST_LOG=debug lhg dbsh --config experimental.lazy-commit-data=1
In [1]: master= repo['master'].node()
In [3]: cl.inner.getcommitrawtext(master) is None
Out[3]: True
In [4]: s=cl.inner.streamcommitrawtext(repo.nodes('master~10::master'))
In [5]: it=iter(s)
...
[2020-09-25T02:09:16.793Z DEBUG hgcommits::hybrid] >> resolve_remote input=[e6c4e203b66f1416e08dc597a2d63b91e93b1466, bfb610989e9dd701e785b4a3a5998e76d9709cab, 68bbfc79602a153895b761089e9479dd8fa33351, 5366fe39ad538463abae6c648eb5150bbb79d4c7, 5ea45d8ab0f8203837ca1736f36ded4a492571b4, 722da0a32eae12de5e85078beea2ae4b7aafe4a4, 4dbe3eab10d13b30697e1762eb7b9ff3ad0cf630, 430ae91aab8028b6572ccef89f8396dafec622c4, 5abd96c5420f0d512c63e768f8cea83f1c6691c9, c84ab3412cebfade730e95a1bc5ebc9b1dd0747b, 790ed2d40e4a0b08fb22fe9b4246fec0165f8a87]
[2020-09-25T02:09:16.793Z DEBUG hgcommits::hybrid] << resolve_remote input=[e6c4e203b66f1416e08dc597a2d63b91e93b1466, bfb610989e9dd701e785b4a3a5998e76d9709cab, 68bbfc79602a153895b761089e9479dd8fa33351, 5366fe39ad538463abae6c648eb5150bbb79d4c7, 5ea45d8ab0f8203837ca1736f36ded4a492571b4, 722da0a32eae12de5e85078beea2ae4b7aafe4a4, 4dbe3eab10d13b30697e1762eb7b9ff3ad0cf630, 430ae91aab8028b6572ccef89f8396dafec622c4, 5abd96c5420f0d512c63e768f8cea83f1c6691c9, c84ab3412cebfade730e95a1bc5ebc9b1dd0747b, 790ed2d40e4a0b08fb22fe9b4246fec0165f8a87]
...
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] >> Zstore::contains id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] << Zstore::contains id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] >> Zstore::insert data_len=1010 id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
[2020-09-25T02:09:16.958Z DEBUG zstore::zstore] << Zstore::insert data_len=1010 id=3422a85c3703dd0bf0030d5d4c1bb65775adff90
...
[2020-09-25T02:09:16.959Z INFO zstore::zstore] >> Zstore::flush
[2020-09-25T02:09:16.959Z DEBUG indexedlog::log] >> Log::sync dirty_bytes=7048
[2020-09-25T02:09:16.959Z DEBUG indexedlog::log] << Log::sync dirty_bytes=7048
[2020-09-25T02:09:16.959Z INFO zstore::zstore] << Zstore::flush
In [6]: list(it)
Out[6]: [...]
The logs about `hgcommits::hybrid ... resolve_remote` shows the remote fetching
is working. The logs about `Zstore::insert` and `Zstore::flush` shows the
commit data were written to disk.
Reviewed By: sfilipco
Differential Revision: D23924148
fbshipit-source-id: a3d77999e29395ce5c603fe66412936947456534
Summary:
Add a way to specify different merge tools for interactive and non-interactive
mode.
This will be used for the default `editmerge` merge tool, which pops up the
`EDITOR` (vim) regardless of interactive mode, causing various user complains
and hangs, including `arc pull` running rebase triggering editor, or VS Code
running rebase triggering editor, and some other rebase hangs or vim errors.
Reviewed By: DurhamG
Differential Revision: D24069105
fbshipit-source-id: ec16fdc704cab6daeedb0c23d4028b4309d96d3f
Summary:
The `hg unhide` command acquired the repo lock without acquiring the wlock.
This causes locking order problems, as it calls other parts of the code that
will acquire the `wlock` (such as autopull during revset resolution) while it
is already holding the `lock`.
This can cause `hg unhide` to deadlock with other `hg` commands that acquire
`wlock` before `lock`.
Reviewed By: kulshrax
Differential Revision: D24129559
fbshipit-source-id: cf31ec661123df329f1773d2b67deb474d6476f8
Summary:
Previously, EdenAPI was using `remotefilelog.debug` to determine whether to print things like download stats. Let's give EdenAPI its own `debug` option that can be configured independently of remotefilelog.
One notable benefit of this change is that download stats will always be printed immediately after the HTTP request completes. This can help rule out network or server issues in situations where Mercurial appears to be hanging during data fetching. (e.g, if hg had downloaded all of the data but was taking a while to process it, the debug output would show this.)
Reviewed By: DurhamG
Differential Revision: D24097942
fbshipit-source-id: bf9b065e7b97fc7ffe50ab74b1b13e2fe364755c
Summary:
Previously phase calculation was done via a simple ancestor check. This
was very slow in cases that required going far back into the graph. Going a year
back could take a number of seconds.
To fix it, let's take the Rust phaseset logic and rework it to make only_both
produce an incremental public nodes set. In a later diff we can switch the
phaseset function to use this as well, but right now phaseset returns IdSet, and
that would need to be changed to Set, which may have consequences. So I'll do it
later.
Reviewed By: quark-zju
Differential Revision: D24096539
fbshipit-source-id: 5730ddd45b08cc985ecd9128c25021b6e7d7bc89
Summary:
Formats a subset of opted-in Python files in fbsource.
Black formatting was applied first, which is guaranteed
safe as the AST will not have changed during formatting.
Pyfmt was then run, which also includes import sorting.
The changes from isort were manually reviewed, and
some potentially dangerous changes were reverted,
and the directive was added to those
files. A final run of pyfmt shows no more changes to
be applied.
Reviewed By: zertosh
Differential Revision: D24101830
fbshipit-source-id: 0f2616873117a821dbc6cfb6d8e4f64f4420312b
Summary:
The bfs fetching path in Rust was broken because it directly called
getdesignatednodes. getdesignatednodes returns False if it wasn't able to
succeed, so we need to be able to fall back in that case. The _prefetchtrees
function is meant for that, so let's just call that.
Reviewed By: singhsrb
Differential Revision: D24090946
fbshipit-source-id: d16c2c8f80f690a22046385f0e95785996a62949
Summary:
Previously, only the first `__iter__` gets wrapped. With D23095468 (34df768136), the first
`__iter__` is used by the "simplify graph" feature, not the main iteration
loop rendering the graph log output, causing the prefetch feature to fail.
File "edenscm/mercurial/commands/__init__.py", line 4196, in log
return cmdutil.graphlog(ui, repo, pats, opts)
# pats = ()
File "edenscm/mercurial/cmdutil.py", line 3250, in graphlog
ui, repo, revdag, displayer, graphmod.asciiedges, getrenamed, filematcher
File "edenscm/mercurial/cmdutil.py", line 3106, in displaygraph
rustdisplaygraph(ui, repo, dag, displayer, getrenamed, filematcher, props)
File "edenscm/mercurial/cmdutil.py", line 3208, in rustdisplaygraph
for (rev, _type, ctx, parents) in dag:
File "edenscm/mercurial/graphmod.py", line 63, in dagwalker
rootnodes = cl.tonodes(revs)
# revs = <baseset- [7408158, 72057594037927936, ...]
File "edenscm/mercurial/changelog2.py", line 196, in tonodes
return self.inner.tonodes(revs)
# revs = <baseset- [7408158, 72057594037927936, ...]
File "edenscm/hgext/phabstatus.py", line 281, in next
return self.__next__()
Lift the "first time" limitation for wrapping `__iter__` to solve the problem.
Reviewed By: simpkins
Differential Revision: D24066918
fbshipit-source-id: 6bbd244e729724e5143147bde60bcb4c8ee4bc80
Summary: Since this is only used in the manifest target, fold it into it.
Reviewed By: DurhamG
Differential Revision: D24062629
fbshipit-source-id: c3241b53bde7abba8a80a2945661d1a24b7e3034
Summary: We now get progress bar output when fetching from memcache!
Reviewed By: kulshrax
Differential Revision: D24060663
fbshipit-source-id: ff5efa08bced2dac12f1e16c4a55fbc37fbc0837
Summary:
This is no longer about datapack, but only about parsing manifest entries, thus
renaming.
Reviewed By: DurhamG
Differential Revision: D24062634
fbshipit-source-id: 5c52b784d20437e87012dd4bc6cb13d879da9cb9
Summary: The code doesn't use anything from libmpatch, we don't need to depend on it.
Reviewed By: DurhamG
Differential Revision: D24055084
fbshipit-source-id: 0f7bac73f1b711da4395e25619577a0a2e0ca959
Summary: These are unused, no need to keep the code around.
Reviewed By: DurhamG
Differential Revision: D24055085
fbshipit-source-id: 6246d746983a575c051ddcb51ae02582a764a814
Summary: This is unused, no need to keep it around.
Reviewed By: DurhamG
Differential Revision: D24054164
fbshipit-source-id: 161b294eb952c6b4584aa0d49d8ff46cd63ee30f
Summary:
This code is effectively unused. The only bit still relevant is that EdenFS
still depends on the Manifest class to parse a manifest.
Reviewed By: DurhamG
Differential Revision: D24037723
fbshipit-source-id: 901ae2ffc8960a95ec655a2e14d79afb8d32dcab
Summary: This is unused, let's remove it.
Reviewed By: DurhamG
Differential Revision: D24037722
fbshipit-source-id: bc8a272809cb1f20f54d651a39ee42ff57169534
Summary:
The 'diff()' template needs to be able to output non-utf8 patches.
To enable this, let's change the templater to output bytes instead of strings.
Note, the templater supports json output, which requires utf8 encoded values. In
that situation we escape any non-utf8 binary and clients would have to be aware
and unescape them.
Reviewed By: quark-zju
Differential Revision: D24029105
fbshipit-source-id: aa011696c0d97281b91c6871c116037d76c178e0
Summary:
This is unused code. It broke a test while I was doing python 3 work,
so let's just delete this.
Reviewed By: quark-zju
Differential Revision: D24063953
fbshipit-source-id: 323b323ff2b9717c9c16eed54adac467a321e2ae
Summary:
From just scanning through the code I identified a couple more spots
that use "" when they should use b"". I also noticed that
_parsecompressedrevision was producing memoryviews instead of bytes, so it's "u"
check would always fail.
Reviewed By: quark-zju
Differential Revision: D24037437
fbshipit-source-id: b4c589e30b5d35a1bcee16e0d6aa37d04f30129d
Summary:
The eden extension was hard coding empty string as the path when
reading from the store. This works fine when the data is available in a local
store (since most data stores don't actually use the path component), but
doesn't work when it needs to go to the server.
This wasn't an issue historically because the above fetch did the fetch for it.
A recent code change caused the above fetch to throw an error though, which was
then eaten, which caused the get code path to try to perform the fetch with an
empty path.
Reviewed By: krallin
Differential Revision: D24020467
fbshipit-source-id: cad717618e947fbdca978b45ce2181d46422d456
Summary:
Check a few backup files for `hg cloud sl` command information.
I added a comment explaining why we would need this.
At the moment some commands like for example `hg cloud switch` or the `hg pull -r` goes through mononoke backend and they will update a different backup cache file comparing with the `hg cloud sync` command that goes through the mercurial backend (write path).
As a result, `hg sl` command displays that some commits haven't been backed up.
Reviewed By: markbt
Differential Revision: D24046449
fbshipit-source-id: 56d924bf641833231cda6e6701e67f63c8453894
Summary:
All the current workspaces should be already populated because the release has
been running for a while. Remove the migration code and also fix a bug with
string to boolean conversion.
Reviewed By: mitrandir77
Differential Revision: D24045272
fbshipit-source-id: 347f0f46d343a13fc1d9f762f912af364813a66f
Summary: Now that we progress bars in Rust, add them to the EdenAPI client bindings and remove any existing progress bars around callsites in the Python code.
Reviewed By: quark-zju
Differential Revision: D24037797
fbshipit-source-id: eb26ccaae35ab23eb76f6f2b2be575a22e1f1e53
Summary: Larges files are chunked and the initial buffer should be bytes.
Reviewed By: quark-zju
Differential Revision: D24034645
fbshipit-source-id: 98156b1901182b345baaeb82c71faeb3cc57b131
Summary: Log changelog backend types so we can filter commands by type.
Reviewed By: DurhamG
Differential Revision: D24022284
fbshipit-source-id: c402aea0ce3bd20d0f310fea167f24cb1b7a3ae6
Summary:
The protocol for getpack is length-prefixed. However, we currently advertise
the number of characters in filenames instead of their byte length. So, the
lengths we send don't necessarily correspond to the amount of data we send.
Indeed, if a filename contains multibyte characters, we'll advertise a lower
byte count than what we actually end up sending. This results in the last
byte(s) of the filename being interpreted by Mononoke as the start of another
piece of data, and eventually causes Mononoke to hang as it waits for more data
that the client will never send.
This fixes that bug in reading, and also fixes an identical instance of the bug
on the server side. I also double checked the gettreepack code, which AFAICT
doesn't have this bug.
Reviewed By: ahornby
Differential Revision: D24013599
fbshipit-source-id: af716f2bf9c02d312c0c8d2f449988e8f8858ab8
Summary:
The `hg cloud switch` command could be nicely used to debug other people workspaces by the Source Control Team.
Sometimes broken workspaces. This option would allow us to switch back from a broken workspace to our original workspace.
Mostly useful for the Source Control Team
Reviewed By: markbt
Differential Revision: D24014167
fbshipit-source-id: f2116cc13897149c8ac79790a31ebcce1f18a260
Summary:
This parameter is gone. Let's stop using it. For now we can achieve the
same result by setting the treemanifest config value. In the long term we'll
probably get rid of depth in favor of smarter algorithms, like bfs traversals.
Reviewed By: genevievehelsel
Differential Revision: D23971898
fbshipit-source-id: cabcf0c088c95557edfe07ae85ce7d07e55a3082
Summary: This just adds some explanation of `-r` to the `hg sparse list` docstring.
Reviewed By: markbt
Differential Revision: D23961027
fbshipit-source-id: 64ab406b07fe5d66fd53d4e520935aad3b0b351b
Summary:
`arc install-certificate` is going away, so let's reference the replacement
Created from Diffusion's 'Open in Editor' feature.
Differential Revision: D23948479
fbshipit-source-id: e629458ed2be1e1c09d9674935e3bcc890b05ad9
Summary:
Now that public phases are separated from draft, hidden behind the
ispublic() api, and no consumers require the entire set anymore, let's make them
lazy by testing commits against a lazy ancestor collection.
Reviewed By: quark-zju
Differential Revision: D23921326
fbshipit-source-id: 7c53345f41ff89d2d0ec11b1c09bbb2f89529c21
Summary:
Previously, _genrevancestor applied it's cutfunc function (which is
usually used to bound an ancestor traversal from processing public nodes) when
producing the parents of the current rev. In the walkrevtree function this means
we have to invoke cutfunc before we enqueue new parents onto the heap. This can
mean we have to process very old commits, even though walkrevtree might never
get to them.
Let's move the cutfunc to be at the deque stage instead of enque. When we make
public phases lazy, this will let us avoid looking up really old commits and
speed things up a lot.
Reviewed By: quark-zju
Differential Revision: D23921324
fbshipit-source-id: 8b520e9184441b9bd7581cdb71414f406cf9b544
Summary:
Unfortunately the current add operator implementation for nameset sorts
the result, which causes lazy namesets to be evaluated. Let's change the phase
revset to avoid that operator for now.
In a future diff we'll make publicrevs lazy, which will require this change to
be performant.
Reviewed By: quark-zju
Differential Revision: D23921325
fbshipit-source-id: c3477c30c1c5061b641c14b8aa73d07da6f7f398
Summary:
In a future diff we'll be changing the way public phase computation
works. Let's start by moving the draft phase computation out to belong to
draftrevs(), so then we can change publicrevs() without worrying about draft. In
the process, let's simplify draftrevs to just be an only() query.
Reviewed By: quark-zju
Differential Revision: D23921327
fbshipit-source-id: 255eab4b796bf7bc4467544cf8231b3ed98c08c8
Summary:
In an upcoming diff we'll be splitting draft and public calculations
apart when in headbased repositories. To make this cleaner, let's stop using
loadphaserevs and start using draftrevs and publicrevs accessors.
Reviewed By: quark-zju
Differential Revision: D23921328
fbshipit-source-id: 26a2a6fdc1201416bf61c7d32e3ede62d9579546
Summary:
Fix the `hg mv` and `hg cp` commands to wait for the lock if the repository is
locked, instead of immediately failing.
The current behavior is quite annoying as `hg mv` can easily fail due to
commitcloud sync commands running in the background and holding the lock.
I can't see any good reason why this command should not wait for the lock to
be held. The current behavior of failing when the lock is held dates back to
2006, in rFBSf5b6e7d2f591, when locking was first added to this command.
Reviewed By: DurhamG
Differential Revision: D23896302
fbshipit-source-id: 29bfeb8fd356abd31b6e1eb30c8f211a31fb044b
Summary:
The remotefilelog.indexedlog{data,history}store configs are no longer in use,
so we shouldn't use them as a way to test whether the indexedlog are present.
Reviewed By: quark-zju
Differential Revision: D23918133
fbshipit-source-id: 902a369f5e1774b8e2fa1e4b3280fbb7cc377775
Summary: Add appropriate EdenAPI calls to allow for HTTP tree fetching in `treemanifest`. Enabling `remotefilelog.http` essentially reroutes `_prefetch` and `getdesignatednodes` to their HTTP equivalents.
Reviewed By: DurhamG
Differential Revision: D23877319
fbshipit-source-id: 8a71934b47d07d2655fa46c103a14fb99e2f7b1f
Summary:
Cast `names` and `nodes` to lists. The reason that they are sets is deduplication, but otherwise the code doesn't rely on them being sets (and in fact casts them to lists at multiple points later).
The main motivation for this is to allow these to be passed to Rust code later. The Rust bindings make a distinction between sequence types and unordered types, so passing a set in place of a list would result in a type error.
Reviewed By: DurhamG
Differential Revision: D23893108
fbshipit-source-id: 9ce2addb824867bcb2d24ba14c589b8791a156e8
Summary: These concatenated strings and bytes and needed to be fixed.
Reviewed By: kulshrax
Differential Revision: D23907301
fbshipit-source-id: 0008d7d54469266ecbae8ddaaa7625820f62cb7e
Summary: Linkrevs are on their way out. Let's switch to linknodes.
Reviewed By: quark-zju
Differential Revision: D23765176
fbshipit-source-id: 0dc1e0db11d732ce1edd24d863f32f08a5a5ce42
Summary:
The rust contentstore doesn't allow runtime manipulation of the list of
stores, which is required in order to insert the bundle store into the store.
Let's continue using the old python union store in these cases. This still let's
us delete the python pack code later, but we'll have to keep around the python
union store until we come up with a better solution.
Reviewed By: quark-zju
Differential Revision: D23689630
fbshipit-source-id: 0c60e6f268e81804149afa24409f64b5303e1e34
Summary:
The high level prefetch trees API had a depth parameter, but the Rust
prefetch path doesn't support that. In the long run we probably want to get rid
of the depth parameter (or make it more useful), so for now let's get rid of it
from the function signature. You can still set the depth via config, and the few
places that needed depth are changed to use the config.
Reviewed By: quark-zju
Differential Revision: D23772384
fbshipit-source-id: a037d7207d4076a47368366ef7fd2dc1cfbf5cfb
Summary:
Treemanifest needs to be able to write to the shared stores from paths
other than just prefetch (like when it receives certain trees via a standard
pull). To make this possible we need to expose the Rust shared mutable stores.
This will also make just general integration with Python cleaner.
In the future we can get rid of the non-prefetch download paths and remove this.
Reviewed By: quark-zju
Differential Revision: D23772385
fbshipit-source-id: c1e67e3d21b354b85895dba8d82a7a9f0ffc5d73
