Summary:
Reading a line over a pipe involves reading every character of the line
individually. This is extremely inefficient and slow, which cause prefetch to
be overly slow when most of the data isn't in memcache.
Using buffered reads tries to read 4096 bytes at a time, drastically reducing
the cost of reading a missing path/node pair from memcache.
Reviewed By: ikostia
Differential Revision: D14507063
fbshipit-source-id: e0910d7a303e15fe2d3c61fe2739e6c13370058f
Summary:
As part of the mononoke write path rollout we want to be
able to dynamically block writes to a repo. This is implemented as
a table in hgsql (repo_lock) that both hg and mononoke can read, but
will only be updated by scmadmin.
Expose a function that can be used by in-process hooks to check if a repo is
locked for mercurial.
Reviewed By: quark-zju
Differential Revision: D14279169
fbshipit-source-id: f8bb4afeeeda67796cf806ab7f3fe42f4089818f
Summary: This is removal of unused code that left from D14455360
Reviewed By: ikostia
Differential Revision: D14502837
fbshipit-source-id: df1912c7997847b0628b492b3fe735d5e3d7f201
Summary:
On connections that might experience higher latency the throughput for a clone
is easy to spike, raising the chunk size results in more consistent throughput.
Given that hardware these days has plenty of memory, this change shouldn't make
any significant difference either.
Reviewed By: farnz
Differential Revision: D14502443
fbshipit-source-id: 0e90f955304e9955df0ec69b5733db7c34b09e83
Summary:
Change `ancestor` to send 24 revs to the native ancestor logic at a time,
instead of 2 revs at a time.
This makes it much faster when calculating ancestor of many commits.
Before:
quark@devvm33994 ~/fbcode/scm/hg % hg.real log -r 'ancestor(limit(draft(),6000))' -T '{node}\n' --time --hidden
ca4beab06459e016ac1b4041400029165b701afa
time: real 66.620 secs (user 44.800+0.010 sys 21.840+0.000)
After:
quark@devvm33994 ~/fbcode/scm/hg % ./hg log -r 'ancestor(limit(draft(),6000))' -T '{node}\n' --time --hidden
ca4beab06459e016ac1b4041400029165b701afa
time: real 3.470 secs (user 2.380+0.000 sys 1.000+0.020)
This also affects `merge.preferancestor`. That's a dangerous config option and
seems fine to be ignored in the revset implementation.
Reviewed By: DurhamG
Differential Revision: D14461521
fbshipit-source-id: ec00921aece0adc6aaca49e5580bff52784c4ca5
Summary:
Use "interestingmaster()" to make it easier to see how "master" gets decided.
Change the type of "master" argument taken by "smartlog" revset from string to
revset. This is more consistent with other revset functions.
Reviewed By: DurhamG
Differential Revision: D14436003
fbshipit-source-id: 5aa166b523f36672f77dc4f161ae8d64c2b50579
Summary:
Add a simple test about smartset revset. It shows "smartlog(master=revset)"
does not work.
Reviewed By: DurhamG
Differential Revision: D14436005
fbshipit-source-id: e566aa35d1cb335718a48db1314454d2dc50411b
Summary:
Similar to `interestingbookmarks()`, this exposes more smartlog logic to the
user.
Reviewed By: DurhamG
Differential Revision: D14436004
fbshipit-source-id: bd4ef1dcee8e7b29c43ce43fe6c1a3e7b5286774
Summary:
Make `heads` in `smartlog` customizable. This makes smartlog more flexible.
Instead of using the default selection, the user can choose draft branches, and
potentially pass in `interestingbookmarks()` to include bookmarks and remote
bookmarks. For example, `smartlog(.::)` shows the current branch and the public
commit it bases on.
Drop `recentdays` as it can now be expressed using `heads` directly. See the
test change.
This would also hopefully make test-fb-hgext-smartlog-hide-before.t stable,
as it no longer uses `time.time()`.
Reviewed By: DurhamG, sfilipco
Differential Revision: D14436007
fbshipit-source-id: 5e0a76e4424b01312fef02fae23a3abd74e863c6
Summary:
The old code basically selects ancestors of heads.
Rewrite the logic using revsets. Assuming we're only interested in ancestors
that are drafts, we can take advantage of `draft() & ::x` optimization.
The new logic also assumes master rev is public. Otherwise it can be slightly
different from the old logic.
The new code is much faster on my repo:
New code:
quark@devvm33994 ~/fbcode/scm/hg % ./hg log -r 'smartlog()' --hidden -T . --time | wc -c
time: real 0.630 secs (user 0.550+0.000 sys 0.030+0.000)
6716
Old code:
quark@devvm33994 ~/fbcode/scm/hg % hg.real log -r 'smartlog()' --hidden -T . --time | wc -c
time: real 5.470 secs (user 3.920+0.000 sys 1.550+0.000)
6716
This might make the ancestorcache hack (D5135746) unnecessary.
Reviewed By: DurhamG, sfilipco
Differential Revision: D14436008
fbshipit-source-id: 3c3bf47ccb67ea0e238542995009da9b9250b43b
Summary:
The `smartlog()` revset does a lot of things.
Add a new revset `interestingbookmarks()` to expose part of the smartlog features.
Reviewed By: DurhamG, sfilipco
Differential Revision: D14436006
fbshipit-source-id: 15b8d203b6547e63f8d062660ad27bdbc25b2c1c
Summary:
The code falls back to head() if there are no remotenames. We don't need that
behavior. Therefore just simplify it by always using `heads(draft())`.
Reviewed By: DurhamG
Differential Revision: D14436009
fbshipit-source-id: 25c2d245ed64a29e3e1677ededb4c2ba7b4a3ceb
Summary: D14387734 added 2 new arguments to the `httprequestpacks` function, but didn't update the callsite to pass those arguments. This diff fixes the problem.
Differential Revision: D14468592
fbshipit-source-id: 7e573838916067fc2cc12204ea1da460eb3955c8
Summary:
The test frequently fails with spurious remote errors such as:
remote: ** Unknown exception encountered with possibly-broken third-party extension treemanifest.
remote: ** Please disable treemanifest and try your action again.
remote: ** If that fixes the bug please report it to the extension author.
remote: ** Python 2.7.11 (default, Feb 5 2019, 17:19:11) [GCC 5.x 20180625 (Facebook) 5.5.0+]
remote: ** Mercurial Distributed SCM (version 4.4.2_dev)
remote: ** Extensions loaded: remotefilelog, rebase, simplecache, treemanifest, conflictinfo, debugshell, errorredirect, githelp, mergedriver, progressfile, pushrebase
remote: Traceback (most recent call last):
remote: File "<string>", line 44, in <module>
remote: File "<string>", line 42, in __run
remote: File "/usr/local/fbcode/gcc-5-glibc-2.23/lib/python2.7/runpy.py", line 162, in _run_module_as_main
remote: "__main__", fname, loader, pkg_name)
remote: File "/usr/local/fbcode/gcc-5-glibc-2.23/lib/python2.7/runpy.py", line 72, in _run_code
remote: exec code in run_globals
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/hg.py", line 42, in <module>
remote: entrypoint.run(False)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/entrypoint.py", line 80, in run
remote: dispatch.run()
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 99, in run
remote: status = (dispatch(req) or 0) & 255
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 393, in dispatch
remote: ret = _runcatch(req)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 575, in _runcatch
remote: return _callcatch(ui, _runcatchfunc)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 584, in _callcatch
remote: return scmutil.callcatch(ui, func)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/scmutil.py", line 135, in callcatch
remote: return func()
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 565, in _runcatchfunc
remote: return _dispatch(req)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 1353, in _dispatch
remote: lui, repo, cmd, fullargs, ui, options, d, cmdpats, cmdoptions
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 1053, in runcommand
remote: ret = _runcommand(ui, options, cmd, d)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 1365, in _runcommand
remote: return cmdfunc()
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/dispatch.py", line 1350, in <lambda>
remote: d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/util.py", line 1337, in check
remote: return func(*args, **kwargs)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/commands/__init__.py", line 5635, in serve
remote: s.serve_forever()
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/sshserver.py", line 107, in serve_forever
remote: while self.serve_one():
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/sshserver.py", line 139, in serve_one
remote: self.handlers[rsp.__class__](self, rsp)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/sshserver.py", line 88, in sendstream
remote: for chunk in gen:
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/bundle2.py", line 710, in getchunks
remote: self._getcorechunk(), self._compopts
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/bundle2.py", line 732, in _getcorechunk
remote: for chunk in part.getchunks(ui=self.ui):
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/bundle2.py", line 1165, in getchunks
remote: pycompat.raisewithtb(exc, tb)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/bundle2.py", line 1141, in getchunks
remote: for chunk in self._payloadchunks():
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/bundle2.py", line 1181, in _payloadchunks
remote: chunk = buff.read(preferedchunksize)
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/util.py", line 2062, in read
remote: for chunk in self.iter:
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/mercurial/util.py", line 2033, in splitbig
remote: for chunk in chunks:
remote: File "/data/users/xavierd/fbsource/fbcode/buck-out/dev/gen/scm/hg/hg#link-tree/edenscm/hgext/treemanifest/__init__.py", line 2255, in _generatepackstream
remote: for subname, subnode, subtext, x, x, x in subtrees:
remote: RuntimeError: std::exception
They don't seem to affect what the test expect (missing node on the server), so
let's disable these
Reviewed By: DurhamG
Differential Revision: D14446457
fbshipit-source-id: 0ecb114bfcdeef1a3fcfef5a927f725401f90d4e
Summary:
Currently archive is almost useless because it fetches each file
one-by-one. Let's add prefetching.
Differential Revision: D14460880
fbshipit-source-id: 1f06e1ac9d03aae3ab27d3064f9fe6141051be06
Summary:
Remotefilelog is not optimized for archive, and does many fetches. A
future test will fix this, but let's add a test first.
Differential Revision: D14460881
fbshipit-source-id: 1797193be3dfec246fabb659ffaeea3dea69b8f3
Summary: Fill in an revset example. Clarify some other places.
Reviewed By: sfilipco
Differential Revision: D14430873
fbshipit-source-id: ae4a8996f8254f1ad374effe3a3da0ce9d2306eb
Summary:
In preparation to support Mononoke clean up the features that are Mercurial
specific and Mercurial infinitepush implementation specific.
For Mononoke migration we will to write a whole new set of logic what to do if
the "infinitepush" path has been changed. So, clean up is useful before
writing this logic.
Reviewed By: singhsrb
Differential Revision: D14455360
fbshipit-source-id: d15c3a9032b4888a1aa391da34ad5e499aba9a15
Summary:
D14448505 added the functionality of supporting globalrevs/svnrevs
with fbconduit. Lets add some more tests for it. In particular, this adds tests
for the failure case.
Reviewed By: quark-zju
Differential Revision: D14461674
fbshipit-source-id: 27d52f88013f694a2cb6bf1af795eba4f5dccc93
Summary: We don't run this binary anymore, no reason to build and ship it.
Reviewed By: quark-zju
Differential Revision: D14437317
fbshipit-source-id: dd6da521783f18a2a518a7aa042be98950894e89
Summary: See the linked task for details.
Reviewed By: quark-zju
Differential Revision: D14448505
fbshipit-source-id: fc2efa71510b718c25a2cea3acf39663d280f19a
Summary:
After going over the code review for D14332967, I have decided to keep
things simple for now and only allow making commits to same target parent as
the original parent. This was already the intention with the existing code.
Therefore, this commit just further enforces the requirement.
Reviewed By: quark-zju
Differential Revision: D14422351
fbshipit-source-id: 2f786fc3596b17c5020de9906adf8f22b50be4dd
Summary:
These was probably introduced by moving to black.
The changes in the diff were generated by script.
Reviewed By: mitrandir77, singhsrb
Differential Revision: D14439667
fbshipit-source-id: 54f6e0bdcc59c1c6deb4eea46dc6f865bcd48cf8
Summary:
The code currently assumes that the target parent is the same as the
original parent by totally ignoring the original parent which can seem
surprising and more importantly, hinders supporting behaviors such as allowing
commits to a new parent. Therefore, this commit fixes the code to identify the
original and target parent separately.
Reviewed By: quark-zju
Differential Revision: D14422352
fbshipit-source-id: bc175f2fe636f9bf47a68f64c8efd52660e3b1b7
Summary:
D14183009 made commit cloud accept cloud bookmarks for hidden commits, rather
that omitting them. However, this only works for future bookmarks. If the
bookmark was already omitted, then `_checkomissions` would not recover the
situation for the same reason.
Update `_checkomissions` to also allow cloud bookmarks on hidden commits.
Reviewed By: liubov-dmitrieva
Differential Revision: D14437656
fbshipit-source-id: 2372323022a59bfd4210bc76f39b9a74872d5efe
Summary:
Now that hg_memcache_client will voluntarily exit when hg terminates, we no
longer need to wait for hg_memcache_client to finish before exiting hg.
Reviewed By: DurhamG
Differential Revision: D14396510
fbshipit-source-id: 7e73d9b70d481e58a0c47cd0f408580e6d548fd9
Summary:
This is useful for testing the `memcommit` command as the clients can
specify different destination parents than the original parents of the commit.
Differential Revision: D14410213
fbshipit-source-id: 846e0d764b9606f00aed95995c694f379457eec7
Summary:
Earlier message suggested the feature is not supported when in fact it
is allowed via a configuration.
Differential Revision: D14410214
fbshipit-source-id: 0ec2a22920417c378cf3ac596565f9d2fa5f6d5c
Summary:
Operations like `hg log -p` will inherently cause many requests to
hg_memcache_client. This will force many small packfiles to be created which
will significantly slow down future invocation of hg.
Now that `hg repack --incremental --packsonly` is fast, we can afford to run it
when mercurial exit after a prefetch operation was run.
Differential Revision: D14387735
fbshipit-source-id: 45f89f1120458c8b2471a1c55cafb6bc87263dd0
Summary:
When using packfiles, history and data are in different files, and thus it's
possible to only fetch one.
For now, besides the requests coming from contentstore and metadatastore, both
will be fetched, as the code hasn't been audited to know whether we only want
history or data.
Differential Revision: D14387734
fbshipit-source-id: 6aafd477ff486b9316458ce0e80636152db45b89
Summary:
For pushbackup it is needed to make hg rage more informative because we store
states for different paths separately anyway.
For cloud sync we will have to write some migration logic: if the remotepath
has been changed, we have to check what the server has to make sure everything
is backed up as cloud sync would expect.
Differential Revision: D14420713
fbshipit-source-id: 2046e9d7b16291a49d1bc40da5285de58017f4f2
Summary:
Corrupted packfiles, or background removal of them could cause repack to fail,
let's simply ignore these transient errors and continue repacking.
Reviewed By: DurhamG
Differential Revision: D14373901
fbshipit-source-id: afe88e89a3bd0d010459975abecb2fef7f8dff6f
Summary:
The spec for both XDG and Mercurial say that when the XDG_CONFIG_HOME variable
is not set, we should default to $HOME/.config.
Windows and macOS also have a designated config folder outside of the home directory.
The `dirs` crate provides consistent access to this folder. I see no harm in looking at config
folders across all operating systems.
Reviewed By: quark-zju
Differential Revision: D14380686
fbshipit-source-id: 5e5a9cd4694aaa49fbc526f4917dc4afdaeb9842
Summary:
quark-zju noticed in code review that `Cursor` could get into an infinite loop when
it's results would be collected into a Vec<_>. That was the motive that I
needed to update `Cursor` to transition to `State::Done` when the cursor
encounters an error. Previously I felt that users of `Cursor` would only be
empowered by having the ability to retry the failure.
Reviewed By: quark-zju
Differential Revision: D14393590
fbshipit-source-id: b3e0974ac15d62f3f17790229121c0dec3a6149e
Summary:
Fixes the below type error when pulling from a git repo using hg-git on my laptop:
File "edenscm/hgext/remotenames.py", line 225, in expull
pullremotenames(repo, remote, bookmarks)
File "edenscm/hgext/remotenames.py", line 314, in pullremotenames
path = activepath(repo.ui, remote)
File "edenscm/hgext/remotenames.py", line 1464, in activepath
rpath = _normalizeremote(remote.url)
File "edenscm/hgext/remotenames.py", line 1439, in _normalizeremote
u = util.url(remote)
File "edenscm/hgext/hggit/__init__.py", line 164, in _url
if not (path.startswith(pycompat.ossep) and ":" in path):
AttributeError: 'function' object has no attribute 'startswith'
Basically, `peer.url()` is the API, `peer._url` is a private field that does
not always exist somehow.
Besides, further remove named branches that can crash hg-git with
NotImplementedError:
File "edenscm/hgext/remotenames.py", line 225, in expull
pullremotenames(repo, remote, bookmarks)
File "edenscm/hgext/remotenames.py", line 322, in pullremotenames
for branch, nodes in remote.branchmap().iteritems():
File "edenscm/hgext/hggit/gitrepo.py", line 73, in branchmap
raise NotImplementedError
Reviewed By: DurhamG
Differential Revision: D14144462
fbshipit-source-id: 2e886c639cf6689480f84626eaf0d5ec25512ea0
Summary:
Since state files have been changed, now it is one per path, we should dump
them in a correct way
Reviewed By: markbt
Differential Revision: D14406311
fbshipit-source-id: 8d74a51e63028ec81bcf5e55ad117d3c960b4651
Summary:
To make "draft()" bounded and train users to hide unused commits manually,
change smartlog to "hide commits before <a static date>", instead of
"hide commits that are older than <a static time span>". Then we can
incrementally decrease the static date, and eventually show everything, and
force the user to think about what commits to keep or hide.
Reviewed By: singhsrb
Differential Revision: D13993584
fbshipit-source-id: 1a2b56f50d7f014a589f798cd2feaa6931e64fe3
Summary:
Subrepo is another unloved features that we don't want to support.
Aggressively remove it everywhere, instead of just turning off configs.
I didn't spend much time to split this commit so it's smaller and more friendly
to review. But it seems tests are passing.
Reviewed By: sfilipco
Differential Revision: D14220099
fbshipit-source-id: adc512a047d99cd4bafd0362e3e9b24e71defe13
Summary:
If an infinitepush bundle contains flat manifests and is served from a
treemanifest repository, it can potentially fail to send all the needed data to
the client.
Understanding the bug requires two bits of context:
1. When sending commits from a tree server to a tree client, we generally don't
send any trees because they can be fetched by the client ondemand later. The one
exception to this is for infinitepush bundles, where the trees inside the bundle
cannot be served ondemand, and therefore must be served at pull time. To do this
we check if a given manifest node exists in the repositories permanent storage,
and if it doesn't, we assume it came from an infinitepush bundle and serve it
with the pull.
2. When we lookup a manifest and fail to find a tree, our last resort is the
ondemandstore which knows how to convert a flat manifest into a tree manifest.
On the server, this is responsible for converting each of the flat bundle's
manifests into treemanifests before we serve the bundle to the client. As part
of converting the flat manifests into treemanifests, it writes the new tree
data into a pack file.
The bug is then, when serving a stack of commits, if we try to package up the
top tree first (i.e. the most recent tree), we end up converting the entire
stack from flat into trees, which inserts the bottom most trees into the
temporary pack file. Because they exist in the temporary pack file, when we
later check if they are part of the repositories store we end up finding them,
which causes us to treat them as not-infinitepush-trees which means we don't
serve them to the client.
The fix is to change the infinitepush tree-serving code to not consider the
mutable packs when checking if it should send trees.
Reviewed By: mitrandir77
Differential Revision: D14403925
fbshipit-source-id: 38043dfc49df5ff9ea2fae1d3cac341c4936509c
Summary:
When calculating bundleroots (the commits that are the common ancestors for the
infinitepush bundle), we currently include all the `nullid` parents (i.e. p2 for
most commits). This bloats the size of the list, and is unnecessary.
Reviewed By: quark-zju
Differential Revision: D14385912
fbshipit-source-id: c518b8b1aa27cff8562c2358a024b8a08ced8cba
Summary: Make it possible to override dates in commits.
Reviewed By: singhsrb
Differential Revision: D13993585
fbshipit-source-id: 59a72302d7ed0cb22f4eff84c1325e167963508c
Summary:
The feature was completed by Phil in D9816270. It's handy and can probably
reduce user support burden like: https://fb.intern.facebook.com/groups/scm/permalink/2039619916087618/
Therefore let's enable it.
Reviewed By: DurhamG
Differential Revision: D14293405
fbshipit-source-id: 54e934e0bf495c090109462e4f743d427df39380
Summary:
Some dependant libraries may only care about the serialization logic. As an
example, see D14332987 where the `hgrepo.py` only needs to depend on the
serialization. Therefore, its cleaner to extract out the serialization from the
`memcommit` data.
Reviewed By: quark-zju
Differential Revision: D14388474
fbshipit-source-id: 6f049dcc596b66b9ad72905f133529bdc9092382
Summary:
The shared code can be potentially called by clients using python 3. Therefore,
let's be compatible with python 3.
Reviewed By: quark-zju
Differential Revision: D14387005
fbshipit-source-id: 2ffb359d4d2762ffcba4a26a3ae5a7b45e89572b
