Commit Graph

477 Commits

Author SHA1 Message Date
Mateusz Kwapich
33cf9ba8ad ignnore draft versions if there's newer attached version
Summary:
a "local changes" was visible in the smartlog even instantly after "jf submit"
because hg was comparing the has in the repo with the latest draft version.

https://fb.workplace.com/groups/scm/permalink/2115574591825483/

Reviewed By: katherinez

Differential Revision: D15098057

fbshipit-source-id: 5ad8e87a802a21ccbb1534c189d4e9de78e29fe7
2019-04-29 03:03:18 -07:00
Durham Goode
42c9625126 hgsubversion: fix subvertpy bindings to allow replacing directories
Summary:
The existing bindings had a bug where if you replace a directory with a
file or symlink in a single commit, it would tell subversion to add the file
before it told it to delete the directory. This resulted in an error from
subversion and meant we couldn't actually push these commits.

Reviewed By: quark-zju

Differential Revision: D15066581

fbshipit-source-id: 1a1db0fa88cd3456fde45a4aec5d4df01837474d
2019-04-26 17:14:14 -07:00
Xavier Deguillard
00506c11d5 treestate: replace error-chain with failure
Summary:
The error-chain crate is un-maintained and triggers warnings when compiling
with new versions of Rust. Let's use the failure crate instead to be consistent
with the other crates.

Reviewed By: singhsrb

Differential Revision: D15106306

fbshipit-source-id: 8edcf9f9aaf4b6e2d5f214b26fed3e72d4f3acd1
2019-04-26 16:23:35 -07:00
Durham Goode
28f410c912 pushrebase: don't create transaction during pushkey
Summary:
D14185380 made it so pushrebase will always force pushkey to get a
transaction. This has the side effect of making infinitepush bundles take the
lock and fire transactions. They should do neither.

Let's instead request the transaction at bookmark edit time.

Reviewed By: singhsrb

Differential Revision: D15094917

fbshipit-source-id: 6573447a7ba61b1853a37eacb1b3e767abb3f27f
2019-04-26 15:42:27 -07:00
Durham Goode
cec671ce4c configs: backup pinned and unserializable configs during overrides
Summary:
When using ui.configoverrides, it was previously just backing up
self._rcfg. It turns out that calling ui.setconfig (which is called as part of
configoverrides) will mark the given config as pinned and potentially
unserializable. If a config is pinned, that means we will never again read a
value into it, so when we open a bundlerepo and attempt to load the repo
specific hgrc into it, any pinned values simply don't get written.

So there are two bugs here:
1. overriding a config should not result in the config key being considered
pinned forever.
2. Cloning a ui should reset the pinned list since we don't actually clone any
of the pinned data into the new ui

Either of these alone would fix our problem with bundlerepos, but both are
necessary for general correctness.

Reviewed By: ikostia

Differential Revision: D15074037

fbshipit-source-id: 37f67f13f1d2975299be97d796b52ec2b97bca24
2019-04-26 12:07:42 -07:00
Mark Thomas
ccab681822 visibility: support undo after split
Summary:
Undo removes visibility of commits one by one, starting at the bottom of the
stack.  This doesn't work, as the bottom of the stack is kept visible by the
commits above it.

Remove them all in one go.

Reviewed By: mitrandir77

Differential Revision: D15079353

fbshipit-source-id: 52335b6dd1bd91c7f87d1a8ee2dbefa8aff7d24b
2019-04-26 01:07:30 -07:00
Mark Thomas
af2e53abb0 visibility: add shelve support
Summary:
Shelve uses hidden commits to store shelved changes.  These need to be made
invisible, too.

Differential Revision: D15079352

fbshipit-source-id: 6063270d18df81d9b4af7823542a38c5feb45e3a
2019-04-26 01:07:30 -07:00
Durham Goode
984af7d7fe push: avoid vanilla push when pushing to svn/git
Summary:
pushrebase, infinitepush, remotenames, hgsubversion, and hggit wrap the
push command in very complicated ways. For instance, if hgsubversion push runs,
it will never call the original push command, and therefore if its wrapper gets
called first none of the other code gets executed. Unfortunately the opposite
can also be true, where the other extensions may make decisions that ultimately
prevent the hgsubversion code from running. I believe this is the cause of our
recent issue where someone pushed directly to hg from an svn backed repo.

The real fix is to clean up our mess of push commands, but for now we can
at least prevent remotenames from avoiding the hgsubversion push path entirely.

I believe this used to work more reliably before D14153207. In that diff it
changed the behavior from passing "None" as the destination (which is what was
passed in originally) to passing the actual url. The remotename code has some
logic that says "if the passed in destination is None, default, or default-push,
check if its svn/git and call orig". By passing in the actual url, this logic is
bypassed and we can head straight to the remotenames push code which may call
exchange.push and begin a vanilla push.

The easy fix is to just validate the passed in url directly if it exists and
call orig if its svn or git.

Reviewed By: quark-zju

Differential Revision: D15075782

fbshipit-source-id: 10c9c1c5669ff647a03987a2de5eaf1e3b474ce0
2019-04-25 22:06:16 -07:00
Xavier Deguillard
06151c9633 remotefilelog: properly drop packs before repacking
Summary:
After trying to enable `remotefilelog.fetchpacks` on windows, I realized that
the inline repack would always fail to remove the old pack files, leaving
temporary files around. The reason for this is that the packfiles are simply
not being garbage collected, even though `self.packs` is properly being
reinitialized. For some reason, that isn't enough to drop the refcount on the
old `self.packs` and therefore the packfiles aren't being released.

Instead of re-initializing `self.packs`, we can simply clear them, which will
release every one properly. In some cases though, it looks like one of the
packfile still isn't being released properly, I haven't figured out why, but at
least only one temporary file will be left around (until a repack 24h later
removes it).

In doing all of this, I had to duplicate some of the code in refresh to avoid
holding packfiles references in `oldpacks`.

Reviewed By: DurhamG

Differential Revision: D15076018

fbshipit-source-id: f05852c3dddc6a25e1eb13bfd633c1fcc9466bb1
2019-04-25 18:53:06 -07:00
Zeyi Fan
6aa90444ab fix used before assignment error
Summary:
I'm getting reports on Eden failed to import data from Mercurial. One of the error message I have seen is `local variable 'rcvd' referenced before assignment` (unfortunately we don't have the backtrace to locate the exact problem), and this is the only place in our codebase has a variable named `rcvd`.

The log contains this error: P62827869$210-217

So if there is any exception raised in `self._connect`, `rcvd` and `total` will not be initialized, and these variables are being used to log the error message in the `except` block:

diffusion/FBS/browse/master/fbcode/scm/hg/edenscm/hgext/remotefilelog/fileserverclient.py;18aa2052b752f1255dd53474d541c4a4177bfef5$732-737

Reviewed By: quark-zju

Differential Revision: D15069162

fbshipit-source-id: 16ec56820107fbbb24d426ce309b38a88d7eae5b
2019-04-25 10:25:44 -07:00
Markus Emrich
44ace3d00a Safer parsing of referenced tasknumber
Summary: A more robust parsing regex for task references. It avoids some additional false positives, yet should cover all valid cases.

Reviewed By: Daij-Djan

Differential Revision: D15053374

fbshipit-source-id: 85410b5181eb9921d513b3a61ef3a1591b54539b
2019-04-24 19:25:11 -07:00
Jun Wu
e7cd31d046 revset: use path instead of relpath in follow()
Summary:
The `path` argument passed to the follow revset is absolute. So `path` instead
of `relpath` should be used.

Reviewed By: DurhamG

Differential Revision: D15071189

fbshipit-source-id: 6aec76fa1a8cabd545a375aa40448cc75dbd1d6d
2019-04-24 17:33:43 -07:00
Arun Kulshreshtha
f1c1cf95d6 edenapi: release GIL during data fetching
Summary: Release the GIL during data fetching to allow for progress bars to update properly. The data fetching code is pure Rust and does not interact with the Python interpreter at all, so releasing the GIL here is safe.

Differential Revision: D15051852

fbshipit-source-id: 144da953720951f9a30aadfc2b7fc8c8bc6b14aa
2019-04-24 13:33:58 -07:00
Mark Thomas
cfcad0d112 mutation: correctly detect split completion
Summary:
When detecting split completion for the purposes of recording mutation, we used
`cmdutil.comparechunks`.  This doesn't work for curses-recordings where the
only change is that some lines of a patch were excluded.

A simpler check is to just generate the patch that split is going to use, and
see if that matches the original patch.

Additionally, prevent the insertion of empty split records, as this can
cause crashes.

Reviewed By: mitrandir77

Differential Revision: D15063983

fbshipit-source-id: ba717d7f065faea93a500caaf10a1dbf582c7ab1
2019-04-24 12:04:10 -07:00
Mark Thomas
0a80521b0e visibility: convert from obsmarkers only when starting tracking
Summary:
Rather than relying on implicit conversion from a missing visibleheads file,
explicitly convert visibility information from obsmarkers when enabling
visibility tracking.

Reviewed By: mitrandir77

Differential Revision: D15063290

fbshipit-source-id: 44309f3cdf92c4ae100570b3bd98a240999ed558
2019-04-24 12:04:09 -07:00
Mark Thomas
ccf6c2f12a visibility: add blackbox logging of visibility changes
Summary: Track visibility changes in the blackbox log to help with debugging issues.

Reviewed By: mitrandir77

Differential Revision: D15062971

fbshipit-source-id: c547618168f5eb08e6343e0b0d97db136e151a7a
2019-04-24 12:04:09 -07:00
Mark Thomas
51139133c4 mutation: handle mutation graph loops in predecessorsset and successorssets
Summary:
Ordinarily loops are prevented in the mutation graph as the predecessors must exist
at the point that the successor is created.  However, backfilling from a complex
obsolescence graph may inadvertently introduce cycles.

Since loops are invalid, we can safely ignore any mutation edges that may
introduce them.  The `allpredecessors` and `allsuccessors` functions already
do this.

Add loop detection and ignoring to the `predecessorsset` and `successorssets`
functions.

Reviewed By: mitrandir77

Differential Revision: D15062399

fbshipit-source-id: fe892d9236c8d8dc4e1322b82618ab4bca35d30a
2019-04-24 12:04:09 -07:00
Mark Thomas
5e64071333 visibility: use graphnode '-' for all invisible commits
Summary:
Use the graphnode `-` for all invisible commits, even obsolete ones.

Users will only see them in their logs if:
- they run log with `--hidden`.
- they have invisible commits that are temporarily unhidden (e.g. they've checked it out).

Reviewed By: mitrandir77

Differential Revision: D15061894

fbshipit-source-id: 86873bd86cb15cef72dae248b8e2a636378cc547
2019-04-24 12:04:09 -07:00
Mark Thomas
78952d3298 mutation: assume a land if the successor is public and op is blank
Summary:
When computing the fate of a commit, if we find the immediate successor is the
next visible commit, we return the operation of the mutation record for that
single operation.  If it's not available, we then look at whether or not the
commit is public to decide between the fallbacks of `land` or `rewrite`.

Unfortunately, not all land operations end up with a `land` op.  In particular,
obsmarkers created with pullcreatemarkers and then backfilled into mutation
records have a blank operation, and so we show `rewrite` here when we should
show `land`.

Switch around how we calculate the fate.  Compute a default operation of `rewrite`
or `land`, based on the phase of the successor, and then use that if the successor
is not the immediate successor, or if the recorded operation is blank.

Reviewed By: mitrandir77

Differential Revision: D15061863

fbshipit-source-id: 753b0b58f84e653b40f9918f7ad3b3adfff359d8
2019-04-24 12:04:09 -07:00
Mark Thomas
a2131ea555 automigrate: add automigrate for mutation and visibility
Summary:
In the automigrate step at the start of pull, perform automigrations for
mutation and visibility.

If `mutation.automigrate` is set to true, then backfill obsmarkers into the
mutation store.  This step can take a couple of seconds for large obsstores, so
print a message.

If `visibility.automigrate` is set to `"start"`, switch to explicit visibility
tracking.  If it is set to `"stop"`, switch back to obsmarker-based tracking.

Reviewed By: mitrandir77

Differential Revision: D15046139

fbshipit-source-id: 284268d42b52c6b296c5c1b73db7bc218ae29a0a
2019-04-24 12:04:08 -07:00
Xavier Deguillard
baacd19749 remotefilelog: enforce a limit on the packfiles size
Summary:
Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to
limit the size of the hgcache. This will be particularly challenging for
Sandcastle/Quicksand as they already see hgcache over 100GB.

The long-term plan is switching to IndexedLog based stores with a log rotate
functionality to control the cache size. In the meantime, we can implement
a basic logic to enforce the size of the hgcache that simply remove packfiles
once the cache is over the configured size.

One complication of this method is that several concurrent Mercurial processes
could be running and accessing the packfiles being removed. In this case, we
can split the packfiles in 2 categories: ones created a while back, and new
ones. Removing packfiles from the first case, lookups will simply raise a
KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is
acceptable. The second category belongs to ones that were just created by
downloading them from Memcache/Mononoke, and the code strongly assume that they
will stick around. A failure at this point will not be recovered.

One way of fixing this would be to handle these failures properly and simply
retry, the other is to not remove new packfiles. A time of 10 minutes was chosen
to categorize the packfiles.

Reviewed By: quark-zju

Differential Revision: D15014076

fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd
2019-04-23 13:00:24 -07:00
Mark Thomas
94257a258b blackbox: clean up blackbox logging and log more during rage
Summary:
Clean up some of the calls to `ui.log` and how they appear in blackbox logging.

* Make the names of the events consistently use `snake_case`.
* For watchman, only log once for each watchman command.  Include whether or not it failed.
* Unify `fsmonitor` logging under the `fsmonitor` event.
* Omit the second argument when it is empty - it is optional and does nothing when empty.
* Increase the number of blackbox lines included in rage to 100.

Reviewed By: quark-zju

Differential Revision: D14949868

fbshipit-source-id: a9aa8251e71ae7ca556c08116f8f7c61ff472218
2019-04-23 02:48:50 -07:00
Arun Kulshreshtha
a13df01b7a remotefilelog: fall back to SSH if HTTP pack fetching fails
Summary: Per the title, if we attempt to fetch file data and history over HTTP and the fetch fails, fall back to SSH rather than crashing.

Differential Revision: D15035947

fbshipit-source-id: 2d00a49a51a0c8809daf1d28a6e3ab7f571415b0
2019-04-22 23:41:44 -07:00
Arun Kulshreshtha
af1b898311 edenapi: add debug config option
Summary: Add a debug option for HTTP data fetching. The intended usage is for it to gate verbose debug messages; the option can be set by Chef for users in the hg_dev tier.

Differential Revision: D15040988

fbshipit-source-id: b7eaa3bab4200e083cffc5822fb9873611725e6b
2019-04-22 18:46:34 -07:00
Xavier Deguillard
10d64adb01 sampling: silence BaseException.message deprecation warning
Summary: Almost all tests are failing on Windows due to a warning about BaseException.message being deprecated. Replace it with str(e).

Reviewed By: kulshrax

Differential Revision: D15039020

fbshipit-source-id: d984af91ec447b2f721eab2e3c6d39a0b350fb57
2019-04-22 17:58:07 -07:00
Durham Goode
bf9fd13411 pushrebase: throw if prechangegroup fails
Summary:
The prechangegroup hook didn't have throw=True set, so if the hooks
failed we ignored it. This seems to have been the case for a long time, but we
only recently hit it.

Reviewed By: kulshrax

Differential Revision: D15038494

fbshipit-source-id: 4fa9ed4924c02732e3e4070e747a80fbe63564c9
2019-04-22 17:10:15 -07:00
Arun Kulshreshtha
6745729cbd edenapi: make file validation configurable
Summary: Add a new config option to toggle file validation.

Differential Revision: D15034687

fbshipit-source-id: 3783ea1dacad9d1e494a5de1388f703db0ed1129
2019-04-22 14:46:29 -07:00
Arun Kulshreshtha
e5bbec4a60 edenapi: use argument unpacking in client constructor
Summary: We have to pass a lot of config options across the FFI boundary; these are currently passed as arguments to the Eden API client constructor. Let's use argument unpacking to avoid repeating a bunch of argument names in the call to the constructor.

Differential Revision: D15034480

fbshipit-source-id: 74d0830c686c8863fcede6e57404aec3f0a58ea1
2019-04-22 14:46:28 -07:00
Xavier Deguillard
324186a60c remotefilelog: avoid double close of mutable packs
Summary:
When eden request a tree, it manually commit the pending mutable pack files. In
the unlikely case where the temporary files are removed from the disk, the
pack.close() operation will fail, since the pending packs aren't reset, the
next commit that happens while the repo object is closed will try again. This
time, it may try to close an already closed packfile, leading to P62634761.

Reviewed By: quark-zju

Differential Revision: D15015632

fbshipit-source-id: 016617334498c0161feed9dcec5ce24df931ad9c
2019-04-22 10:15:42 -07:00
Marla Azriel
9e6d77b35a commands: update help text for amend
Summary: Updated help text for hg amend and moved some options to verbose

Reviewed By: kulshrax

Differential Revision: D15004588

fbshipit-source-id: 4c9e0bffb522184ac8750ed8aa4eb5a53b309bd0
2019-04-19 11:16:56 -07:00
Stefan Filip
a802e610d1 revisionstore: rename Store to LocalStore
Summary:
I want to give Store a more specific name so that it doesn't get
confused with other Store abstractions that we will add in the
future.

Reviewed By: singhsrb

Differential Revision: D15007383

fbshipit-source-id: 499bcda4aecd5389e3bc1eba5206ba72a69c4c3d
2019-04-19 09:51:29 -07:00
Jun Wu
ec5f4f4f6a smartset: fix a possible StopIteration exception
Summary:
Python's `next()` can raise the `StopIteration` exception, unlike Rust's `next`
which just returns `None` instead.

Fix it by providing a default value `None`.

Reviewed By: singhsrb

Differential Revision: D15008773

fbshipit-source-id: df885c63b8130ceac38f86c89f2547dde2d519ba
2019-04-18 19:32:54 -07:00
Xavier Deguillard
ef4844624e remotefilelog: remove fetchwarning
Summary:
The warning isn't that useful, and can actually cause more harm than good, as running `hg prefetch -r .`
can download gigabytes of unnecessary data to the hgcache.

Reviewed By: quark-zju

Differential Revision: D14999458

fbshipit-source-id: b0ff2c2ad0e441622066fac10a5efafe8de588db
2019-04-18 14:09:09 -07:00
Jun Wu
ae6bf97297 remotenames: make push with tracking bookmark work with infinitepush
Summary:
The `if dest` test fails with infinitepush changing `dest` to non-None.
Fix it by also checking if `dest` matches `default-push` (or `default`).

Reported by: fryfrog

Reviewed By: markbt

Differential Revision: D14965995

fbshipit-source-id: 91e68368eda4457d06059387307a9572bc6d2906
2019-04-17 08:41:12 -07:00
Kostia Balytskyi
029666aa4b debugcasecheckcollisions: fix case collision detection corner case
Summary:
The way this command was implemented before this change would collapse
existing case collisions into a single value of the `lowerdirlist` dict.
The value that was chosen would be dependent on the traversal ordering.

If this value would be equivalent to the newly-introduced file, we would
miss the collision.

Reviewed By: markbt

Differential Revision: D14971701

fbshipit-source-id: d352e96e512154d92fe6bc49dd76aec63b954fef
2019-04-17 07:48:31 -07:00
Jun Wu
177b4ef156 fastlog: patch follow revset instead
Summary:
Change the fastlog approach to patch the "follow" revset function instead of
"getlogrevs". This makes it more general purposed so it can speed up commands
like `hg log -r 'reverse(follow("dir"))'`.

Note: This will regress `log dir` performance, which will be fixed in the next
diff.

Reviewed By: sfilipco

Differential Revision: D14764074

fbshipit-source-id: c2a4c8e91359d971e6ea668e5ff1f0ab6eb0534c
2019-04-16 22:19:19 -07:00
Jun Wu
861bf3595a revset: move "log dir" special path to "follow()" revset
Summary:
The "hg log" implementation chooses to not use the "follow" revset when logging
a directory, but use "_matchfiles" instead. In an upcoming change, we'd like
"follow" to handle directories so fastlog only needs to patch the "follow"
revset.

The "follow" revset can take a directory pattern just fine. The problem is
"follow" will follow *every* file inside the specified directory, which is
quite expensive.

For now, I just moved the "_matchfiles" fallback path to "follow" so when it
detects there are too many files to follow, it will switch to "_matchfiles"
directly.

In theory, tree manfiest repos would have "tree manifest" infomation that speed
up "follow" on a directory. But that's a bigger change, and it's probably very
slow in our setup because our trees are lazy.

This changes some behaviors subtly, as reflected in tests:

- `-f path` can use DAG range instead of rev range, which is a good thing as
  rev range does not make much sense to end-users. This removes a "BUG" in
  test-commit-amend.t
- `-f dir` can follow renames if the directory contains just a few files.
  This looks like a good thing, and is reflected in `test-log.t`.

Reviewed By: sfilipco

Differential Revision: D14863134

fbshipit-source-id: 99ddff46d43f63ce03dc7bf005e3ac1cb9b39d03
2019-04-16 22:19:19 -07:00
Jun Wu
59cafd4595 smartset: add a fast path for spanset
Summary:
With upcoming changes, I noticed `limit(reverse(:.) & follow("path:fbcode/scm/hg"), 10)`
is much slower than `limit(reverse(::.) & follow("path:fbcode/scm/hg"), 10)`. I tracked
it down to the fact that spanset introduces a lot of unnecessary checks. Optimize it
by avoid using spanset in this case.

The revset pattern is used by `hg log`, the `reverse(:.)` part is to "define" the
order. Perhaps we sould replace it with `reverse(::.)`. But that's a BC change that
might have some unwanted side effects.

Reviewed By: sfilipco

Differential Revision: D14863135

fbshipit-source-id: 6ba8a02b58e1109bdf8370f03965a3b302cba6c0
2019-04-16 22:19:18 -07:00
Stefan Filip
dd6dd0f998 types: remove deprecated Key.name() and Key.set_name()
Summary: Removing this function in favor of using Key.path

Reviewed By: quark-zju

Differential Revision: D14945331

fbshipit-source-id: 6b6bb70375629edf37b2b04a86545f18e15b33b4
2019-04-16 15:34:31 -07:00
Stefan Filip
9c04803a33 pyrevisionstore: use Key::new instead of Key::from_name_slice
Summary: migration

Differential Revision: D14945334

fbshipit-source-id: 1244f0d4fb6df59ebd3b72c55dbd1e1013531464
2019-04-16 15:34:31 -07:00
Stefan Filip
09aa2f900f bindings: use RepoPath in edenapi
Summary: migration

Differential Revision: D14945335

fbshipit-source-id: 0aa90bee100fdae8755e7abfa6ac614f5bcaf266
2019-04-16 15:34:31 -07:00
Saurabh Singh
8479d48934 infinitepush: fix the help text for scratch bookmark commands
Reviewed By: quark-zju

Differential Revision: D14955186

fbshipit-source-id: 1ea79762621f54adc3d8f282472fc21d596593ba
2019-04-16 13:47:36 -07:00
Xavier Deguillard
8e078ed66b memcache: add an option to use the indexedlog instead of packfile
Summary:
When the option is set, the gets will be stored in the indexed log instead of
in a packfile. With the Python code now able to read from it, this will allow
us to get more real world testing.

Reviewed By: quark-zju

Differential Revision: D14895509

fbshipit-source-id: d73d49a028f7af199b7a0873551d7b18b047e50c
2019-04-16 10:47:09 -07:00
Saurabh Singh
0b0f262d0f infinitepush: introduce server side command for moving a scratch bookmark
Reviewed By: markbt

Differential Revision: D14919847

fbshipit-source-id: 5ae1750e78b24ac6126b621ca34942d7fb46d4fe
2019-04-16 09:54:27 -07:00
Saurabh Singh
c6aab56946 infinitepush: introduce server side command for creating scratch bookmark
Reviewed By: markbt

Differential Revision: D14919850

fbshipit-source-id: 2eef682fa528b82d0b9bdac1bad66ec47d552d3c
2019-04-16 09:54:27 -07:00
Mateusz Kwapich
802213bcbb phabricator: query the draft changes too
Summary: The super-smartlog shows "local changes" even when you have some unattached draft changes associated with the diff. This diff changes that.

Reviewed By: quark-zju

Differential Revision: D14936057

fbshipit-source-id: 32204ef9c5fa862923a165624913ecb344ed4a73
2019-04-16 06:33:30 -07:00
Stanislau Hlebik
e1e5d4bb0d pushrebase use hookargs correctly if transaction was taken
Summary:
Context: for Mononoke sync job we are considering applying a few pushrebase
bundles under the same transaction to make sync job faster. More details -
https://fburl.com/y4we86hc

Doing two pushrebases under the same transaction seems to fail for  just one
reason - `hookargs` is None on the second bundle. The reason is this line -
https://fburl.com/y4we86hc, after gettransaction() is called
bundleoperation.hookargs is set to None, but transaction.hookargs is set to
current hookargs value.

In this diff pushrebase is changed to check both bundleoperation.hookargs and
transaction.hookargs

Reviewed By: quark-zju

Differential Revision: D14930970

fbshipit-source-id: e4a4dd1c85b1fdca2699dd431040d65f2642ec8a
2019-04-16 03:10:02 -07:00
Mark Thomas
f337e0d5e4 sampling: fail gracefully when ui.log arguments are not a valid format
Summary:
`ui.log` expects to be called with valid format arguments.  If the arguments
are not a valid format string, or the number of arguments doesn't match the
number of format placeholders, formatting will fail.

In this case, catch the exception and fail gracefully.  Don't even bother
formatting if there is exactly one argument.

The `blackbox` extension already does this, so extend to the `sampling`
extension.

Also fix the place where `perftrace` calls `ui.log` with a string that might
contain formatting placeholders.

Reviewed By: quark-zju

Differential Revision: D14938952

fbshipit-source-id: 1d9802308dba925109c018124d51273c348526b4
2019-04-16 01:21:55 -07:00
Chad Austin
179e2d2db4 fix crash in sampling
Summary: `log` itself does the interpolation, so don't do it in advance.

Reviewed By: quark-zju

Differential Revision: D14941862

fbshipit-source-id: 6885202cfbff4ce0310d68bb1676369c3ef64c53
2019-04-15 17:27:53 -07:00
Xavier Deguillard
312286d5d0 remotefilelog: do not attempt repack on not existing directory
Summary:
Looking at the Hgerrors scuba table, I noticed that a lot of the sandcastle
machines had repack failures due to "No such file or directory". I'm suspecting
that's due to not having a local store to repack, and therefore listing of
files to repack would fail. Let's verify that the directory is present before
repacking to avoid this issue.

Reviewed By: quark-zju

Differential Revision: D14906503

fbshipit-source-id: 98fbe57310511df4fc9856bf71f836adefb3d855
2019-04-15 11:06:05 -07:00
Saurabh Singh
0f155ea3f7 infinitepush: move scratch bookmark matcher to common module
Summary:
This functionality is generic enough to be inside the common module so
that it can be shared easily. Upcoming commits will make use of this
functionality by importing from the common module.

Reviewed By: markbt

Differential Revision: D14919848

fbshipit-source-id: b3b7eef9eb0929334cba7ce869e7c9377c265da2
2019-04-15 09:29:31 -07:00
Saurabh Singh
839a5ffa93 infinitepush: move infinitepush server check to common module
Summary:
This functionality is generic enough to be inside the common module so
that it can be shared easily. Upcoming commits will make use of this
functionality by importing from the common module.

Reviewed By: markbt

Differential Revision: D14919849

fbshipit-source-id: 74c6ad5158bd196bb062e2ec7b04ca121b5d2c3a
2019-04-15 09:29:30 -07:00
Max Kareta
27abd9e79d dispatch: fixed arguments escaping before invoking hooks
Summary:
This diff adds args escaping using single quote symbol before sending them to hooks.
Before all arguments where joined by space symbol " " which was producing incorrect result when argument itself contains space symbol.

Reviewed By: markbt

Differential Revision: D14799188

fbshipit-source-id: df5a4324d138515a4b881df96f2991de03df7a5b
2019-04-15 06:56:50 -07:00
Mark Thomas
f4245770a3 shelve: add shelvename template keyword
Summary:
Add `shelvename` template keyword, which expands to the name of the shelve for
commits that contain shelves.

Reviewed By: farnz

Differential Revision: D14932985

fbshipit-source-id: cddebd2dbc6454f7c61ed296f37822179da8a2de
2019-04-15 05:44:49 -07:00
Stefan Filip
7e2b3c256f types: rename Key::new to Key::from_name_slice
Summary:
We should update the builder for Key to take a repo path. We could build
the key directly using the default struct constructor but representing
the two constructors as functions is more clear.

Reviewed By: quark-zju

Differential Revision: D14877543

fbshipit-source-id: 328906521cdbad535e28df22fea82f21e8b5410a
2019-04-14 19:56:50 -07:00
Stefan Filip
78d11002eb types: remove Key::node()
Summary:
This function is difficult to justify in the context of the Rust borrow checker.
The primary concern for this pattern is preventing mutation when the object is
passed around.

We can always add the function back if it has to more than just return the
underlying value.

Reviewed By: quark-zju

Differential Revision: D14877545

fbshipit-source-id: acdd796e1bee5445c1bce5ce0ceb41a7334e4966
2019-04-14 19:56:47 -07:00
Stanislau Hlebik
8adfcdd88c hgsql: use fetchall method in committodb
Summary:
It makes this method 25-30% faster (shaves off 250-300 ms).

Also it counts number of fetched rows correctly - fetchall method was
overriden, but looks like __iter__ method wasn't

Reviewed By: ikostia

Differential Revision: D14915472

fbshipit-source-id: 313695c1a83d05dac2fc801792226b6b64539cb5
2019-04-13 03:20:27 -07:00
Arun Kulshreshtha
106c528c30 amend: remove dead code
Summary: This extension has been refactored many times, resulting in some code and config options that are (as far as I can tell) completely unused. Let's remove them.

Reviewed By: singhsrb

Differential Revision: D14920751

fbshipit-source-id: 8d716a1d6144ba51cdb2750f6cfc6d6262b3b8a3
2019-04-12 14:24:46 -07:00
Mark Thomas
9160d34e9e histedit: fix call to hex in loginfo
Reviewed By: mitrandir77

Differential Revision: D14911288

fbshipit-source-id: f0cfda26022da98c3722946511e52d889f965810
2019-04-12 01:57:56 -07:00
Xavier Deguillard
5ddf39f788 remotefilelog: add an indexedlog contentstore
Summary:
While the Rust code can read/write content out of an indexedlog, the Python
code cannot. For now, all the writes will be done in Rust, and the Python code
will only be able to read from it.

Reviewed By: quark-zju

Differential Revision: D14894330

fbshipit-source-id: 5c1698d31412bc93e93dabb93be106a2ef17d184
2019-04-11 12:07:58 -07:00
Xavier Deguillard
301c0f173a remotefilelog: only take repacklock once per repack
Summary:
While not a correctness issue, taking and releasing the repack lock multiple
times per repack makes `hg debugwaitonrepack` unreliable, which in turns makes
test flakey.

Reviewed By: quark-zju

Differential Revision: D14877896

fbshipit-source-id: 682b649f388d19fd51bcf8dd205ac96944039e86
2019-04-11 10:43:38 -07:00
Durham Goode
c32fd83458 tracing: log manifold perftrace samples
Summary:
We currently send perf traces to the blackbox, but we also want to log
them to our metrics. This configures hg to log the perf trace as a sample. Later
diffs will configure our wrapper to route these samples to manifold.

Reviewed By: mitrandir77

Differential Revision: D14881308

fbshipit-source-id: 07690f9af9fe14279a9ebf5089548c5880cba45d
2019-04-11 10:06:37 -07:00
Mark Thomas
a8907ff92c visibility: add telemetry logging
Summary: Add telemetry logging for visible head count.

Reviewed By: ikostia

Differential Revision: D14892255

fbshipit-source-id: 8a141e48ee834c9916b4f6233024a110728734b3
2019-04-11 08:57:52 -07:00
Mark Thomas
105e0dccf6 obsolete: don't create autorels for the final successor
Summary:
Autorels attempts to detect the scenario where `P -> Q` and `X -> Y` are
being added, and there already exists a `P -> ... -> X` relationship.

In this case it will create a `Q -> Y` "copy" marker to express the fact
that `Q` should be copied.

However, this also triggers in the case where `Q == Y`, creating a revive
marker for `Q`.

Normally this is benign, as Q is probably visible anyway, however when there
are two commits associated with a diff that has been landed, pullcreatemarkers
can create two markers: `P -> L` and `X -> L`.  Since P and X are for the same
diff, there probably exists a `P -> ... -> X` relationship, and so autorels
attempts to make an `L -> L` marker.  This fails because L is public.

Differential Revision: D14891063

fbshipit-source-id: 3f076a003508dd7b7d17e3eb7cdaeb8ac09e6b15
2019-04-11 07:49:21 -07:00
Mark Thomas
fd1fc9e705 visibility: pullcreatemarkers should exclude hidden commits
Summary:
With explicit visibility tracking, some commits may be hidden but not obsolete.
Pullcreatemarkers should not include these hidden commits when it is considering
what to link to landed commits.

Differential Revision: D14891062

fbshipit-source-id: 542363c979aca85d6de2e2161c00d91f67f0add3
2019-04-11 07:49:21 -07:00
Mark Thomas
9c9b28498c registrar: document 'subonly' option to the command decorator
Summary:
The `subonly` option in the command decorator indicates that a command must be
called with a subcommand.  Document this in the decorator documentation.

Differential Revision: D14876782

fbshipit-source-id: d121f94437e47cad2a7f3abe0faac300792c1c76
2019-04-11 07:49:21 -07:00
Mark Thomas
511b52863e visibility: support cloud sync
Summary:
Add support for explicit visibility tracking in commit cloud sync.

This means commit cloud reads the visibleheads and syncs these with the commit
cloud heads directly, removing the source of problems where obsmarkers disagree
on different hosts.

Commit cloud requires that the ordering of heads is maintained to get stable
ordering of new commits.  Update the visibleheads tracking to maintain
ordering, rather than using sets.

Finally, the calculation of the replacement node was slightly off.  This was
revealed in the new test case that is being added, so it is also fixed.

Differential Revision: D14876266

fbshipit-source-id: fe5b6bffd196d3bd74e7582e29484969495eac8e
2019-04-11 07:49:20 -07:00
Mark Thomas
0b42418c85 visibility: improve documentation
Summary: Add doc comments describing what the visibility functions do.

Reviewed By: DurhamG

Differential Revision: D14871230

fbshipit-source-id: e0117ed1357a3a14b9f42034bf51a6cbb38ba67b
2019-04-11 02:45:16 -07:00
Mark Thomas
d562032896 mutation: improve performance of obsolete check
Summary:
The computation of whether a commit is obsolete or not can be improved.

We can cache which commits are known to not be obsolete.

We can also have a cache for each filter type so that we only need to compute
obsolete nodes that match the filter.

Finally, when we need to compute all obsolete commits, we can start by looking
for commits which are made obsolete by only their closest successors, and then
filling back obsolescence to the predecessors of these obsolete commits.

Reviewed By: DurhamG

Differential Revision: D14858655

fbshipit-source-id: 1d03e214ad878ecb6ae548f80373702e2a184146
2019-04-11 02:45:16 -07:00
Mark Thomas
a5207547aa mutation: add absorb support
Summary: The `absorb` command should also record mutation information when it modifies commits.

Reviewed By: DurhamG

Differential Revision: D14871232

fbshipit-source-id: 46bc95b7f5781f0b5f5e057a34c755fcfe653f7e
2019-04-11 02:45:16 -07:00
Mark Thomas
07f5225572 visibility: don't let hiddenoverride pin commits visible
Summary:
With explicit visibility tracking, the hiddenoverride feature of the amend
extension is no long necessary.  When upgrading to explicit visibility,
include the pinned visible commits, however after that point, hiddenoverride
should have no effect.

Users can permanently reveal obsoleted commits using `hg unhide`.

Reviewed By: DurhamG

Differential Revision: D14851269

fbshipit-source-id: b516fa97ed1545d39b51baaf9c574cb73a571645
2019-04-11 02:45:15 -07:00
Mark Thomas
d803ebb3c7 visibility: make enabling or disabling of tracking a command
Summary:
Rather than implicitly upgrading or downgrading a repo based on the
`visibility.tracking` config option, add a new `hg debugvisibility` command to
do this explicitly.

Reviewed By: DurhamG

Differential Revision: D14871231

fbshipit-source-id: 73f4648408b3eca9ac12bd77e54d2d37ee342069
2019-04-11 02:45:15 -07:00
Mark Thomas
fb42aba542 infinitepushbackup: exclude hidden commits from backup heads
Summary:
When run with `--hidden`, the calculation for `backupheads()` may include extra
commits (ones that are hidden but not obsolete).  This can cause some of the old
nodes to apper with "Not backed up" annotations.  They should be excluded, too.

Reviewed By: DurhamG

Differential Revision: D14851035

fbshipit-source-id: f176b50a7ac62dc493a5c4df1e0c72532c43b680
2019-04-11 02:45:15 -07:00
Mark Thomas
38930fd664 pullcreatemarkers: also create synthentic mutation entries and update visibility
Summary:
The pullcreatemarkers extension creates fake markers for landed commits by comparing
the differential revision of draft commits with those of landed commits.

In the future, mutation records will handle lands correctly, however until that stage,
replicate the pullcreatemarkers behaviour.

Reviewed By: DurhamG

Differential Revision: D14834746

fbshipit-source-id: e196ee28645270ccab1775ea896f0e453740c03e
2019-04-11 02:45:15 -07:00
Mark Thomas
2f0922256b mutation: add debugmutationfromobsolete to backfill mutation records
Summary:
Add `hg debugmutationfromobsolete` which backfills mutation information from
obsmarkers.

Reviewed By: DurhamG

Differential Revision: D14603199

fbshipit-source-id: 64458c52ba0e9f7bc9b38abeb94660cd1423fca7
2019-04-11 02:45:14 -07:00
Kostia Balytskyi
8e92391e67 mononoke: preserve more output from hg side in the sync job
Summary: This will help us debugging of what's going on on the server side.

Reviewed By: StanislavGlebik

Differential Revision: D14876167

fbshipit-source-id: 17aaa6dca9d5b5d7bbf69be46b58e454fe9f3fc3
2019-04-10 18:05:37 -07:00
Mark Thomas
87d3ef1330 mutation: record predecessors and operation in commit loginfo
Summary:
Record the commit predecessors and the mutation operation in the commit loginfo
so that it can be logged to telemetry.

Reviewed By: quark-zju

Differential Revision: D14798032

fbshipit-source-id: 9c4ac1a4df3c91087c776d1f8e5fca94713b0390
2019-04-10 15:16:17 -07:00
Xavier Deguillard
543c250138 pyrevisionstore: convert to rust 2018
Summary:
Most of the extern crate are now removed. The cpython one is kept around as it
heavily depends on internal macros that would need to be manually imported.

Reviewed By: quark-zju

Differential Revision: D14864995

fbshipit-source-id: 05405ac7310e4dca65daf230cc8f574da32ed4c9
2019-04-10 10:24:55 -07:00
Stanislau Hlebik
0c019f20fd remove accidentally added lines
fbshipit-source-id: 70f4621d5adb859d39bf3bb8adb2ff84aa75ac5e
2019-04-10 00:45:56 -07:00
Stanislau Hlebik
dfdfbf232e experimental stash test
fbshipit-source-id: 5ff859300513dbb13889f3fc3bb781f7d56b85b6
2019-04-10 00:36:40 -07:00
Arun Kulshreshtha
de2605bf4d edenapi: make data and history batch sizes separately configurable
Summary: Make the batch size of data and history requests independently configurable, since data responses are typically much larger than history responses (since the former contains actual file data whereas the latter is only metadata).

Differential Revision: D14859686

fbshipit-source-id: c87c31f3e6611a55ae712e7f0ed9bb392d31a579
2019-04-09 17:00:01 -07:00
Arun Kulshreshtha
9fdd71e4df edenapi: use curl multi interface
Summary: Use the curl multi interface to fetch multiple batches of files or history entries concurrently.

Differential Revision: D14718547

fbshipit-source-id: c5a740c7e9106b719e825540f8182be31a72bae7
2019-04-09 14:59:44 -07:00
Aida Getoeva
0af9f95998 add backup command
Summary:
The command is supposed to check if the given commits are backed up and if not to back them up.

This is needed for `jf submit` to make sure that all the commits submitted are backed up by hg.

Reviewed By: markbt

Differential Revision: D14685222

fbshipit-source-id: 224a438ec1c5c17af6988c511be4af4a3a988a79
2019-04-09 12:14:08 -07:00
Aida Getoeva
6f4459dd42 don't skip metadata while creating data wirepacks
Summary:
Currently we skip metadata when constructing data wirepacks.
We need copy metadata in the data packflies, because the client expects the base text to contain copy metadata since the delta base on the server also contains copy metadata.

It is also needed for future file nodes validation.

Differential Revision: D14851678

fbshipit-source-id: 1a3f79dc2565cdb864bee2400d331ae3a7c3751b
2019-04-09 10:37:01 -07:00
Mateusz Kwapich
d5bae8360a phrevset: avoid crashing on null graphql reply
Summary:
The Dxxxx revset does two things in parallel:
 * walks the changelog in the search of commit associated with Dxxx.
 * makes graphql query to resolve Dxxxx to the hash associated with the last
   submitted version.

This behaviour is clowny and confusing and should be changed long-term. This is
just a short term fix to prevent it from crashing in cases there's no local
commit associated with Dxxx (or it's very old) and the graphql query comes back
with empty result (because there are no versions for the diff).

Reviewed By: markbt

Differential Revision: D14851313

fbshipit-source-id: 14f7531aa4691007266291a38a016242de563de4
2019-04-09 07:53:43 -07:00
Mateusz Kwapich
12ad4df265 phabricator: fix the getrevisioninfo for Diffs in reserved state
Summary:
Diffs in reserved state have latest_active_diff=null (but it's present). That's why our current logic failed for it.

=== Query

{P62246960}

=== Query Parameters

```
{
  "params": [
    {
      "numbers": 14814207
    }
  ]
}
```

=== Response

```
lang=json
{
  "data": {
    "query": [
      {
        "results": {
          "nodes": [
            {
              "number": 14814207,
              "diff_status_name": "Reserved",
              "latest_active_diff": null,
              "created_time": 1554501017,
              "updated_time": 1554501017,
              "is_landing": false,
              "differential_diffs": {
                "count": 0
              }
            }
          ]
        }
      }
    ]
  }
}
```

=== Run in GraphiQL

[[https://www.intern.facebook.com/intern/graphiql/intern/?query_paste_id=62246962 | Intern]]

Reviewed By: katherinez

Differential Revision: D14836776

fbshipit-source-id: 213afac5d51bf100973753ef8078ab0dcd18d52f
2019-04-08 13:32:17 -07:00
Mark Thomas
15242183ef treemanifest: consider all parents of outgoing nodes for tree comparison
Summary:
When determining the contexts to compare trees against, consider all parents
of the missing nodes that aren't themselves missing.

In the case of:
```
  C
 / \
D   B
    |
    A
```
Where A and D are common and B and C are missing, the parents of the roots
are only A, which means D's tree will not be compared against and all of its
trees will be included.

Reviewed By: StanislavGlebik

Differential Revision: D14798681

fbshipit-source-id: c80add907338a3011daf54a2304ea19415b7f9fb
2019-04-08 03:32:00 -07:00
Mark Thomas
6dc8896bd1 infinitepush: fix debugbundle for mutation parts
Summary:
The debugbundle implementation for mutation parts was not updated with
the recent changes made to mutation entries.  Update it.

Reviewed By: quark-zju

Differential Revision: D14798682

fbshipit-source-id: eb1defa33e8a0eed807360af2c130a225af15030
2019-04-08 02:54:39 -07:00
Arun Kulshreshtha
6705e2d120 bindings: allow choice between edenapi backends
Summary: Allow users to configure which HTTP client backend to use for the Eden API via the `edenapi.backend` config option. Valid options are `curl` and `hyper`, with `curl` being the default.

Reviewed By: quark-zju

Differential Revision: D14657871

fbshipit-source-id: 7a9972d2380fbbd5ed62d1accae764dc03ca4c29
2019-04-05 17:34:14 -07:00
Arun Kulshreshtha
6c8c87dea1 edenapi: add curl-based client
Summary:
Add a new Eden API client based on libcurl (via the rust-curl crate). This should help us work around issues with Hyper.

This implementation is based on curl's "easy" API, and is intentionally naive. I intend to update it to use curl's "multi" API to send several concurrent HTTP requests per operation in a later diff.

Differential Revision: D14656756

fbshipit-source-id: 1f71074506844104f0f3237023b38317a7f41979
2019-04-05 17:34:14 -07:00
Stefan Filip
9751716173 commitctx: refuse to commit files names that can't be interpreted as UTF-8
Summary: This is a stop gap measure while we move internal path encodings to UTF-8.

Reviewed By: quark-zju

Differential Revision: D14791761

fbshipit-source-id: e4d5a845aa4f28ed0744cf80021fd1446f3d6129
2019-04-05 16:30:00 -07:00
Stefan Filip
60e885a890 localrepo: import error module as errormod
Summary: `commitctx` takes error as a named parameter shadowing the module import.

Reviewed By: quark-zju

Differential Revision: D14805901

fbshipit-source-id: 359a6d19d6a08b3c65ebf1af8a5e5baaec6711a2
2019-04-05 16:30:00 -07:00
Marla Azriel
36a67da6f7 commands: help text for commit
Summary: Updated help text for hg commit

Reviewed By: kulshrax

Differential Revision: D14811336

fbshipit-source-id: dbc26b3409094b1f70aca01b0f9ff89693ce7861
2019-04-05 16:23:14 -07:00
Xavier Deguillard
99edfae94a remotefilelog: reduce packfile iteration cost
Summary:
In most cases, we won't have more packfiles than what is held in the LRU cache,
and therefore, building a set is unecessary. Testing for the length is a O(1)
operation and is therefore a quick test to verify if more packs needs to be
iterated on.

Reviewed By: quark-zju

Differential Revision: D14802940

fbshipit-source-id: 8b39befb27368d474ab71f3eeac2340d8183e70e
2019-04-05 15:50:06 -07:00
Xavier Deguillard
dbb35a26fa remotefilelog: Don't iterate on packs after a null refresh
Summary:
When many packfiles are present, and searching for missing data, we will
iterate on the packs twice. We can avoid the second iteration by testing if new
packfiles were detected.

Reviewed By: quark-zju

Differential Revision: D14802942

fbshipit-source-id: 624d9a86e65395af3141ea10d42bcfb8ee18db83
2019-04-05 15:50:05 -07:00
Xavier Deguillard
f8299c5d7c remotefilelog: cache pack path
Summary:
Building a set is a O(n) operation, and in the cases where refresh is called
often, it starts showing up in profiles. Instead of rebuilding it everytime, we
can simply update it when adding/removing packs.

Reviewed By: quark-zju

Differential Revision: D14802945

fbshipit-source-id: f8967d09d2d7d0cc0d7400b047a587d536315002
2019-04-05 15:50:05 -07:00
Xavier Deguillard
6e6a7e3e8b remotefilelog: cache remotefilelog.descendantrevfastpath
Summary:
The repo.ui.configbool was showing in profiles when running `hg log -p`. Since
its value won't change for the duration of the operation, let's cache it.

Reviewed By: quark-zju

Differential Revision: D14802941

fbshipit-source-id: c1a2a764cb4d4a1c0d45d5118bb1e892c10798a6
2019-04-05 15:50:05 -07:00
Jun Wu
31c8ecbdf4 dispatch: print shorter crash header
Summary:
Change the crash header to just a single line without blaming extensions or
Python. This makes the crash log shorter, and easier to read when there are
crashes in run-tests.py diffs.

Remove `ui.supportcontact` since it's no longer used.

Reviewed By: sfilipco

Differential Revision: D14393982

fbshipit-source-id: 3ede8d3d3d8fd5d125944a9a750350d3ce356b14
2019-04-04 23:38:34 -07:00
Mark Thomas
9867fdb893 cmdutil: add checkout identifier to commit loginfo
Summary:
Add the checkout identifier to the loginfo for commit contexts that take data
from the working copy.

Reviewed By: quark-zju

Differential Revision: D14751601

fbshipit-source-id: 6ddcbf17b970ee24237edf501ae3d7fc8d320193
2019-04-04 11:29:50 -07:00
Mark Thomas
5274006a31 context: add loginfo for committable contexts
Summary:
Add `loginfo`, a new field to committable contexts, the contents of which will
be logged when the commit is committed to the repository.

Initially the logging only includes the commit node and author, but more things
will be added later.

Reviewed By: quark-zju

Differential Revision: D14751602

fbshipit-source-id: 2432dfb92dd2489cb8ad3464316a42ebaa79657c
2019-04-04 11:29:50 -07:00