Summary: Added logging of a number of the currently tracked accessed remote bookmarks.
Reviewed By: mitrandir77
Differential Revision: D15080683
fbshipit-source-id: c03c417afcd24683998689365c893d9e16f265f8
Summary:
Track remote names that are used as destination for hg update.
Resolving name in the repo happens actually twice: as validation during parsing and tokenization of the given specs and actual resolving. This is still fine to update file with used bookmarks twice, because the update/pull operations are much heavy, and updating the file won't make things much slower. Implementing kind-of-cache for used remote names so we could update the file once, isn't worth it, as the feature will be temporarily enabled and won't be needed after the selective pull rollout.
Reviewed By: markbt
Differential Revision: D15048105
fbshipit-source-id: 5b03443a6ab349e3bd88613d21e7b1efdc1ff6cf
Summary: `accessed(..)` method will allow to log the fact of a name access, that is needed to track interesting remotenames.
Reviewed By: markbt
Differential Revision: D15047919
fbshipit-source-id: 29566653e742b3f24b742ffce2282baed833ea61
Summary:
Tracking remote bookmarks that was pulled with
```
hg pull -B <remote name>
```
All these remotenames, if they exist, will be stored in `.hg/selectivepullusedbookmarks` file.
It will allow us to estimate how much memory do we need to keep remote names in sync state in Commit Cloud and automatically mark collected remote bookmarks as "interesting" when the selective pull will be enabled.
Reviewed By: markbt
Differential Revision: D14912903
fbshipit-source-id: 3001869175553327c0840e2cfb829724dfd82893
Summary:
The former is no longer maintained and throws warnings with recent rust
versions.
Reviewed By: singhsrb
Differential Revision: D15109706
fbshipit-source-id: 94479cdedf42c4dd99e35fa8e337d2fc73f74eb5
Summary:
D15053374 changed this to a great regex, but lost a single piece in translation - which is the fact that both `T` and `t` are valid prefixes.
And majority of tooling (like Tasks tool) is actually using `T` prefix for tasks.
This fixes it.
Reviewed By: mitrandir77
Differential Revision: D15125683
fbshipit-source-id: 30a5f5e01bff93613a964d6a0a13ed08067f8323
Summary:
a "local changes" was visible in the smartlog even instantly after "jf submit"
because hg was comparing the has in the repo with the latest draft version.
https://fb.workplace.com/groups/scm/permalink/2115574591825483/
Reviewed By: katherinez
Differential Revision: D15098057
fbshipit-source-id: 5ad8e87a802a21ccbb1534c189d4e9de78e29fe7
Summary:
The existing bindings had a bug where if you replace a directory with a
file or symlink in a single commit, it would tell subversion to add the file
before it told it to delete the directory. This resulted in an error from
subversion and meant we couldn't actually push these commits.
Reviewed By: quark-zju
Differential Revision: D15066581
fbshipit-source-id: 1a1db0fa88cd3456fde45a4aec5d4df01837474d
Summary:
D14185380 made it so pushrebase will always force pushkey to get a
transaction. This has the side effect of making infinitepush bundles take the
lock and fire transactions. They should do neither.
Let's instead request the transaction at bookmark edit time.
Reviewed By: singhsrb
Differential Revision: D15094917
fbshipit-source-id: 6573447a7ba61b1853a37eacb1b3e767abb3f27f
Summary:
Undo removes visibility of commits one by one, starting at the bottom of the
stack. This doesn't work, as the bottom of the stack is kept visible by the
commits above it.
Remove them all in one go.
Reviewed By: mitrandir77
Differential Revision: D15079353
fbshipit-source-id: 52335b6dd1bd91c7f87d1a8ee2dbefa8aff7d24b
Summary:
Shelve uses hidden commits to store shelved changes. These need to be made
invisible, too.
Differential Revision: D15079352
fbshipit-source-id: 6063270d18df81d9b4af7823542a38c5feb45e3a
Summary:
pushrebase, infinitepush, remotenames, hgsubversion, and hggit wrap the
push command in very complicated ways. For instance, if hgsubversion push runs,
it will never call the original push command, and therefore if its wrapper gets
called first none of the other code gets executed. Unfortunately the opposite
can also be true, where the other extensions may make decisions that ultimately
prevent the hgsubversion code from running. I believe this is the cause of our
recent issue where someone pushed directly to hg from an svn backed repo.
The real fix is to clean up our mess of push commands, but for now we can
at least prevent remotenames from avoiding the hgsubversion push path entirely.
I believe this used to work more reliably before D14153207. In that diff it
changed the behavior from passing "None" as the destination (which is what was
passed in originally) to passing the actual url. The remotename code has some
logic that says "if the passed in destination is None, default, or default-push,
check if its svn/git and call orig". By passing in the actual url, this logic is
bypassed and we can head straight to the remotenames push code which may call
exchange.push and begin a vanilla push.
The easy fix is to just validate the passed in url directly if it exists and
call orig if its svn or git.
Reviewed By: quark-zju
Differential Revision: D15075782
fbshipit-source-id: 10c9c1c5669ff647a03987a2de5eaf1e3b474ce0
Summary:
After trying to enable `remotefilelog.fetchpacks` on windows, I realized that
the inline repack would always fail to remove the old pack files, leaving
temporary files around. The reason for this is that the packfiles are simply
not being garbage collected, even though `self.packs` is properly being
reinitialized. For some reason, that isn't enough to drop the refcount on the
old `self.packs` and therefore the packfiles aren't being released.
Instead of re-initializing `self.packs`, we can simply clear them, which will
release every one properly. In some cases though, it looks like one of the
packfile still isn't being released properly, I haven't figured out why, but at
least only one temporary file will be left around (until a repack 24h later
removes it).
In doing all of this, I had to duplicate some of the code in refresh to avoid
holding packfiles references in `oldpacks`.
Reviewed By: DurhamG
Differential Revision: D15076018
fbshipit-source-id: f05852c3dddc6a25e1eb13bfd633c1fcc9466bb1
Summary:
I'm getting reports on Eden failed to import data from Mercurial. One of the error message I have seen is `local variable 'rcvd' referenced before assignment` (unfortunately we don't have the backtrace to locate the exact problem), and this is the only place in our codebase has a variable named `rcvd`.
The log contains this error: P62827869$210-217
So if there is any exception raised in `self._connect`, `rcvd` and `total` will not be initialized, and these variables are being used to log the error message in the `except` block:
diffusion/FBS/browse/master/fbcode/scm/hg/edenscm/hgext/remotefilelog/fileserverclient.py;18aa2052b752f1255dd53474d541c4a4177bfef5$732-737
Reviewed By: quark-zju
Differential Revision: D15069162
fbshipit-source-id: 16ec56820107fbbb24d426ce309b38a88d7eae5b
Summary: A more robust parsing regex for task references. It avoids some additional false positives, yet should cover all valid cases.
Reviewed By: Daij-Djan
Differential Revision: D15053374
fbshipit-source-id: 85410b5181eb9921d513b3a61ef3a1591b54539b
Summary: Release the GIL during data fetching to allow for progress bars to update properly. The data fetching code is pure Rust and does not interact with the Python interpreter at all, so releasing the GIL here is safe.
Differential Revision: D15051852
fbshipit-source-id: 144da953720951f9a30aadfc2b7fc8c8bc6b14aa
Summary:
Once `remotefilelog.fetchpacks` is enabled, `hg gc` will no longer be able to
limit the size of the hgcache. This will be particularly challenging for
Sandcastle/Quicksand as they already see hgcache over 100GB.
The long-term plan is switching to IndexedLog based stores with a log rotate
functionality to control the cache size. In the meantime, we can implement
a basic logic to enforce the size of the hgcache that simply remove packfiles
once the cache is over the configured size.
One complication of this method is that several concurrent Mercurial processes
could be running and accessing the packfiles being removed. In this case, we
can split the packfiles in 2 categories: ones created a while back, and new
ones. Removing packfiles from the first case, lookups will simply raise a
KeyError and data will be re-fetched from Memcache/Mononoke, ie: failure is
acceptable. The second category belongs to ones that were just created by
downloading them from Memcache/Mononoke, and the code strongly assume that they
will stick around. A failure at this point will not be recovered.
One way of fixing this would be to handle these failures properly and simply
retry, the other is to not remove new packfiles. A time of 10 minutes was chosen
to categorize the packfiles.
Reviewed By: quark-zju
Differential Revision: D15014076
fbshipit-source-id: 014eea0251ea3a630aaaa75759cd492271a5c5cd
Summary:
Clean up some of the calls to `ui.log` and how they appear in blackbox logging.
* Make the names of the events consistently use `snake_case`.
* For watchman, only log once for each watchman command. Include whether or not it failed.
* Unify `fsmonitor` logging under the `fsmonitor` event.
* Omit the second argument when it is empty - it is optional and does nothing when empty.
* Increase the number of blackbox lines included in rage to 100.
Reviewed By: quark-zju
Differential Revision: D14949868
fbshipit-source-id: a9aa8251e71ae7ca556c08116f8f7c61ff472218
Summary: Per the title, if we attempt to fetch file data and history over HTTP and the fetch fails, fall back to SSH rather than crashing.
Differential Revision: D15035947
fbshipit-source-id: 2d00a49a51a0c8809daf1d28a6e3ab7f571415b0
Summary: Add a debug option for HTTP data fetching. The intended usage is for it to gate verbose debug messages; the option can be set by Chef for users in the hg_dev tier.
Differential Revision: D15040988
fbshipit-source-id: b7eaa3bab4200e083cffc5822fb9873611725e6b
Summary: Almost all tests are failing on Windows due to a warning about BaseException.message being deprecated. Replace it with str(e).
Reviewed By: kulshrax
Differential Revision: D15039020
fbshipit-source-id: d984af91ec447b2f721eab2e3c6d39a0b350fb57
Summary:
The prechangegroup hook didn't have throw=True set, so if the hooks
failed we ignored it. This seems to have been the case for a long time, but we
only recently hit it.
Reviewed By: kulshrax
Differential Revision: D15038494
fbshipit-source-id: 4fa9ed4924c02732e3e4070e747a80fbe63564c9
Summary: Add a new config option to toggle file validation.
Differential Revision: D15034687
fbshipit-source-id: 3783ea1dacad9d1e494a5de1388f703db0ed1129
Summary: We have to pass a lot of config options across the FFI boundary; these are currently passed as arguments to the Eden API client constructor. Let's use argument unpacking to avoid repeating a bunch of argument names in the call to the constructor.
Differential Revision: D15034480
fbshipit-source-id: 74d0830c686c8863fcede6e57404aec3f0a58ea1
Summary:
When eden request a tree, it manually commit the pending mutable pack files. In
the unlikely case where the temporary files are removed from the disk, the
pack.close() operation will fail, since the pending packs aren't reset, the
next commit that happens while the repo object is closed will try again. This
time, it may try to close an already closed packfile, leading to P62634761.
Reviewed By: quark-zju
Differential Revision: D15015632
fbshipit-source-id: 016617334498c0161feed9dcec5ce24df931ad9c
Summary: Updated help text for hg amend and moved some options to verbose
Reviewed By: kulshrax
Differential Revision: D15004588
fbshipit-source-id: 4c9e0bffb522184ac8750ed8aa4eb5a53b309bd0
Summary:
I want to give Store a more specific name so that it doesn't get
confused with other Store abstractions that we will add in the
future.
Reviewed By: singhsrb
Differential Revision: D15007383
fbshipit-source-id: 499bcda4aecd5389e3bc1eba5206ba72a69c4c3d
Summary:
The warning isn't that useful, and can actually cause more harm than good, as running `hg prefetch -r .`
can download gigabytes of unnecessary data to the hgcache.
Reviewed By: quark-zju
Differential Revision: D14999458
fbshipit-source-id: b0ff2c2ad0e441622066fac10a5efafe8de588db
Summary:
The `if dest` test fails with infinitepush changing `dest` to non-None.
Fix it by also checking if `dest` matches `default-push` (or `default`).
Reported by: fryfrog
Reviewed By: markbt
Differential Revision: D14965995
fbshipit-source-id: 91e68368eda4457d06059387307a9572bc6d2906
Summary:
Change the fastlog approach to patch the "follow" revset function instead of
"getlogrevs". This makes it more general purposed so it can speed up commands
like `hg log -r 'reverse(follow("dir"))'`.
Note: This will regress `log dir` performance, which will be fixed in the next
diff.
Reviewed By: sfilipco
Differential Revision: D14764074
fbshipit-source-id: c2a4c8e91359d971e6ea668e5ff1f0ab6eb0534c
Summary: Removing this function in favor of using Key.path
Reviewed By: quark-zju
Differential Revision: D14945331
fbshipit-source-id: 6b6bb70375629edf37b2b04a86545f18e15b33b4
Summary:
When the option is set, the gets will be stored in the indexed log instead of
in a packfile. With the Python code now able to read from it, this will allow
us to get more real world testing.
Reviewed By: quark-zju
Differential Revision: D14895509
fbshipit-source-id: d73d49a028f7af199b7a0873551d7b18b047e50c
Summary: The super-smartlog shows "local changes" even when you have some unattached draft changes associated with the diff. This diff changes that.
Reviewed By: quark-zju
Differential Revision: D14936057
fbshipit-source-id: 32204ef9c5fa862923a165624913ecb344ed4a73
Summary:
Context: for Mononoke sync job we are considering applying a few pushrebase
bundles under the same transaction to make sync job faster. More details -
https://fburl.com/y4we86hc
Doing two pushrebases under the same transaction seems to fail for just one
reason - `hookargs` is None on the second bundle. The reason is this line -
https://fburl.com/y4we86hc, after gettransaction() is called
bundleoperation.hookargs is set to None, but transaction.hookargs is set to
current hookargs value.
In this diff pushrebase is changed to check both bundleoperation.hookargs and
transaction.hookargs
Reviewed By: quark-zju
Differential Revision: D14930970
fbshipit-source-id: e4a4dd1c85b1fdca2699dd431040d65f2642ec8a
Summary:
`ui.log` expects to be called with valid format arguments. If the arguments
are not a valid format string, or the number of arguments doesn't match the
number of format placeholders, formatting will fail.
In this case, catch the exception and fail gracefully. Don't even bother
formatting if there is exactly one argument.
The `blackbox` extension already does this, so extend to the `sampling`
extension.
Also fix the place where `perftrace` calls `ui.log` with a string that might
contain formatting placeholders.
Reviewed By: quark-zju
Differential Revision: D14938952
fbshipit-source-id: 1d9802308dba925109c018124d51273c348526b4
Summary:
Looking at the Hgerrors scuba table, I noticed that a lot of the sandcastle
machines had repack failures due to "No such file or directory". I'm suspecting
that's due to not having a local store to repack, and therefore listing of
files to repack would fail. Let's verify that the directory is present before
repacking to avoid this issue.
Reviewed By: quark-zju
Differential Revision: D14906503
fbshipit-source-id: 98fbe57310511df4fc9856bf71f836adefb3d855
Summary:
This functionality is generic enough to be inside the common module so
that it can be shared easily. Upcoming commits will make use of this
functionality by importing from the common module.
Reviewed By: markbt
Differential Revision: D14919848
fbshipit-source-id: b3b7eef9eb0929334cba7ce869e7c9377c265da2
Summary:
This functionality is generic enough to be inside the common module so
that it can be shared easily. Upcoming commits will make use of this
functionality by importing from the common module.
Reviewed By: markbt
Differential Revision: D14919849
fbshipit-source-id: 74c6ad5158bd196bb062e2ec7b04ca121b5d2c3a
Summary:
Add `shelvename` template keyword, which expands to the name of the shelve for
commits that contain shelves.
Reviewed By: farnz
Differential Revision: D14932985
fbshipit-source-id: cddebd2dbc6454f7c61ed296f37822179da8a2de
Summary:
We should update the builder for Key to take a repo path. We could build
the key directly using the default struct constructor but representing
the two constructors as functions is more clear.
Reviewed By: quark-zju
Differential Revision: D14877543
fbshipit-source-id: 328906521cdbad535e28df22fea82f21e8b5410a
Summary:
This function is difficult to justify in the context of the Rust borrow checker.
The primary concern for this pattern is preventing mutation when the object is
passed around.
We can always add the function back if it has to more than just return the
underlying value.
Reviewed By: quark-zju
Differential Revision: D14877545
fbshipit-source-id: acdd796e1bee5445c1bce5ce0ceb41a7334e4966
Summary:
It makes this method 25-30% faster (shaves off 250-300 ms).
Also it counts number of fetched rows correctly - fetchall method was
overriden, but looks like __iter__ method wasn't
Reviewed By: ikostia
Differential Revision: D14915472
fbshipit-source-id: 313695c1a83d05dac2fc801792226b6b64539cb5
Summary: This extension has been refactored many times, resulting in some code and config options that are (as far as I can tell) completely unused. Let's remove them.
Reviewed By: singhsrb
Differential Revision: D14920751
fbshipit-source-id: 8d716a1d6144ba51cdb2750f6cfc6d6262b3b8a3
Summary:
While the Rust code can read/write content out of an indexedlog, the Python
code cannot. For now, all the writes will be done in Rust, and the Python code
will only be able to read from it.
Reviewed By: quark-zju
Differential Revision: D14894330
fbshipit-source-id: 5c1698d31412bc93e93dabb93be106a2ef17d184
