Summary:
This updates our apply updates logic to record how many files and how
many bytes were written to disk.
Reviewed By: quark-zju
Differential Revision: D15173846
fbshipit-source-id: c26d2b17f45e407f837c78c93fc440e960d4f922
Summary:
We already had tracing for tree prefetches, but in some cases the
fetching happens ondemand and goes through a different code path. Now that we
have pack file receive size tracking, let's annotate this appropriately.
Differential Revision: D15172482
fbshipit-source-id: 2db1dc8464ed9e0b8f6987718c448e391744f445
Summary: This will let us get a feel for download speeds.
Reviewed By: kulshrax
Differential Revision: D15168153
fbshipit-source-id: 86291b8eb3393d54271d574250291ed691a64a86
Summary: Records how many memcache misses we hit.
Reviewed By: kulshrax
Differential Revision: D15168154
fbshipit-source-id: 53e105999d0af566666fcfaecc5bcaf414c5e804
Summary:
Adds tracing support for recording how many bytes we download for lfs,
including the download speed.
Reviewed By: kulshrax
Differential Revision: D15167587
fbshipit-source-id: fb76bbf15cd44dcef2601d9e750794816d273120
Summary:
The prefetch code was not refreshing the pack stores after writing
stuff into packs. This meant that in some cases we would execute a prefetch,
then when we went to read an entry from the store it wouldn't be found. That
would trigger a one-by-one download, which would happen to trigger a pack store
refresh (since remotecontentstore had a refresh) which would cause the rest of
the prefetched data to then become visible.
This fixes it to make prefetch responsible for refreshing the pack data.
This also caused issues where prefetching lfs files requires the hg fetching to
get the metadata, then it reads that metadata and does an lfs fetch. Since the
original metadata wasn't visible (since the packs had not refreshed), this
resulted in infinite recursion until the default 100ms refresh time passed.
Differential Revision: D15171701
fbshipit-source-id: 190b1da542675265efaad75a2a6826cbe3c9631c
Summary:
`bug29988wrapper` is a non-camel-case type, which fits the Mercurial Python
code style, but the Rust compiler warns about it. Suppress the Rust warning.
Reviewed By: quark-zju
Differential Revision: D15197827
fbshipit-source-id: 54d0b4020f8e43e3bc24f9ab37e0ede55dc919e9
Summary:
File that tracks accessed bookmarks doesn't interact with anything else and doesn't really need repo.wlock. However if does use wlock, hg commands like `hg log -r`, which normally are read-only, start waiting for the lock and thus are blocked by other writing commands, that might be running in different checkouts of the same repo.
Let's use a different lock for this feature.
Reviewed By: simpkins
Differential Revision: D15165788
fbshipit-source-id: f04c7196d51db67069c6420545be24d2b7c0af27
Summary:
`infinitepush.sqlindex` depends on `mysql.connector`, which is only needed on
infinitepush servers using the sqlstore, and may not be otherwise available.
Delay the import of this module until we know for sure that it is needed.
Reviewed By: ikostia
Differential Revision: D15197758
fbshipit-source-id: 34f166ceab2530c875a4539089e544755f199b7b
Summary:
Do not keep all revlogs in memory. Detect memory usage and drop filelogs on
demand.
Reviewed By: ikostia
Differential Revision: D15186221
fbshipit-source-id: 8311d479e8dd4f8f449981873a98940410c20c9e
Summary:
On a memcache miss, let's put the server data into a pending packfile. This
will greatly reduce the number of packfiles being created, and thus reduce the
need to run repack.
Reviewed By: DurhamG
Differential Revision: D15154509
fbshipit-source-id: 661efd7fde4fc4f3f6441eebeaf7d3ff4c43871a
Summary:
The code to handle pending mutable packs was effectively duplicated in 3
places. The new class allows refactoring of all of it. The use of several
lambdas is unfortunate, but required as the repo and the cache path are dynamic
and can't be obtained when constructing the pendingmutablepack object.
Reviewed By: DurhamG
Differential Revision: D15023059
fbshipit-source-id: 1eae68fe66ce741eb36baa0c8db318ba32957b41
Summary:
This diff adds a new progress reporting framework to the Eden API crate and uses it to power progress bars for HTTP file downloads in Mercurial.
The new `ProgressManager` type is designed to aggregate progress values across multiple concurrent HTTP transfers. The API is currently designed to integrate well with libcurl's progress callback API, allowing all of the curl handles within a curl multi session to concurrently report their progress.
This progress can then be reported (in aggregate) to a user-provided callback. In most cases, this callback will be a Rust wrapper around a callback provided by the Python code. The `EdenAPI` trait and FFI bindings have been updated accordingly to allow optionally passing in a callback for long-running operations.
Lastly, in `remotefilelog`'s Python code, the callback is specified as a Python closure that simply updates the progress bar.
Reviewed By: quark-zju
Differential Revision: D15179983
fbshipit-source-id: ee677b71beff730f91aafe0364124f7ea0671387
Summary:
This introduces `repo.sqlreporeadonlystate()`, which works similarly to `repo.sqlisreporeadonly()`, but also gets the reason why the read only state is the way it is.
The underlying goal is to use this in a repo hook to report the lock state.
I retained the `sqlisreporeadonly` function so that we don't need to synchronize deployment of this code and the underlying hook.
Reviewed By: quark-zju
Differential Revision: D15165946
fbshipit-source-id: 0a62147167fa826b575178dd261990a956b5f317
Summary:
Move all the store creation and access code into bundlestore. Clean up the
interfaces to the index and store, removing the abstract base classes as no
code was being shared.
Reviewed By: DurhamG
Differential Revision: D15170164
fbshipit-source-id: f22ec4a266333b3100074b21cf9577c82f56e4c6
Summary:
Add support for the `unamend` command. It should use the mutation predecessors
if mutation is enabled, and update the visibility of the commits.
Reviewed By: quark-zju
Differential Revision: D15146976
fbshipit-source-id: e9ee4d26f45ba9e5c3c05a7bca80c8ac326adb9c
Summary: Per title, `hg debughttp` now prints out the hostname that the API server reports rather than the hostname in the URL we used to connect to it. The reason for this is that if the API server is behind a VIP, we get the actual hostname rather than just the VIP URL.
Differential Revision: D15170618
fbshipit-source-id: 9af5480f9987d8ea9c914baf3b62a00ad88d1b32
Summary:
I noticed this takes 500+ms in some cases. Let's track both the old
computation and the new, and flag them so we can differentiate between them.
Differential Revision: D15156559
fbshipit-source-id: ec2983081e4fa01d017212783e3fcb040f9fd4e5
Summary: The packfile code already does the sort, no need to do it twice.
Reviewed By: DurhamG
Differential Revision: D15154517
fbshipit-source-id: 6ecfc65315a7e443e0c385a5a011f1b491b3f6ac
Summary:
The caller specifies whether data or history or both is needed. Let's only
download what is requested to avoid the network transfer and the disk usage.
Reviewed By: DurhamG
Differential Revision: D15151457
fbshipit-source-id: 5f8600e2ef6c45a98e05b17448fcee8d6574fa36
Summary:
We have an existing method which can capture the desired functionality
without dropping all the data in the `revision_references` table. This
primarily solves these problems:
- Reuse of code.
Reviewed By: farnz
Differential Revision: D15107057
fbshipit-source-id: 5f9970ffd13536808c1b201481b6d2015fbe8295
Summary: This diff improves `hg debugedenimporthelper` reliability by retrying when a possible race condition happens in Mercurial. This could save us from the cost of restarting the importer.
Reviewed By: simpkins
Differential Revision: D15150902
fbshipit-source-id: 67af2e044e9c3a16c073a0d1252fbfc2e5ce1014
Summary: We are seeing some errors are being thrown as `RepoResetError` but it is quite difficult to tell what exactly is the error from our current logging. This diff improves the logging to help us debug the problem.
Reviewed By: chadaustin, simpkins
Differential Revision: D15067819
fbshipit-source-id: fefb27497076e42736a09bb3be3a0734fd67668d
Summary: Added logging of a number of the currently tracked accessed remote bookmarks.
Reviewed By: mitrandir77
Differential Revision: D15080683
fbshipit-source-id: c03c417afcd24683998689365c893d9e16f265f8
Summary:
Track remote names that are used as destination for hg update.
Resolving name in the repo happens actually twice: as validation during parsing and tokenization of the given specs and actual resolving. This is still fine to update file with used bookmarks twice, because the update/pull operations are much heavy, and updating the file won't make things much slower. Implementing kind-of-cache for used remote names so we could update the file once, isn't worth it, as the feature will be temporarily enabled and won't be needed after the selective pull rollout.
Reviewed By: markbt
Differential Revision: D15048105
fbshipit-source-id: 5b03443a6ab349e3bd88613d21e7b1efdc1ff6cf
Summary: `accessed(..)` method will allow to log the fact of a name access, that is needed to track interesting remotenames.
Reviewed By: markbt
Differential Revision: D15047919
fbshipit-source-id: 29566653e742b3f24b742ffce2282baed833ea61
Summary:
Tracking remote bookmarks that was pulled with
```
hg pull -B <remote name>
```
All these remotenames, if they exist, will be stored in `.hg/selectivepullusedbookmarks` file.
It will allow us to estimate how much memory do we need to keep remote names in sync state in Commit Cloud and automatically mark collected remote bookmarks as "interesting" when the selective pull will be enabled.
Reviewed By: markbt
Differential Revision: D14912903
fbshipit-source-id: 3001869175553327c0840e2cfb829724dfd82893
Summary: We have turned off hgignroe globally for a while. Let's remove the code now.
Reviewed By: kulshrax
Differential Revision: D15110678
fbshipit-source-id: d0c2061413f6a48a6bfda5b26d15007e7890d20c
Summary:
The former is no longer maintained and throws warnings with recent rust
versions.
Reviewed By: singhsrb
Differential Revision: D15109706
fbshipit-source-id: 94479cdedf42c4dd99e35fa8e337d2fc73f74eb5
Summary:
D15053374 changed this to a great regex, but lost a single piece in translation - which is the fact that both `T` and `t` are valid prefixes.
And majority of tooling (like Tasks tool) is actually using `T` prefix for tasks.
This fixes it.
Reviewed By: mitrandir77
Differential Revision: D15125683
fbshipit-source-id: 30a5f5e01bff93613a964d6a0a13ed08067f8323
Summary:
This diff adds support for extension source code inlined into configuration.
The main intended use case is to use Chef to quickly deploy config changes.
The config change can be inlined extensions that patch broken logic.
Since extensions are more powerful than pure config changes, we gain the
ability to hotfix more complicated issues.
In theory, Chef can also write files to the filesystem. It's less convenient
because different OSes might require different paths, cleaning them up might
be extra work.
Reviewed By: ikostia
Differential Revision: D15073429
fbshipit-source-id: 7696bdce72bf7222debc72002173feb7de95198f
Summary:
To be able to investigate problems with dangling lock files "infinitepushbackup" and "prefetch".
- "infinitepushbackup", file "infinitepushbackup.lock" in repo.sharedvfs
- "prefetchlock", file "prefetchlock" in repo.svfs
Also some small fixes around.
- Crate "undolog" directory if it doesn't exist.
- `report` function was restructed. Now if lock checking method is not passed
lock will be checked with `lockmod.lock`.
Reviewed By: markbt
Differential Revision: D14949807
fbshipit-source-id: 02143ff923145e67e88c5627cf3355a834823b6d
Summary:
a "local changes" was visible in the smartlog even instantly after "jf submit"
because hg was comparing the has in the repo with the latest draft version.
https://fb.workplace.com/groups/scm/permalink/2115574591825483/
Reviewed By: katherinez
Differential Revision: D15098057
fbshipit-source-id: 5ad8e87a802a21ccbb1534c189d4e9de78e29fe7
Summary:
The existing bindings had a bug where if you replace a directory with a
file or symlink in a single commit, it would tell subversion to add the file
before it told it to delete the directory. This resulted in an error from
subversion and meant we couldn't actually push these commits.
Reviewed By: quark-zju
Differential Revision: D15066581
fbshipit-source-id: 1a1db0fa88cd3456fde45a4aec5d4df01837474d
Summary:
The error-chain crate is un-maintained and triggers warnings when compiling
with new versions of Rust. Let's use the failure crate instead to be consistent
with the other crates.
Reviewed By: singhsrb
Differential Revision: D15106306
fbshipit-source-id: 8edcf9f9aaf4b6e2d5f214b26fed3e72d4f3acd1
Summary:
D14185380 made it so pushrebase will always force pushkey to get a
transaction. This has the side effect of making infinitepush bundles take the
lock and fire transactions. They should do neither.
Let's instead request the transaction at bookmark edit time.
Reviewed By: singhsrb
Differential Revision: D15094917
fbshipit-source-id: 6573447a7ba61b1853a37eacb1b3e767abb3f27f
Summary:
When using ui.configoverrides, it was previously just backing up
self._rcfg. It turns out that calling ui.setconfig (which is called as part of
configoverrides) will mark the given config as pinned and potentially
unserializable. If a config is pinned, that means we will never again read a
value into it, so when we open a bundlerepo and attempt to load the repo
specific hgrc into it, any pinned values simply don't get written.
So there are two bugs here:
1. overriding a config should not result in the config key being considered
pinned forever.
2. Cloning a ui should reset the pinned list since we don't actually clone any
of the pinned data into the new ui
Either of these alone would fix our problem with bundlerepos, but both are
necessary for general correctness.
Reviewed By: ikostia
Differential Revision: D15074037
fbshipit-source-id: 37f67f13f1d2975299be97d796b52ec2b97bca24
Summary:
Undo removes visibility of commits one by one, starting at the bottom of the
stack. This doesn't work, as the bottom of the stack is kept visible by the
commits above it.
Remove them all in one go.
Reviewed By: mitrandir77
Differential Revision: D15079353
fbshipit-source-id: 52335b6dd1bd91c7f87d1a8ee2dbefa8aff7d24b
Summary:
Shelve uses hidden commits to store shelved changes. These need to be made
invisible, too.
Differential Revision: D15079352
fbshipit-source-id: 6063270d18df81d9b4af7823542a38c5feb45e3a
Summary:
pushrebase, infinitepush, remotenames, hgsubversion, and hggit wrap the
push command in very complicated ways. For instance, if hgsubversion push runs,
it will never call the original push command, and therefore if its wrapper gets
called first none of the other code gets executed. Unfortunately the opposite
can also be true, where the other extensions may make decisions that ultimately
prevent the hgsubversion code from running. I believe this is the cause of our
recent issue where someone pushed directly to hg from an svn backed repo.
The real fix is to clean up our mess of push commands, but for now we can
at least prevent remotenames from avoiding the hgsubversion push path entirely.
I believe this used to work more reliably before D14153207. In that diff it
changed the behavior from passing "None" as the destination (which is what was
passed in originally) to passing the actual url. The remotename code has some
logic that says "if the passed in destination is None, default, or default-push,
check if its svn/git and call orig". By passing in the actual url, this logic is
bypassed and we can head straight to the remotenames push code which may call
exchange.push and begin a vanilla push.
The easy fix is to just validate the passed in url directly if it exists and
call orig if its svn or git.
Reviewed By: quark-zju
Differential Revision: D15075782
fbshipit-source-id: 10c9c1c5669ff647a03987a2de5eaf1e3b474ce0
Summary:
After trying to enable `remotefilelog.fetchpacks` on windows, I realized that
the inline repack would always fail to remove the old pack files, leaving
temporary files around. The reason for this is that the packfiles are simply
not being garbage collected, even though `self.packs` is properly being
reinitialized. For some reason, that isn't enough to drop the refcount on the
old `self.packs` and therefore the packfiles aren't being released.
Instead of re-initializing `self.packs`, we can simply clear them, which will
release every one properly. In some cases though, it looks like one of the
packfile still isn't being released properly, I haven't figured out why, but at
least only one temporary file will be left around (until a repack 24h later
removes it).
In doing all of this, I had to duplicate some of the code in refresh to avoid
holding packfiles references in `oldpacks`.
Reviewed By: DurhamG
Differential Revision: D15076018
fbshipit-source-id: f05852c3dddc6a25e1eb13bfd633c1fcc9466bb1
Summary:
I'm getting reports on Eden failed to import data from Mercurial. One of the error message I have seen is `local variable 'rcvd' referenced before assignment` (unfortunately we don't have the backtrace to locate the exact problem), and this is the only place in our codebase has a variable named `rcvd`.
The log contains this error: P62827869$210-217
So if there is any exception raised in `self._connect`, `rcvd` and `total` will not be initialized, and these variables are being used to log the error message in the `except` block:
diffusion/FBS/browse/master/fbcode/scm/hg/edenscm/hgext/remotefilelog/fileserverclient.py;18aa2052b752f1255dd53474d541c4a4177bfef5$732-737
Reviewed By: quark-zju
Differential Revision: D15069162
fbshipit-source-id: 16ec56820107fbbb24d426ce309b38a88d7eae5b
Summary: A more robust parsing regex for task references. It avoids some additional false positives, yet should cover all valid cases.
Reviewed By: Daij-Djan
Differential Revision: D15053374
fbshipit-source-id: 85410b5181eb9921d513b3a61ef3a1591b54539b
Summary:
The `path` argument passed to the follow revset is absolute. So `path` instead
of `relpath` should be used.
Reviewed By: DurhamG
Differential Revision: D15071189
fbshipit-source-id: 6aec76fa1a8cabd545a375aa40448cc75dbd1d6d
