Summary:
It's useful to know because we can track down slow requests and potentially fix
them on the client.
Reviewed By: farnz
Differential Revision: D10127735
fbshipit-source-id: d21088b50065f322b0c312f048f5828b3d96f99f
Summary:
Add the `sparse switchprofile` command, which lets users specify a set of
sparse profiles they would like to switch to. This is equivalent to `hg sparse
reset` followed by `hg sparse enableprofile`, but in a single step.
Reviewed By: farnz
Differential Revision: D10141938
fbshipit-source-id: 5a1e547d21e3246237fffb17f3b512da18fac804
Summary: Reformat all opted-in python code with version `18.9b0` of Black.
Reviewed By: ambv
Differential Revision: D10126605
fbshipit-source-id: 82af0d645dd411ce8ae6b8d239e151b3730cd789
Summary:
*Symptom:*
I have taken requests with longest time completion on Mercurial.
According to args logged in the request, remotelogfileserver was processing files with duplicates (same path and same filenodeid), which were going one by one.
*Investigations:*
According to some requests, duplicates among files are files of large size (> 1158513 bytes).
I have made a separate 'getfiles' request from local machine with the changed prefix_key for cacheprocess, to make sure, that remotefilelog.cacheprocess does not duplicate file_ids.
The changed key is required to get cache miss.
hg --config remotefilelog.cacheprocess=scmmemcache changed_prefix_key
I did not managed to simulate the case, with large file chunk cache miss.
*Solution*
To get rid of duplicates before running to server. Make a devel.warn if duplicates appear.
Reviewed By: quark-zju
Differential Revision: D10108311
fbshipit-source-id: e21ae4f8feab94a6762d53a74b503bb469783f60
Summary:
In D10023543 stackpush was added, and conflict check can happen earlier. In
that case failed pushrebase wasn't recorded. This diff fixes it.
Reviewed By: quark-zju
Differential Revision: D10119679
fbshipit-source-id: ef12e00a43375151f81a95eec2d9f02db0a9b12b
Summary: When stripping rev x, revlog revisions with linkrev >= x should be deleted.
Reviewed By: phillco
Differential Revision: D10108592
fbshipit-source-id: 2f9f5663327c4494bd7e836ab24ffc7e507530f4
Summary: Make sure local lock is acquired before acquiring the SQL lock.
Reviewed By: phillco
Differential Revision: D10108595
fbshipit-source-id: bc1167426d9812e17c1833aadf32d7f3039c6032
Summary:
If the database is writable, then the local repo must be up-to-date to avoid
writing lagged content to the database.
Reviewed By: phillco
Differential Revision: D10108597
fbshipit-source-id: a944f56907d559597b618a322e3ba75c296761ab
Summary:
* Rename takelock to dbwritable
The word "lock" is unclear what lock (local or SQL) it is. Make it clear it's
all about database writes.
* Rename waitforlock to enforcepullfromdb
Again, unclear what lock it is. It's also unclear what it does. Rename to
"enforce pull from db" to make it obvious.
* Rename syncdb to pullfromdb
"sync" is unclear about what direction to sync. Use "pullfromdb" to make it
clear. The hook name is unchanged for compatibility.
Reviewed By: phillco
Differential Revision: D10108594
fbshipit-source-id: fff405e2df9e926f5db436ef74cb5a9aacaebdb4
Summary:
We have seen cases where the local repo lock is obtained in the SQL critical
section, which is bad. Fix it by acquiring local transaction first.
As we're here, make incorrect lock order a hard error inside unbundle.
Other code paths are to be fixed.
Reviewed By: phillco
Differential Revision: D10108591
fbshipit-source-id: be152acc5ae7ef38541223c177398aa6883f7a0c
Summary:
The lock check should be making sure local repo lock is taken before the SQL
lock.
The current code does not work well. One of the reasons is because repo locks
can be nested.
Clean up the lock check so we now check at SQL lock functions to make sure we
still have the repo lock held. This way we don't have to replace the repo
lock functions.
It is not working yet because there are other code paths to fix.
Reviewed By: phillco
Differential Revision: D10108593
fbshipit-source-id: 489e4f27c06671191cf2084c7a726381be24a81f
Summary:
Delay updating the working copy until after the transaction that created the
obsshelve commit has completed. This means we won't update away from the
user's changes until after their work is safe, so if the update is interrupted
for any reason, the user will be able up do a clean update and then unshelve to
get their work back.
Reviewed By: liubov-dmitrieva
Differential Revision: D10102089
fbshipit-source-id: 5709d3915a6c458ba7cfb37ba5e0be5c6e8fcbb2
Summary:
If the `update` step is interrupted, then the transaction that created the
obsshelve commit is rolled back, but the update can't be rolled back, so the
files that have been shelved are lost.
Reviewed By: liubov-dmitrieva
Differential Revision: D10102088
fbshipit-source-id: f5bcac5c92069cc6ff3d1d9b7fb7ee507003d8eb
Summary:
This is for sandcastle and other automated tools.
They will not need to run extra commands like hg log after amend to learn new
hash.
Reviewed By: markbt
Differential Revision: D10101887
fbshipit-source-id: 7c9931776a03f4335bdfe0a19e7d569e3dc4c4ba
Summary:
In the hgsubversion tests, a develwarn from another component (which doesn't
have a `config` key) crashes with a `KeyError`. Change to `get` so that
it doesn't crash.
Reviewed By: quark-zju
Differential Revision: D9699155
fbshipit-source-id: c360d5fb5ab7daf7f609d41e0ddfb7456b022666
Summary:
These are no longer necessary. Callers should use the sharedvfs to access the
shared repo's .hg directory.
Reviewed By: quark-zju
Differential Revision: D9699163
fbshipit-source-id: 9b9cd584d721c174a7eab06f6abcedc3a943233b
Summary:
Update commitcloud and infinitepush to use repo.sharedvfs and repo.localvfs
as appropriate, rather than the vfs of the srcrepo.
Reviewed By: quark-zju
Differential Revision: D9699154
fbshipit-source-id: c1bb322f3724e756ab510044680e56c3a9a2443f
Summary:
Update journal to use sharedvfs and localvfs to access the correct parts of
the repository, rather than dealing with shared repos manually.
Reviewed By: quark-zju
Differential Revision: D9699168
fbshipit-source-id: 70b22ed7f8f44c9c87a71613c055292a0c1aec3c
Summary: Update rage to use `repo.sharedvfs`, instead of the vfs of the srcrepo.
Reviewed By: quark-zju
Differential Revision: D9699152
fbshipit-source-id: 24dc92997c24785ffc3399b737baa5ad00f96b21
Summary:
Update hgsubversion to consistently use repo.sharedvfs, rather than using the
vfs of the srcrepo.
Reviewed By: quark-zju
Differential Revision: D9699165
fbshipit-source-id: 1f4dacdd23ebc3baaa0d09b65e45d4ceeb067559
Summary:
Update most locations in the hg extensions to use `repo.localvfs` instead of
`repo.vfs`.
Reviewed By: quark-zju
Differential Revision: D9699153
fbshipit-source-id: 48d5f9678caa4961063db30477d6fbe0d6f34347
Summary: Update hggit to use the new repo.sharedvfs, rather than constructing its own.
Reviewed By: quark-zju
Differential Revision: D9699166
fbshipit-source-id: 21198d889085f368339d48b8b1ed9698d3e6a51a
Summary: Update references in core mercurial to use repo.localvfs instead of repo.vfs.
Reviewed By: quark-zju
Differential Revision: D9699162
fbshipit-source-id: 0401677d2b0a1340e66cffb7ee907a0d93aa6717
Summary:
Update bookmarks to work correctly when shared. Rather than copying the
bookmarks file over after transaction handling, update the file in both
locations when bookmarks are shared.
Reviewed By: quark-zju
Differential Revision: D9699158
fbshipit-source-id: 2f75aaac364ffe02e59441ac0f39fb7d8e5d5d2e
Summary:
Update `localrepo.repofilecache` to support file being in either the localvfs
or the sharedvfs, or both.
Reviewed By: quark-zju
Differential Revision: D9699159
fbshipit-source-id: 99f9485b8febd828b868061a7552ad54cd06a759
Summary:
Add `HG_SHAREDPENDING` which contains the path to the shared primary repository,
similar to how `HG_PENDING` contains the path to the local repository.
Repositories that are not shared check whether either of these refer to the
local repository path. Repositories that are shared check whether the pending
directory matches their own path, or the shared-pending directory matches their
shared path, via the new `trysharedpending` function.
This fixes the asymmetry in shared repos where pending changes made in a shared
repo were not visible in the primary repo, even though they were visible the
other way around.
Reviewed By: quark-zju
Differential Revision: D9699164
fbshipit-source-id: 31bc5fb2df6e9b9468b6ef39aabf877045c2a011
Summary:
Split the `repo.vfs` object into two. When a repo is not shared, these are
both the `.hg` directory of the repo. When it is shared:
* `repo.localvfs` represents the `.hg` directory of the local repository.
* `repo.sharedvfs` represents the `.hg` directory of the shared primary
repository.
The old `vfs` is an alias for `localvfs`. In the future, access through
this name will be deprecated to force callers to think whether they want
the local or shared hg directory.
Reviewed By: quark-zju
Differential Revision: D9699160
fbshipit-source-id: 6600df855c59b6df13e919399192789a873231c6
Summary:
Move the patching that is done by the `share` extension into core, so that
Mercurial is always share-aware, even in non-shared repositories, as they
themselves might be shared elsewhere.
Reviewed By: quark-zju
Differential Revision: D9699161
fbshipit-source-id: ffb6bb6cb89a67d6d6004052a272f7279ee0929b
Summary:
The library is different on different platforms, on dev servers it is
mandatory to add check_hostname param in 2 places, on other platforms in one place only.
In mercurial the library is defines as:
httplib = pycompat.httplib
So the implementations behind it are different on different platforms.
The exception was about unknown named param.
Reviewed By: rlangst
Differential Revision: D10084382
fbshipit-source-id: dce9d60a310951d18156b8357ffb0cf4410ac33a
Summary:
Previously, statprof uses a naive `time.sleep`, which cannot be interruptted
easily (considering Windows compatibility). Change it to `threading.Event.wait`
so it can end early without waiting for the full cycle of the sampling period.
Reviewed By: wez, kulshrax
Differential Revision: D10067234
fbshipit-source-id: 24e5fc0ab05491cb3e7ff34024402842ac3d7d44
Summary:
Wall clock time is more useful than CPU time because there could be lots of I/O
that affects end-user experience.
Reviewed By: phillco, kulshrax
Differential Revision: D10067235
fbshipit-source-id: dff6f124bb78b0c6c9a0ff7e2f7f08121bb5bc7a
Summary:
This gives us more information about how large the hgsql operation is.
As we're here, move `sqlwriteunlock` to an earlier place so the logging is outside
the sql lock. Also document `unbundle` a bit so it's clear `repo.transaction()` only
takes sql lock in `unbundle` context.
Reviewed By: phillco
Differential Revision: D10056307
fbshipit-source-id: 5d3361b4044e6fcf01e60409ef1ecb34da34ccac
Summary:
RocksDB and InnoDB are highly compatibile. There is no need to test RocksDB
engine for every hgsql related tests. Only use rocksdb for 2 of the tests.
Reviewed By: phillco
Differential Revision: D10055068
fbshipit-source-id: f9b7ef546fe7d457b0390e49014ebbe56d3c12c1
Summary:
This would be useful for investigating individual cases where sql lockheld time
is long.
Reviewed By: phillco
Differential Revision: D10055069
fbshipit-source-id: f3cdd7a198bdec777c12b98dd6a48e87d862c1bf
Summary:
Our hypothesis is that if the prepushrebase hooks take a significant amount of time, the repo state will get out date, causing hgsql to degrade under load.
This uses the previous test to simulate new commits coming in to the database while a single server is busy running prepushrebase hooks. The new code causes a second sync to occur just after running the hooks.
Reviewed By: quark-zju
Differential Revision: D9999683
fbshipit-source-id: 43d2390b476d090a66353555247c9a623386e75a
Summary:
This test case is a bit simpler given the problem -- there's no need for the holding push hook.
Instead, you just need a server that's missing a public commit. (Simulate an `hg strip` on one of the servers. `hg strip` isn't allowed, so create another repo with the extra public commit instead.)
Reviewed By: quark-zju
Differential Revision: D10043378
fbshipit-source-id: 532d8a2791abe5aaa6b6932747c7e0145202e8fe
Summary: Previously, `remotefilelog` would open a pipe for the `scmmemcache` process's `stderr` stream, but never read anything from the pipe. If the process actually writes to `stderr`, it is possible for the pipe buffer to fill up and block the child process, deadlocking hg. To avoid this, redirect `stderr` to `/dev/null`.
Reviewed By: quark-zju
Differential Revision: D10001508
fbshipit-source-id: be9b50133696424ea738d8bcd5365ed4d57cd9c6
Summary:
We are seeing issues with `arc feature --cleanup` (See
https://fb.facebook.com/groups/scm/permalink/1822421374474141/). This is
because of incorrect handling when `g<hash>` does not correspond to any
Mercurial hash. This commit fixes the issue.
Reviewed By: phillco, quark-zju
Differential Revision: D10042576
fbshipit-source-id: 922cbad0419a03aaa19ab1e1fafd7ac45e353ad7
Summary:
The fast path avoids recreating the bundlerepo in the critical section, which
is like a reliable 0.8s win.
See the docstring in stackpush.py for details. It does not replace all use cases
that the old code path supports. So the old path is preserved.
Since it's a drop-in replacement, make it the default.
Reviewed By: phillco
Differential Revision: D10023543
fbshipit-source-id: eaceb9ae5067ab9040aa10cc65170ae54abd3331
Summary:
The next patch will change pushrebase code path so it does not have to go
through pushrebase._commit. Change globalrevs to wrap the lower-level
repo.commitctx to make sure commits have global revs assigned. Pushrebase
and hgsubversion wrappers are removed accordingly.
This also affects "hg commit" running in the server-side repo. Therefore
the test is changed to keep the change minimal.
It's also incorrect to reset the next revision counter at `repo.invalidate`.
It should only be reset at transaction abort. The original concern was
to get an up-to-date view of the revision number. Doing it in `repo.invalidate`
is risky. Enforce hgsql lock when reading the number to make it safer.
As we're here, change the transaction code so it does not wrap `_abort`
unnecessarily for nested transactions.
Reviewed By: singhsrb
Differential Revision: D10023541
fbshipit-source-id: 82d4b57dc2eafa8bc3cdf553e891db6e8c5ff741
Summary:
Inside the critical section, "needsync" runs twice and each time it scans
O(bookmarks + heads), which could be slow.
If the repo is actually up-to-date, there could be a faster path for testing -
just hashing all bookmarks and tip. Note: "heads" is skipped to save some time.
Since commits are append-only, "heads" change implies "tip" movement. So
checking "tip" is enough.
As we're here, fix the empty repo case so syncing is skipped for an empty repo.
Since this is a drop-in replacement for `needsync()[0]`, turn it on by default.
Reviewed By: phillco
Differential Revision: D10022866
fbshipit-source-id: f9e304865db3575515d66444762f21071d5665a3
Summary:
This line traced back to [unreviewd 652f3c86ddb5 (Increase mysql idle timeout to 5 minutes)](https://bitbucket.org/facebook/hgsql/commits/652f3c86ddb5).
The phaseroots only tracks draft commits. In a hgsql repo, the phaseroots file
is empty. All commits are public and nothing needs to be done for the new
commits to be public.
Removing this saves about 0.1 seconds in a hgsql sync.
Reviewed By: phillco
Differential Revision: D10021847
fbshipit-source-id: 5a3ca2533a7f8260ddfc90358b8026389ecd615f
Summary:
When entering the critical section (creating a transaction), hgsql might wait
for SQL lock for a long time (minutes). And it currently does nothing during
that wait time. If the lock was acquired after a long time, the work of
catching up with what the database already has might be too much. That is a
waste within the precious hgsql lock held critical section.
Instead of waiting for the lock for N seconds and doing nothing, wait for k
(k < N) seconds and also try to sync with SQL periodically. This makes the
repo closer to what the database has and can reduce work needed to pull from
the database in the critical section.
This adds CPU pressure to the MySQL tier. I added a config option to control
it. Worse case, we can change `hgsql.syncinterval` to make the sync less
frequent.
As we're here, try document the configs.
Reviewed By: phillco
Differential Revision: D10002578
fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
Summary: We want to add a test for this case before fixing it.
Reviewed By: quark-zju
Differential Revision: D10034448
fbshipit-source-id: 9244c57311d71a0bf1d75643caa31960a2f3519f
Summary: We need an in-depth test that tests both pushrebase and hgsql and shows how often we are reading uncached manifests. This is such a test.
Reviewed By: quark-zju
Differential Revision: D10023492
fbshipit-source-id: cf9e76e505c3bc478a7fc13d8d1adf1adeb6c1e4
Summary:
Multiple calls to the wrapped function should, in theory, be cached, so that calling them isn't as expensive as the first time.
Let's track what should be cached ourself so we can provide useful output. This is less risky than trying to separate the underlying logic in revlog.py.
Also, properly handle rev numbers (we are called with both) and turn them into nodes.
Reviewed By: quark-zju
Differential Revision: D10023518
fbshipit-source-id: 9a9ac1ed223136ca19d20db2ee6eb1ca11d5ae49
Summary:
Before this patch, `%include` support on Windows is:
# Works fine - UNC path: `\\?\c:\1.rc`.
%include c:\1.rc
# Works fine - UNC path: `\\?\c:\1.rc`.
%include \1.rc
# Works fine - UNC path: `\\?\c:\1.rc`.
%include c:/1.rc
# Bad - UNC path: `\\?\c:/1.rc`.
%include /1.rc
People expect `%include /1.rc` to work on Windows. Fix it by normalizing
the path in `%include` handling.
More context:
Normally, `/` and `\` can be used interchangeably on Windows. But it's not true
for UNC paths. The config parser uses `std::fs::canonicalize` to normalize
paths. The following Python script demonstrates the difference:
>>> import os
>>> open('c:\\1.rc').close()
>>> os.path.exists('\\\\?\\c:\\1.rc')
True
>>> os.path.exists('\\\\?\\c:/1.rc')
False
Reviewed By: phillco
Differential Revision: D10036882
fbshipit-source-id: fd85e0bc86d1e5776701077751ac875e71d60568
Summary:
Rust `std::fs::canonicalize` adds UNC prefix (`\\?\`) on Windows.
It's more correct as we can now access reversed names like `con`,
`com`, etc. Internally we'd probably want to use them.
But they're unfriendly to end-users. Let's strip them for display.
Related: https://github.com/rust-lang/rust/issues/42869.
Reviewed By: phillco
Differential Revision: D10036884
fbshipit-source-id: 8798ec5b5302f0d833896f89b228dc118086f41e
