Summary:
The fast path avoids recreating the bundlerepo in the critical section, which
is like a reliable 0.8s win.
See the docstring in stackpush.py for details. It does not replace all use cases
that the old code path supports. So the old path is preserved.
Since it's a drop-in replacement, make it the default.
Reviewed By: phillco
Differential Revision: D10023543
fbshipit-source-id: eaceb9ae5067ab9040aa10cc65170ae54abd3331
Summary:
The next patch will change pushrebase code path so it does not have to go
through pushrebase._commit. Change globalrevs to wrap the lower-level
repo.commitctx to make sure commits have global revs assigned. Pushrebase
and hgsubversion wrappers are removed accordingly.
This also affects "hg commit" running in the server-side repo. Therefore
the test is changed to keep the change minimal.
It's also incorrect to reset the next revision counter at `repo.invalidate`.
It should only be reset at transaction abort. The original concern was
to get an up-to-date view of the revision number. Doing it in `repo.invalidate`
is risky. Enforce hgsql lock when reading the number to make it safer.
As we're here, change the transaction code so it does not wrap `_abort`
unnecessarily for nested transactions.
Reviewed By: singhsrb
Differential Revision: D10023541
fbshipit-source-id: 82d4b57dc2eafa8bc3cdf553e891db6e8c5ff741
Summary:
Inside the critical section, "needsync" runs twice and each time it scans
O(bookmarks + heads), which could be slow.
If the repo is actually up-to-date, there could be a faster path for testing -
just hashing all bookmarks and tip. Note: "heads" is skipped to save some time.
Since commits are append-only, "heads" change implies "tip" movement. So
checking "tip" is enough.
As we're here, fix the empty repo case so syncing is skipped for an empty repo.
Since this is a drop-in replacement for `needsync()[0]`, turn it on by default.
Reviewed By: phillco
Differential Revision: D10022866
fbshipit-source-id: f9e304865db3575515d66444762f21071d5665a3
Summary:
This line traced back to [unreviewd 652f3c86ddb5 (Increase mysql idle timeout to 5 minutes)](https://bitbucket.org/facebook/hgsql/commits/652f3c86ddb5).
The phaseroots only tracks draft commits. In a hgsql repo, the phaseroots file
is empty. All commits are public and nothing needs to be done for the new
commits to be public.
Removing this saves about 0.1 seconds in a hgsql sync.
Reviewed By: phillco
Differential Revision: D10021847
fbshipit-source-id: 5a3ca2533a7f8260ddfc90358b8026389ecd615f
Summary:
When entering the critical section (creating a transaction), hgsql might wait
for SQL lock for a long time (minutes). And it currently does nothing during
that wait time. If the lock was acquired after a long time, the work of
catching up with what the database already has might be too much. That is a
waste within the precious hgsql lock held critical section.
Instead of waiting for the lock for N seconds and doing nothing, wait for k
(k < N) seconds and also try to sync with SQL periodically. This makes the
repo closer to what the database has and can reduce work needed to pull from
the database in the critical section.
This adds CPU pressure to the MySQL tier. I added a config option to control
it. Worse case, we can change `hgsql.syncinterval` to make the sync less
frequent.
As we're here, try document the configs.
Reviewed By: phillco
Differential Revision: D10002578
fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
Summary: We want to add a test for this case before fixing it.
Reviewed By: quark-zju
Differential Revision: D10034448
fbshipit-source-id: 9244c57311d71a0bf1d75643caa31960a2f3519f
Summary: We need an in-depth test that tests both pushrebase and hgsql and shows how often we are reading uncached manifests. This is such a test.
Reviewed By: quark-zju
Differential Revision: D10023492
fbshipit-source-id: cf9e76e505c3bc478a7fc13d8d1adf1adeb6c1e4
Summary:
Multiple calls to the wrapped function should, in theory, be cached, so that calling them isn't as expensive as the first time.
Let's track what should be cached ourself so we can provide useful output. This is less risky than trying to separate the underlying logic in revlog.py.
Also, properly handle rev numbers (we are called with both) and turn them into nodes.
Reviewed By: quark-zju
Differential Revision: D10023518
fbshipit-source-id: 9a9ac1ed223136ca19d20db2ee6eb1ca11d5ae49
Summary:
Before this patch, `%include` support on Windows is:
# Works fine - UNC path: `\\?\c:\1.rc`.
%include c:\1.rc
# Works fine - UNC path: `\\?\c:\1.rc`.
%include \1.rc
# Works fine - UNC path: `\\?\c:\1.rc`.
%include c:/1.rc
# Bad - UNC path: `\\?\c:/1.rc`.
%include /1.rc
People expect `%include /1.rc` to work on Windows. Fix it by normalizing
the path in `%include` handling.
More context:
Normally, `/` and `\` can be used interchangeably on Windows. But it's not true
for UNC paths. The config parser uses `std::fs::canonicalize` to normalize
paths. The following Python script demonstrates the difference:
>>> import os
>>> open('c:\\1.rc').close()
>>> os.path.exists('\\\\?\\c:\\1.rc')
True
>>> os.path.exists('\\\\?\\c:/1.rc')
False
Reviewed By: phillco
Differential Revision: D10036882
fbshipit-source-id: fd85e0bc86d1e5776701077751ac875e71d60568
Summary:
Rust `std::fs::canonicalize` adds UNC prefix (`\\?\`) on Windows.
It's more correct as we can now access reversed names like `con`,
`com`, etc. Internally we'd probably want to use them.
But they're unfriendly to end-users. Let's strip them for display.
Related: https://github.com/rust-lang/rust/issues/42869.
Reviewed By: phillco
Differential Revision: D10036884
fbshipit-source-id: 8798ec5b5302f0d833896f89b228dc118086f41e
Summary:
This allows versions that don't know about storerequirements still access newly
created repos with this version. We will turn this on at a later date.
Reviewed By: singhsrb
Differential Revision: D10033964
fbshipit-source-id: e1065e05c33544d0287eda5eb852baff07c13147
Summary:
I want to centralize knowledge about specific sqllocks so I can change the
locking mechanism in the next commits of this stack
Reviewed By: quark-zju
Differential Revision: D9993477
fbshipit-source-id: 9398476b0ba8c3175ce84a7e0a809bbb8e60b7df
Summary:
With hgsql.verbose turned on, print useful messages like how many commits the
server is syncing, how long the lock was held or waited, to stderr so they are
visible to the client. This is useful for the client to understand why the
server "hangs" for a couple of seconds, and is useful in tests.
As we're here, drop no-flake8 comment.
Enable the flag for the hgsql-sync test to verify it.
Reviewed By: phillco
Differential Revision: D10019937
fbshipit-source-id: 8d304ce5208dbc5b92ed20f69daba02e9040c73f
Summary:
The watchman command will make sure watchman has the required filesystem
tree built internally so following commands like "clock" won't be easily
timed out.
Reviewed By: wez
Differential Revision: D10016298
fbshipit-source-id: 3874e670c10dd5387825e1a521eca135ad4de3b0
Summary:
When creating a VFS, Mercurial attempts to check if it is case sensitive by
stating the root of the VFS with differently cased names. If the root of the
VFS is a directory on an autofs mount, this check can be slow. Don't bother
checking if the VFS root has a filesystem type that we know is always case
sensitive.
Reviewed By: quark-zju
Differential Revision: D9993177
fbshipit-source-id: 4395b16f5f7b32218836422b44dae953dfb8c52b
Summary:
Using namespaces is a much cleaner and predicable way to handle commit
identifiers. Therefore, it makes sense to migrate to using namespaces in place
of stringset.
Reviewed By: quark-zju
Differential Revision: D9995107
fbshipit-source-id: 14d1fd0ebc796bcfe4583c08ee00bed8e5a09662
Summary:
Using namespaces is a much cleaner and predicable way to handle commit
identifiers. Therefore, it makes sense to migrate to using namespaces in place
of stringset.
Reviewed By: quark-zju
Differential Revision: D9995106
fbshipit-source-id: 61dbc602f14fda95c8f43e70deee84eaf92bee1e
Summary:
This commit just assigns a more appropriate priority to the existing
namespaces by keeping the higher priority i.e 70 for the namespaces which
resolve Phabricator diffs.
Reviewed By: quark-zju
Differential Revision: D9995104
fbshipit-source-id: ec0ae3eb53ee00ac0316039198f7fd0f2fa439ed
Summary:
Basically the plan to move everyone on dev servers out of tokens will be:
* set up all TLS paths in advance (they are suitable for interngpath as well)
* switch at the last moment these 2 options
```
[commitcloud]
remote_host=<new host>
tls.notoken=true
```
I made a fake token because I don't want to make all the logic more complicated
with more checks. If get function always returns something valid it will shadow
the rest of the logic about tokens.
First it will make everyone who already connected out of tokens
And make `hg cloud join` don't ask a token.
For users who are not yet connected we should think how to connect them
Reviewed By: markbt
Differential Revision: D9982195
fbshipit-source-id: 6c14218ed79717148d6f3295a57bb2d0dce71129
Summary:
This is nice for performance reasons
The only new logic is about checking that the revset effectively filter anything out from all set of draft commits.
If not, we just ignore it.
The rest of changes is cosmetic repair.
As we are close to enable everyone I think I will have to set this option
```
[commitcloud]
custom_push_revs=date('>Apr 01 2018')
```
or similar to minimize issues until mononoke and all old commits tree manifest conversion.
Reviewed By: markbt
Differential Revision: D9981311
fbshipit-source-id: fe09ff84f90a692e45ee58170e85b563743736be
Summary:
We have seen build errors on platforms like Ubuntu due to an old version of
Rust was installed. Upgrading Rust on all platforms supported by Mercurial
could be fairly painful. Therefore just install Rust directly as part of the
build step.
This patch changes `distutils_rust` to install Rust to `build/` and use `cargo`
from there.
Reviewed By: singhsrb
Differential Revision: D9985861
fbshipit-source-id: 07ddd3880b11ac67822333352290ad30b3b14b79
Summary: `path` has a different type on Windows.
Reviewed By: singhsrb
Differential Revision: D9991649
fbshipit-source-id: 7e0b071fa2f09903a295eb5e777bd00b46d3def8
Summary:
Sometimes watchman can return a very long list of files. That would make
blackbox rotate too frequently. Truncate that list.
Reviewed By: phillco
Differential Revision: D9997658
fbshipit-source-id: 1314b7c299a5f50ed344e0b85befd4c0525b1da7
Summary:
We want to make the merge conflict messages more expressive, and the first step is to make it possible for them to take additional arguments.
Let's make _onfailure potentially take a `callable`, into which we'll pass `r, repo, mynode, orig, fcd, fco, fca` -- all the context objects you could want related to the merge.
To keep this diff small, we won't change the error message at all yet.
Reviewed By: quark-zju
Differential Revision: D9814765
fbshipit-source-id: caebae3ce39f55432a2e702a0d76248f979ab8c9
Summary: Any nonzero return value is considered a failure here. Let's return the number of failed hunks so it can be used in the next diff.
Reviewed By: quark-zju
Differential Revision: D9806278
fbshipit-source-id: cdbcb2bfef2ec77555ffced3f38254ffea80c1be
Summary:
Part of push contention debugging. We want to be able to turn this on server-side to see how many uncached manifest reads we're doing.
If `pushrebase.debugprintmanifestreads` is set it'll log to blackbox.
If `pushrebase.debugprintmanifestreads.user` is set it'll also be sent to client logs, which is fine since most pushes are Landcastle these days.
Stolen from my test in D4785464.
Reviewed By: quark-zju
Differential Revision: D9994420
fbshipit-source-id: ff1294dfda2f2774fc6be78e096dad9fd9a28a5e
Summary:
It's cleaner for the config parser to take care of environment variable
handling.
A side effect of this change is, `$HGPROF` only affects `profiling.type`,
not `profiling:foo.type`, which is more desirable since we don't want
`profiling:foo.type` to be overridden by `$HGPROF`.
Reviewed By: markbt
Differential Revision: D9828547
fbshipit-source-id: 27be3683beee60a4eee6040ca1b4160dc1a89f73
Summary:
Previously, the `[profiling]` config section defines the profiler. Now it can be
defined by other sections, `[profiling:1]`, `[profiling:2]`, etc. Those sections
are sorted by name, and the first one with `enabled = true` is effective.
This makes it possible to specify secondary profiling configs, like having a
low-overhead profiler to be always enabled and logged if the command exceeds
certain threshold:
[profiling:background]
enabled = 1
freq = 1
minelapsed = 30
output = blackbox
type = stat
The config will only be used when the primary profiling (`--profile`, or
`profiling.enabled`) is disabled.
Reviewed By: markbt
Differential Revision: D9828548
fbshipit-source-id: 301a2b049d569d6fc065d05ab5b02e85b4ea8cc5
Summary:
Make "profiling" a variable. No logic change.
The next patch changes the config section name dynamically.
Reviewed By: markbt
Differential Revision: D9828551
fbshipit-source-id: b6b76012462f2dc1a76cd162381d949a94dc498b
Summary:
statprof hardcodes colors. When outputting to blackbox or other files, colors
should be disabled.
Note: since fp is buffered, we cannot get the information from `isatty`
reliably.
Reviewed By: markbt
Differential Revision: D9828546
fbshipit-source-id: 2c0ab839850ab213c2dc5cf4e2754dd6e357545c
Summary:
This config allows us to only log the profiling result if it exceeds a given
threshold. It's helpful for always enabling profiling and only log meaningful
data.
Reviewed By: markbt
Differential Revision: D9828549
fbshipit-source-id: f7e8f7e1e1f834d980519c2a639f3c7fd02f7f42
Summary:
The next patch adds a threshold to silence profiling output if the command is
fast. Therefore the profiling output shouldn't be written to terminal or files
directly.
Previously, only blackbox output is buffered. This simplifies the code since
self._fpdoclose, self._closefp, and self._output are no longer necessary.
Reviewed By: markbt
Differential Revision: D9828554
fbshipit-source-id: 7ac45ddc2cd332bd38dd97d08d9f7c3a27f1a16b
Summary:
traceprof uses `fprintf`, which requires a real `FILE *` to write.
That means it does not work for stringio, which is not backed by `FILE *`.
Fix it by always buffering the output in a temporary file.
As we're here, fix the Windows version of traceprof where Python and Cython can
have incompatible `FILE *`s, by only using `FILE *` in the Cython world.
Reviewed By: markbt
Differential Revision: D9828552
fbshipit-source-id: 8271d1aa3eb562ae7669b73af8f80473ef733a68
Summary:
In pushrebase code path, `_getmanifest` needs to be used to obtain the manifest
correctly for pushes from treeonly repos. The merge commit code path missed this
change. Fix it.
Besides, the rename checking code patch can also hit flat manifest errors via
repo.commitctx -> ... -> fctx._copied -> ctx.filenode -> ctx._fileinfo. Patch
that path too.
Reviewed By: phillco
Differential Revision: D9977904
fbshipit-source-id: 8c51aa9cc6bd85b0147d88c61a5af074e36cdb61
Summary: Part of tech debt week; let's make it so any time we reject an update due to conflicts, we print what the conflicts are.
Reviewed By: quark-zju
Differential Revision: D9784659
fbshipit-source-id: 1ee232b728bb602dbeaa4d2ecbc11016edd48dce
Summary:
the class now supports TLS setting, so other code will decide what to pass.
also the diff contains some minor code improvements
Reviewed By: markbt
Differential Revision: D9967713
fbshipit-source-id: 7326415e9b4e59220e9f15600d66405d36366fcc
Summary:
This commit adds a way to run the `globalrevs` extension only in read
only mode. This is useful in cases where repository `x` pulls from repository
`y`, both the repositories have `globalrevs` and `hgsql` enabled, we only
want to embed `globalrevs` in the commits for repository `y`, and we want to be
able to read the `globalrevs` in repository `x`.
Reviewed By: phillco, quark-zju
Differential Revision: D9978146
fbshipit-source-id: b9530e8e9cfadc28e2cd30e51649cee9e6e144bf
Summary:
The long filename encoding method introduced in D8527475 is insufficient for
files that have very long names that include some underscores.
For these files, we encode the underscores as `:`. This character is valid for
filenames on Linux, but won't work on Windows.
Reviewed By: mitrandir77
Differential Revision: D9967059
fbshipit-source-id: 67ea662f48f9fcc40e00b36c86697f3be6aac978
Summary:
this is a boolean option and the correct function should be used
Basically it was always true.
secrets_tool is our keychain like infrastructure, that is used to backup tokens.
This makes tokens available on any machine once backed up.
Reviewed By: ikostia
Differential Revision: D9967198
fbshipit-source-id: 8c1d8fc6b7a560c3e8879451c0c898717967f7dc
Summary: Date field is double in thrift schema of Commit Cloud Service.
Reviewed By: markbt
Differential Revision: D9952540
fbshipit-source-id: f07aa213e6ac3d534afd776dd248ca56c2f73f29
Summary:
Create Python bindings so that the BookmarkStore can be interacted with
from Python.
Reviewed By: quark-zju
Differential Revision: D9768565
fbshipit-source-id: 00d75b5250d8bc7dbeddd90d80ff4a23c60d00f9
Summary:
Create a storage object that can be used to load bookmarks from a
mercurial file, modify and query the bookmarks in memory and then write back
to a mercurial bookmark file.
Reviewed By: quark-zju
Differential Revision: D9768564
fbshipit-source-id: ed469d0e588ae2200d614bf62a5a0b577e7c6f74
Summary:
Copy functions from Mononoke to implement the Display trait
for a Node.
Reviewed By: quark-zju
Differential Revision: D9768566
fbshipit-source-id: 6961026a9e4cdaf4a0f2592dc9284abebadb0aa3
Summary: this diff improves errors for security commands we use to store tokens on OSX
Reviewed By: markbt
Differential Revision: D9943539
fbshipit-source-id: 1b59ed22d8d524c6b9b1b4ddda5c73276c19c6e8
Summary:
Preserve leading (but not tailing) new lines so the config (where `_` denotes a
space):
x_=__
__Foo
__
is parsed as `"\nFoo"`.
This is useful in template configs.
Reviewed By: ryanmce
Differential Revision: D9929764
fbshipit-source-id: e30659df94937c7c2121627f42ea425191003fb1
Summary:
When syncing with the commit cloud, if we have commits that are hidden locally,
but visible in the cloud workspace, and no local obsmarkers waiting to be
synced, revive the cloud-visble commits.
This prevents two repos from oscillating between two views when they have
a conflicting set of obsolescence information that has not been cloud shared.
Reviewed By: liubov-dmitrieva
Differential Revision: D9845984
fbshipit-source-id: 2f44ff8dbc636e2afa56d5efbb3ea5114472c41c
