Summary:
Connecting to large workspaces can result in many pulls within a single sync.
This re-uses the connection to the server. We have seen instances where the
server can't serve all of these pulls and crashes with a MemoryError.
Pragmatically close the connection between each pull to prevent this.
Reviewed By: farnz
Differential Revision: D16858728
fbshipit-source-id: 7561c6f01c38df2706bd7fd76f5a7387e9728dc8
Summary:
So far the hashes of manifest/commits may vary, mostly because of
different path separators on different platforms (`/` vs `\\`) and its serialization to string in the snapshot manifest.
Reviewed By: markbt
Differential Revision: D16856130
fbshipit-source-id: a19dff113b9b24f1c7f387b9bc5a5e39e83ef8af
Summary:
Now that we are writing the git hash inside the commit itself, if we are translating
from hg to git hash, we should just use that rather than loading the entire map and
build a dict and look it up.
The performance of this translation will also improve quite a lot.
Some small sample benchmarking:
```
suiting@devvm5006:configerator (659199b|remote/master)$ time ~/fbcode/scm/hg/hg log -r master -T '{node} || {gitnode}\n' --config exte
nsions.fbconduit=! --config extensions.hggit=
659199bf7c7850ea9ffa9e0ad50eb84597977dea || 69f05aeec13f09c44dd00d2a85fe9d461ba6841e
real 0m0.563s
user 0m0.004s
sys 0m0.007s
suiting@devvm5006:configerator (659199b|remote/master)$ time ~/fbcode/scm/hg/hg log -r master -T '{node} || {gitnode}\n'
659199bf7c7850ea9ffa9e0ad50eb84597977dea || 69f05aeec13f09c44dd00d2a85fe9d461ba6841e
real 0m14.706s
user 0m0.006s
sys 0m0.006s
```
Reviewed By: quark-zju
Differential Revision: D16833526
fbshipit-source-id: 7d3096649cf24967d13596e70463bc125081ba4f
Summary:
The upcoming changelog change will *have to* use a different set of numbers.
Some of the numbers might be generated on demand, which has difficulity
resolving a number to a commit hash.
There are a lot of internal (especially, tests) usecases of revision numbers.
The deprecation message is only shown for user-provided inputs (ex. things
that go through `scmutil.revrange`).
This does not affect the use of `{rev}` template yet.
Reviewed By: sfilipco
Differential Revision: D16795522
fbshipit-source-id: 7a5578ecc0afdcc86830238471ff95203c96dc3f
Summary: Pass `uiconfig` to changelog so it can read config options.
Reviewed By: sfilipco
Differential Revision: D16683785
fbshipit-source-id: a64cfbe2cefa6b20ec695d2766bcfe878c764323
Summary:
A large portion of `ui` is config handling. Split it to a separate class so we
can pass that separate class to places that need pure configuration.
Some `develwarn`s are skipped since the new class does not have access to fout
or ferr.
Reviewed By: DurhamG
Differential Revision: D16683787
fbshipit-source-id: d823b9e5fc6f8ed08fff3401ab3502ad3c434f00
Summary:
Allow the user to check out on the snapshot by its revision id.
Snapshot == "a commit with extra containing a key `snapshotmanifestid`".
This corresponding value can be `None`, that would mean that snapshot does not contain untracked/missing/merge files.
Reviewed By: mitrandir77
Differential Revision: D16788479
fbshipit-source-id: bf4a9508acc940290b18123d3dd7b7fefae83782
Summary:
Now `hg debugcheckoutsnapshotmanifest` overwrites files if given the `--force` flag.
It also gives a more detailed output on the changes it makes.
Reviewed By: mitrandir77
Differential Revision: D16786334
fbshipit-source-id: b41d6241ffb478bd6c30a01c154b095d1ea92d78
Summary:
Add the `debugsnapshot` command, which will be renamed to `snapshot` later.
It creates a snapshot manifest that features information about
* untracked files,
* missing files,
* merge state artifacts from `.hg/merge` and `.hg/rebasestate`.
The snapshot manifest is stored in local lfs.
Then it creates the actual snapshot -- a fake commit, which has the snapshot oid in extra data.
It does not handle unresolved merge conflicts and other difficult states on this stage.
~~Finally, it restores the working copy and dirstate to status quo.~~
It doesn't need to be done, now we create only the commitctx, which does not wreck the dirstate.
Reviewed By: mitrandir77
Differential Revision: D16716359
fbshipit-source-id: 743f7427ce89c3fca6f844487bac1c456338e613
Summary:
Whenever a checkout is done, the new commit is sent to commitcloud.
This currently works with the hook on update, but the hooks on commit are not working
Reviewed By: mitrandir77
Differential Revision: D16687423
fbshipit-source-id: a0b861d301c84764f31787454cdec594b0519fa3
Summary:
The fastlog code says it cannot handle file renames, which is not true as far
as I know. Therefore, try use it for files. This affects the `follow()` revset.
I picked a random file, and fastlog is faster than the alternative code path
even with warm cache:
With fastlog:
% ./hg log edenscm/mercurial/smartset.py --time --pager=off -T '.' --debug --config fastlog.files=1
found common parent at a320f95a23716d0d106d922fa07ffcf0f838d3ff
starting fastlog at a320f95a23716d0d106d922fa07ffcf0f838d3ff
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
time: real 2.060 secs (user 1.870+0.000 sys 0.290+0.000)
Without fastlog, cold cache:
% sudo rm /var/cache/hgcache/fbsource/packs/*.hist*; ./hg log edenscm/mercurial/smartset.py --time --pager=off -T '.' --debug --config fastlog.files=0
fastlog: not used because fbcode/scm/hg/edenscm/mercurial/smartset.py is not a directory
.fetching content for 1 file over HTTPS
35.56 kB downloaded in 147 ms over 1 request (0.24 MB/s; latency: 147 ms)
fetching history for 1 file over HTTPS
445 B downloaded in 56 ms over 1 request (0.01 MB/s; latency: 56 ms)
running ssh -oControlMaster=no hg.vip.facebook.com 'hg -R '\''/data/scm/fbsource?stage1_read'\'' serve --stdio'
sending hello command
sending between command
remote: 713
remote: capabilities: unbundlehash gettreepack pushkey getfile getflogheads listkeyspatterns knownnodes getbundle lookup treeonly stream_option remotefilelog known unbundle=HG10GZ,HG10BZ,HG10UN clienttelemetry branchmap changegroupsubset unbundlereplay batch bundle2=HG20%0Ab2x%253Ainfinitepush%0Ab2x%253Ainfinitepushmutation%0Ab2x%253Ainfinitepushscratchbookmarks%0Ab2x%253Arebase%0Abookmarks%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps%0Atreemanifest%3DTrue%0Atreemanifestserver%3DTrue%0Atreeonly%3DTrue streamreqs=generaldelta,lz4revlog,revlogv1 getannotate stream-preferred
remote: 1
sending clienttelemetry command
connected to hg020.prn2.facebook.com
sending getpackv2 command
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
876 files fetched over 876 fetches - (1 misses, 99.89% hit ratio) over 5.98s
(running background incremental repack)
time: real 11.920 secs (user 5.050+0.000 sys 0.470+0.000)
Without fastlog, warm cache:
quark@devvm33994 ~/fbcode/scm/hg % ./hg log edenscm/mercurial/smartset.py --time --pager=off -T '.' --debug --config fastlog.files=0
fastlog: not used because fbcode/scm/hg/edenscm/mercurial/smartset.py is not a directory
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
time: real 2.530 secs (user 2.050+0.000 sys 0.070+0.000)
This is one step forward getting rid of the legacy filenode / ancestormap
logic. It regresses on offline experience, but we can build simple caches
around the `follow()` revset later.
Reviewed By: markbt
Differential Revision: D16365331
fbshipit-source-id: 16ffd6c3ec2daa5530fed94fb9c41fe75efd6182
Summary:
This option has been on for several months now, let's enable it by default in
the tests too. Since the behavior is slightly different, the tests had to be
adjusted:
- The packfile hashes are different due to a different repack algorithm,
- test-fb-hgext-remotefilelog-gcrepack.t removed due to gc not being a thing
with the Rust repack,
- test-fb-hgext-remotefilelog-repack-corrupt.t corrupt packs aren't detected
and removed at repack time, but later,
- test-fb-hgext-remotefilelog-repack.t we no longer re-delta context in the
Rust repack
Reviewed By: quark-zju
Differential Revision: D16774858
fbshipit-source-id: 7b7a3d96598c1ded0f64237047c3d97510050e4a
Summary:
When rust repack is default, and the packs directory doesn't exist, we wouldn't
try to run the loosefile repack, even if we had some. Let's move the packs
directory check a bit later to allow the loosefile repack to run.
Reviewed By: quark-zju
Differential Revision: D16774857
fbshipit-source-id: 1d1b67ae93c1b3aa2d07d5d15325e9be16a01196
Summary:
Let's always make options non-None. This was causing some issues when switching
to the Rust repack by default.
Reviewed By: quark-zju
Differential Revision: D16774856
fbshipit-source-id: 16fcda0b5a38327d389ed03f87c73d8bd00d5877
Summary:
Currently, only contents coming from `getfiles` request can replace
`BLACKLISTED_CONTENT` with `BLACKLISTED_MESSAGE`, because this is implemented
in `basestore.py`. Let us add an additional piace of logic to the
`remotefilelog.revision`, which would handle this the same way.
Note: this diff does not remove the old implementation. I think we need to do
this, but I want to focus on solving immediate problem, which is the absense
of support for censored results in `getpackv1`/`getpackv2`, before I fix the
tech debt. Note also that current doubling of the replacement logic does not hurt,
since it can't apply twice.
Reviewed By: xavierd
Differential Revision: D16735899
fbshipit-source-id: f10b72997614c58432476bcf95a3bde774bc278e
Summary:
This diff adds handling of mergestate and rebasestate -related data from the `.hg` directory.
That is the whole `.hg/merge` folder (both for merge and rebase state) and the `.hg/rebasestate` file.
We do not store any other information in lfs (e.g. number of parents for merge state).
Snapshot manifest oid will be added as an extra field to a fake snapshot commit, which will contain such data.
Differential Revision: D16708733
fbshipit-source-id: efc9b72b7593d85063307528c713c363e061065b
Summary: We were accidentally double-fetching data over both HTTP and SSH due to a missing return statement in the HTTP path. Fix it so that we only fetch data once. (The return was originally there, but was accidentally removed as part of a refactor.)
Reviewed By: xavierd
Differential Revision: D16745110
fbshipit-source-id: e06f129d39dec3801a32f90257367266006bb1cc
Summary:
We have can have situations where the parsing is broken because we are
parsing an attribute that does not exist. See T48658402 for an example.
Reviewed By: kulshrax
Differential Revision: D16745842
fbshipit-source-id: f693bc5aca29923b074a789b952e24b5ce72cdfd
Summary:
The logic is currently a bit broken in a weird combination:
The current configurations are
- usefirstpublicancestorinsvninfo=False
- useglobalrevinsvninfo=True
This means hg will have to traverse back to the last hg commit with svnrev. It will get
[slower and slower](https://fburl.com/scuba/skfj8p7u) to find those commits as we are having more commits without snvrev.
Ideally we can turn `usefirstpublicancestorinsvninfo` to True and that will then only traverse back to the last public ancestor. However it will trigger another bug because this public ancestor won't be in the old svn revmap and therefore hg will just bail with "Not a child of an svn revision".
Therefore the fix is to fix that bug by moving the `if pn not in hashes:` into the else part of the config and also get the subdir value from config rather than from the `extras['convert_info']`.
The next step is to set
- usefirstpublicancestorinsvninfo=True
- useglobalrevinsvninfo=True
- usereposubdirinsvninfo=True
once this diff is deployed to stable.
Reviewed By: DurhamG
Differential Revision: D16717554
fbshipit-source-id: c05b9882aa9ddeeeefd9315b478ba085fbc9bae7
Summary:
Created some classes and refactored the code.
Now it looks better. :)
Reviewed By: markbt
Differential Revision: D16690240
fbshipit-source-id: f26127d55c5cace7b88e225c85ec13cc278150c8
Summary:
It would really help to be able to checkout the manifests, restoring all the data from them.
Deleting the missing files, creating the unknown files, adding mergestate info to svfs (not available yet :) ).
Reviewed By: mitrandir77
Differential Revision: D16668312
fbshipit-source-id: 62af8a1fb11541c162f7b5ceb8d6d058cad9a319
Summary:
Add automigrate to commitcloud, which will automatically connect the user's
repo to commit cloud on pull.
Reviewed By: mitrandir77
Differential Revision: D16666810
fbshipit-source-id: a1d0857164d2ce6bf1db5784360681f04d35ed90
Summary:
Running `hg cloud leave` on a repo that was never connected should mark the
repo as explicitly disconnected so that automatic joining will not try to
connect the repo later on.
Reviewed By: quark-zju
Differential Revision: D16687470
fbshipit-source-id: 0552ffa42a1dac40874bba30eeb93509a2227aeb
Summary:
The `SubscriptionManager` is never a long-lived object, and cannot be
constructed if the user is not connected to a workspace. Simplify this to
function calls, which are safe to call at all times.
Reviewed By: quark-zju
Differential Revision: D16687467
fbshipit-source-id: c7f81290a3e8012f7b154af3398771ac9254387f
Summary:
`hg cloud rejoin` limits joining to when the user has an authentication token
and the workspace already exists. If the user uses a different authentication
method, or if the workspace is brand new, the rejoin does nothing.
Change this to always attempting to join, and only printing a message if the
join failed because of an authentication error, prompting the user to authenticate.
Reviewed By: mitrandir77
Differential Revision: D16666365
fbshipit-source-id: 3ea4542125a1b5266711fab2c31d9455ab764cef
Summary:
When `treemanifest.usehttp` is enabled, make `remotetreestore` fetch tree nodes over HTTP rather than SSH.
Note that unlike the SSH prefetch operation, this downloads only a single tree node, which will change the I/O behavior of Mercurial to fetch manifest nodes on demand. In general, this will result in excessive round trips, but for certain commands (such as `hg cat`), this works reasonably well.
Future diffs will expand upon this functionality to make the HTTP behavior more performant. For now, this won't affect anyone unless they specifically enable the aforementioned config option.
Reviewed By: xavierd
Differential Revision: D16663197
fbshipit-source-id: 4117e057996ed30f0bfd186238264e31484c2620
Summary:
Add the `debuguploadsnapshotmanifest oid` command.
It checks that the local lfs has `oid`, then uploads it and all the related blobs to remote lfs.
Differential Revision: D16667158
fbshipit-source-id: 2978a6c0e7c58c3710f8253cf7b9ab36b24886ce
Summary:
Previously, we would use `None` as the start value for HTTP data fetching progress bars so that the bar would be rendered as a spinner until we received a response (which would contain a content size header, allowing us to set the progress bar total).
This would result in incorrect behavior if the server took a long time to return a response, because by default if no start value is provided the progress bar framework will display the number of seconds that have elapsed. Given that we've specified a byte count format function for the progress value, the progress bar would end up displaying the number of seconds elapsed as a byte count, which was very confusing.
Reviewed By: quark-zju
Differential Revision: D16663124
fbshipit-source-id: 0e224b3b670abc803356214cd12205a6f4259cd9
Summary: Matching more of the existing API.
Reviewed By: quark-zju
Differential Revision: D16607233
fbshipit-source-id: 7a71f22089067ecfccbfcb2ad072fbf21e360439
Summary:
The initial attempt at supporting tree fetching over HTTP involved reimplementing the existing Mercurial prefetch functionality (corresponding to the gettreepack wireproto command) in the API server. Unfortunately, while this worked from a functionality perspective, it was far too slow to be usable in practice.
Rather than maintaining this unused code path, let's remove it and repurpose the config flag for a more sensible version of HTTP tree fetching.
Reviewed By: farnz
Differential Revision: D16663198
fbshipit-source-id: a489d5ca177b788f93383b8a9f758316e889ca5b
Summary:
The hgcache packs directory is being written to in several places, most of the
files being created there are temporary ones and renamed once ready to be
visible. In some cases, when hg crashes midway, or when the packfile is mmap'ed
on windows, temporary files are left around.
While we were trying to remove these files before, the solution taken involved
removing files we knew were safe to be removed. Unfortunately, that meant we
missed some of the temporary files. Let's instead switch to removing everything
but packfiles and the repacklock.
Reviewed By: kulshrax
Differential Revision: D16657105
fbshipit-source-id: f4dd45aa71c02acba52a745b9bec3019d20e03c4
Summary:
Add `not public()` to the unhide revset to improve its performance. We can't
unhide public commits anyway, as they can't be hidden.
Reviewed By: mitrandir77
Differential Revision: D16645983
fbshipit-source-id: f9f23e7e31050e652f1389489734416fe3278219
Summary:
Add a self-descriptive command `debugcreatesnapshotmanifest`.
For now it supports only deleted and unknown files.
1) Uploads all the untracked files to the local lfs storage.
2) Creates a json-like snapshot with the following structure:
```
{
"deleted": {
"path/to/file": None, # this is done for consistency
"path/to/another/file": None
},
"unknown": {
"added": {
"oid": "oid in local blobstore",
"size": 42
}
"another/added": {
"oid": "another oid in local blobstore",
"size": 24
}
}
}
```
...and stores it in the local blobstore.
Reviewed By: markbt
Differential Revision: D16621864
fbshipit-source-id: 6c497d1bb756561b3c3368483b838a2307b0b5f9
Summary:
The original code path is extremely slow because it has to iterate all files in manifest.
The new path instead only has to lookup the entries in keptdirs and therefore is O(change).
Reviewed By: mitrandir77
Differential Revision: D16646075
fbshipit-source-id: cb2c152d236ffa6f01349c223c9470205c540379
Summary:
To avoid the scenario where we are incorrectly uploading the data that we're
supposed to upload, we need to reset the file pointer to read from the
beginning so that for each retry of uploading we get the correct data.
Previously before this patch, Mercurial would have nothing to upload on a retry
and the connection would 'hang' waiting for the data to be send to the server.
Reviewed By: krallin
Differential Revision: D16620072
fbshipit-source-id: 2238b99ff612e06dc0640717bc42fe345c35037d
Summary: Make the number of retries for HTTP operations configurable. Previously, they were hardcoded to 3, which may not be ideal in situations where the retries are caused by server side timeouts. (If a single timeout takes 30 seconds it would take 90 seconds to fall back to SSH.)
Reviewed By: quark-zju
Differential Revision: D16592862
fbshipit-source-id: a9ba5273a1271532ff9814bcc47cb32e60ee87b7
Summary:
Similar to diff, the `match` argument can't be easily expressed using CPython
crate bindings. This argument is not used a lot so we can rename it.
Reviewed By: quark-zju
Differential Revision: D16571840
fbshipit-source-id: 19c7dea82924b7ec4c0b66d1675b9ad4569f8b62
Summary:
`match` is a keyword in Rust. This is causing troubles in adding Rust bindings
for functions that use that as a keyword argument. Renaming the argument to
something else seems to be the easiest path forward.
Reviewed By: quark-zju
Differential Revision: D16496134
fbshipit-source-id: c923f49577564527a99d43dda3d3d9da43122b3e
Summary:
The diff algorithm takes the `clean` flag. When `clean` is used all the files
that are not changed between the two manifests are returned. In short the set
of files is equal to |files(M1) U files(M2)|.
This functionality would have to be implemented in the Rust manifest. I don't
feel that a flag on the diff algorithm should be used in this case. First, I
don't like how it interacts with the core diff algorithm, it changes it to the
point where it feel like it should be a different function. Second is that this
behavior can be achieved by getting all the files in the manifest and removing
the items in the diff. Third is that this operation is done quite rarely, being
so expensive.
The downside is that the places where this flag is used get a bit more
expensive.
Reviewed By: quark-zju
Differential Revision: D16496136
fbshipit-source-id: 205dcc23517b896de5c14634683bcbd5f2aa6666
Summary:
This is used by Jellyfish and sent to Phabricator. In the future, we won't have svnrev
and therefore we should use globalrev instead.
More context: https://fb.workplace.com/groups/248282915800791/permalink/372855146676900/
Reviewed By: singhsrb
Differential Revision: D16561651
fbshipit-source-id: 284ad26b1bf77f222086bb7e2104b1c2dbf65449
Summary:
This command uses svnrev directly. However once we migrate to www-hg, this fields will
go and we can only use globalrev instead.
Let's add that and put it behind a config.
More context: https://fb.workplace.com/groups/248282915800791/permalink/372855146676900/
Differential Revision: D16560447
fbshipit-source-id: de3100ed1e6cc39eaaeff2fe11af04d2f1e2c41a
Summary: This makes IPython work without disabling demandimport.
Reviewed By: xavierd
Differential Revision: D16509106
fbshipit-source-id: d4443d8b58c90a0fd7a34c620756e88f4f788337
Summary: Added shortcuts a, d that allow you to skip 1 day forward or back in the historical smartlogs interactive view
Reviewed By: mitrandir77
Differential Revision: D16438722
fbshipit-source-id: e61a126ed7390b5ee77ee71694d8b23f3351854a
Summary: The interactive view doesn't support the pager, so it is problematic for longer smartlogs. As a quick fix the interactive view only contains two weeks of commits. The full version is accessible through hg cloud sl --version_number
Reviewed By: mitrandir77
Differential Revision: D16496065
fbshipit-source-id: 7224258b426724872bf796af618ef70cbf4c9d0b
Summary:
With fastdatapack gone, the cdatapack bindings are no longer necessary and can
be removed.
Reviewed By: singhsrb
Differential Revision: D16476584
fbshipit-source-id: 130a9c5aed4e4f005876c420961f09d398f6e6aa
Summary: This is no longer used anywhere, we can remove it.
Reviewed By: singhsrb
Differential Revision: D16476582
fbshipit-source-id: fe6a8f33cbc3c37fb1d8fb33226352e41bcbaa2a
Summary: This code isn't used anywhere, let's get rid of it.
Reviewed By: singhsrb
Differential Revision: D16476583
fbshipit-source-id: d42376dbb2cf631a170ade3e2764d1f70922d882
Summary: This code is no longer used anywhere.
Reviewed By: singhsrb
Differential Revision: D16462343
fbshipit-source-id: 3e11994575a6de75562cb1ebd8839f7058abc075
Summary:
Our reverse sync job is too slow when pushing to svn. It seems like the slowness comes
keep doing push-pull-rebase which doesn't seem necessary in our case.
From my understanding, we need to do push-pull-rebase because we have multiple writers.
In our sync job, we only have one writer and therefore can we skip the pulls and rebases?
Reviewed By: DurhamG
Differential Revision: D16442559
fbshipit-source-id: 926d1c516e8e6d59298d310fc67927ace37f72c9
Summary:
Util method to generate Enum, since some machines don't have the
python library installed.
Reviewed By: suitingtseng
Differential Revision: D16491020
fbshipit-source-id: e06470a609605c482f591418583b516918c49d93
Summary: There no users left, let's get rid of it.
Reviewed By: kulshrax
Differential Revision: D16423137
fbshipit-source-id: 6898e681f800ab677010d7b6cdd36d377e3ef644
Summary: The Rust code is the default.
Reviewed By: kulshrax
Differential Revision: D16423134
fbshipit-source-id: 30baf041a5e7c786a28b31cfe5025e893b34594c
Summary:
The Rust code is now always taken, let's start removing all the references to
the python mutabledatapack code.
Reviewed By: kulshrax
Differential Revision: D16392287
fbshipit-source-id: dfccd4f4ec052a46975b6f9144106b51c3282988
Summary: This is now enabled for the entire fleet, let's hardcode this in the code now.
Reviewed By: kulshrax
Differential Revision: D16392289
fbshipit-source-id: 462152ded12d00cf8218526d51a911d6fe5975ca
Summary: Checks if the retrieved content of a file is equal to a `magic string` representing a blacklisted file. If so, then the content is replaced by a readable text which suggests a `rebase` or `update` to a newer commit.
Reviewed By: ikostia
Differential Revision: D16260011
fbshipit-source-id: ac1d40132b9c947927271d8e6efda98b19dce984
Summary: The debug messages from Eden API provide Source Control team members with useful diagnostic information about HTTP data fetching, but they have the potential to be spammy when written to log files. To prevent log spam, let's only print these messages during interactive usage.
Reviewed By: quark-zju
Differential Revision: D16445346
fbshipit-source-id: 001dc75e440eaf797f4f953648453086421f624e
Summary:
When using fetchpacks, memcache will write to an indexedlog, which can't be
repacked. I'm also suspecting that there is a race between this repack, and
hg_memcache_client opening the newly created packfiles which can cause the
misses to not be sent to memcache.
Reviewed By: kulshrax
Differential Revision: D16432538
fbshipit-source-id: 62362682474883bcd58249791c02b9fed5cb8fea
Summary: Bindings so that the rust manifest code can be used in Python.
Reviewed By: quark-zju
Differential Revision: D16352532
fbshipit-source-id: 34d4522f5e084f531f31bcd21770950f15f2fe13
Summary:
Copytrace modified the global definitions table which was making it very difficult to keep track of side-effects as the code was executed, as well as making it harder to replace the fancyopts calls with native Rust.
Since the copytrace behavior can be achieved through a configuration, it now will no longer modify the global definitions table, and will display the correct flag for a user to use in order to get this same behavior.
Reviewed By: quark-zju
Differential Revision: D15902449
fbshipit-source-id: 1c254162d56823e65085b7047bb37513f187b487
Summary:
When rebasing or showing a diff for a commit which moved files, remotefilectx.ancestors (called by _tracefile) calculates the linkrev for each ancestor. Sometimes [1], this is a disaster:
* remotefilectx._linkrev executes its slow path and scans the change log.
* For each entry in the change log, remotefilectx._linkrev downloads trees if needed.
* Hg downloads trees one-by-one (as of D15964145).
remotefilectx.ancestors is only calculating linkrevs so it can sort the ancestors topologically. _tracefile only needs the ancestors to be ordered topologically, not by linkrev. remotefilectx.ancestors's calls to remotefilectx._linkrev are redundant.
Optimize _tracefile's use of remotefilectx.ancestors: order remotefilectx objects topologically (breadth-first) without sorting by linkrev. Create a new function for this purpose (topological_ancestors) to avoid possibly breaking other callers of remotefilectx.ancestors. As a side effect, make this new function return remotefilectx objects lazily, similar to the filectx.ancestors function.
On my machine, with warm caches, this speeds up 'hg diff -c' and 'hg rebase' for a modestly-sized commit. 'hg diff -c' takes 0.64 seconds, down from 65.6 seconds.
[1] Hypothesis: After 'hg amend', 'hg bundle' packages linkrevs which refer to the pre-amend commit (which is not serialized into the bundle) rather than the post-amend commit. 'hg unbundle' thus creates linkrevs referring to a missing commit.
Reviewed By: DurhamG
Differential Revision: D16297426
fbshipit-source-id: 407597d5e36fc06b33719c28f5ea5052e01dc7a3
Summary:
If one of the repo checkouts doesn't have remote bookmarks synchronization enabled, it must not affect the Cloud's remote bookmarks state.
I also changed `localserver` (is used for our tests) because Commit Cloud server deletes the "old" bookmark names from the db and inserts "new". However, the `localserver` just set the "new" books dictionary as a new state, which is not quite correct.
Reviewed By: simpkins
Differential Revision: D16338257
fbshipit-source-id: d77d9218b1c35ea1a097bbe7393d0910ce7b4d38
Summary: While checking if one public commit is an ancestor of another public commit, I used string hashes instead of binary nodes. Fixing.
Reviewed By: farnz
Differential Revision: D16338256
fbshipit-source-id: e27ed8d79ec9ed3a188cdee93366ab2f96526152
Summary:
This commit is just a simple refactor of the method for the resolving
the `svnrev` template keyword. In particular, we split the method so that it
can be easily wrapped by other extensions like globalrevs in D16361887.
Reviewed By: quark-zju
Differential Revision: D16361888
fbshipit-source-id: 9f20fb33afd2b286c4f30571fa257b8284f2bb54
Summary:
When we are migrating www to hg as source of truth, we would like to maintain the
reverse sync for some time in case we need to rollback.
In order to achieve that, we need to know the latest svn and hg commit to operate on. We
would like to record this information in the svn commit itself so it doesn't require
extra syncing and transaction.
In ordre to get this info, we can run `svn log -l 1` and parse the commit message from
there.
Reviewed By: quark-zju
Differential Revision: D16337012
fbshipit-source-id: acf66babdb48c07f95e9eb49daac0d3d3e6a96a0
Summary:
There are multiple places that want to know whether the code is running inside
tests or not. Add `util.istest` for that and migrate those callsites to use it.
Reviewed By: xavierd
Differential Revision: D16344715
fbshipit-source-id: faa4261c83e4d889d59d63968b440954c4cac2ce
Summary:
We've had cases where hg_memcache_client failed to write to the
indexedlogdatastore, but since it was returning a success code, hg wouldn't
find the node, and crash.
Let's add a new error code to signify that all the requested objects should be
considered as misses.
Reviewed By: quark-zju
Differential Revision: D16344530
fbshipit-source-id: 4e627fee1603f9e4d5f542ef2d1f0e881a085bad
Summary:
the default value was being set to type int, changed it in both occasions to "".
in cloudsync I had to do some parsing for the version into an int not to break the rest of the code
Reviewed By: mitrandir77
Differential Revision: D16340048
fbshipit-source-id: 7f7142ea4c7862c0a093a68d5c8ceb1d2e22df24
Summary: Updated help text for hg undo and fixed test
Reviewed By: kulshrax
Differential Revision: D15937058
fbshipit-source-id: 577d05fb9c16237f8b879ef9ebf60341d5fdf37e
Summary: Modified the hg cloud sl command to have two options --date and --version_number that display that historical version of the smartlog
Reviewed By: mitrandir77
Differential Revision: D16120273
fbshipit-source-id: 4f202ed49488247f43bd682574ff52dcf17c958e
Summary:
Bad callers could try to add nodes that had the same value as the
parent. Let's catch this and prevent it.
Reviewed By: quark-zju
Differential Revision: D16189602
fbshipit-source-id: 08220c18ee96743e4eda00e5e953a5946b40d95c
Summary:
The python runtime update in D15894694 broke ctypes for all APIs that accept a
Windows `HANDLE`. This was used by the `fsmonitor` code that tries to connect
to watchman on windows.
As a result Mercurial would silently fail to use watchman. Commands like
`hg status` became very slow, with no user-visible output to indicate that
anything was wrong. (It wouldn't print a progress bar or any other
information either.)
Reviewed By: quark-zju
Differential Revision: D16198869
fbshipit-source-id: 675007674e0ac742c5b62739d766b8c3016e0c1b
Summary:
Following D16198896, we can really simplify the logic for resolving
globalrev/svnrev to commit hash. In particular, we can make much more stronger
assumptions about what is indexed now and therefore, narrow the search to only
the commits that are not cached.
Reviewed By: quark-zju
Differential Revision: D16199072
fbshipit-source-id: cec66cbda24ee6df200fc77aeed42c11c29edeba
Summary:
When looking for the matching filenode when searching for the linkrev of a
filectx, override fetchdepth to 1 so that we don't download the whole manifest
just to examine a single file, and add tracing and a progress bar, as this can
be slow.
Reviewed By: mitrandir77
Differential Revision: D15964145
fbshipit-source-id: d338de330c82d4240bed5569caf25769778458aa
Summary:
With treemanifest and the eden api, descendantrevfastpath (which downloads all
the manifests) is slower than just downloading the history from the eden api.
We still want to record the descendantrev if we already know the real linkrev
of the current filenode, as that will still be fast.
Reviewed By: quark-zju
Differential Revision: D15943076
fbshipit-source-id: c6013801822bdfa3196e60cbcd34a9ce184c5c5f
Summary:
Adjusting linknodes can sometimes take a long time. Add a progress bar and perftrace
for when this happens.
Reviewed By: sfilipco
Differential Revision: D15943077
fbshipit-source-id: f91bc6bb9f7c6d39ffe9ac7b2bf4ab0f32b4d9d3
Summary:
During `cloud sync`, disable the following options, as they slow sync down and
aren't really necessary.
* Set `pull.automigrate` to False, as there is no need to perform pull migrations
in the sync pull.
* Set `treemanifest.pullprefetchrevs` to False, as we won't have updated the
remote bookmarks, so there is no need to try to prefetch the trees.
* Set `treemanifest.prefetchdraftparents` to False, as synced commits are not
usually checked out immediately, so there is no need to prefetch the tree.
Reviewed By: quark-zju
Differential Revision: D16054353
fbshipit-source-id: 0eeab49e5242fdddc593843ed67e6a17281ec912
Summary:
Add `treemanifest.prefetchdraftparents`, which controls whether or not to
prefetch the trees for the parents of newly added draft commits.
We will want to avoid this in commit cloud, as the prefetch may take too
long, and the user might not access the commits anyway.
Reviewed By: quark-zju
Differential Revision: D16054354
fbshipit-source-id: cf590451fb5252d215748567d6ad421c589ed581
Summary:
Previously I tried to not call to SVN if we have the 2 configs set. However I made a
mistake that the args will always be evulated anyway (computer science 101).
Let make it really lazy and add a test for that.
Reviewed By: ikostia
Differential Revision: D16163470
fbshipit-source-id: 6ec0f855b10164ae9a210bc70789b2f59fd19858
Summary:
On a shared host (like the devgpu), the shared cache will be used by many
users, and thus the repacklock needs to be able to be read/write by all the
users. Despite the code trying to open it as 664, the umask downgraded it to
644, only allowing the file creator to run repack.
Reviewed By: quark-zju
Differential Revision: D16151053
fbshipit-source-id: 4d3ecc19f2f6fbf9a8b831f7ea39d0a53bc4b8e9
Summary:
Fixing bugs I've discovered by enabling remote bookmarks sync for myself:
* turned out, adding "None" to the dict field in Python doesn't work as a null-value of optional field in C++ Thrift structure
* I didn't update lastsyncstate in some of the cases, also fixing
Reviewed By: mitrandir77
Differential Revision: D16131799
fbshipit-source-id: bdee423f519820d702719b3dfe76865a7547221e
Summary: It will be used in non-fsmonitor cases.
Reviewed By: markbt
Differential Revision: D15755286
fbshipit-source-id: 75ba7c5f227e1c9224c5d092b7e766afdf45c428
Summary:
The new log is typed and concise.
Old log:
[fsmonitor]> clock='c:1559600325:3956709:1:34762' len(nonnormal)=0
[fsmonitor]> setlastclock 'c:1559600325:3956709:1:36405'
[fsmonitor]> setlastisfresh False
[fsmonitor]> watchman returned ["x"]
[fsmonitor]> getlastclock 'c:1559600325:3956709:1:36405'
[fsmonitor]> set clock='c:1559600325:3956709:1:36405' notefiles=["x"]
New log:
[fsmonitor] clock: "c:1559600325:3956709:1:34762" -> "c:1559600325:3956709:1:36405"; need check: [] + ["x"]
In JSON form:
{"fsmonitor":{"new_clock":"c:1559600325:3956709:1:36425","new_files":{"len":1,"short_list":["x"]},"old_clock":"c:1559600325:3956709:1:34762"}
The new logging does not cover every information exposed by the old logging.
For example:
- Non-treestate events like why fsmonitor state gets invalidated.
Assuming most clients are on treestate now. These are removed.
- "fsmonitor_state = normal | unavailable | fresh" scuba logging. This can be
inferred, and will be added in a later diff.
- New "notefiles". The next "fsmoniotr" event will log the information.
Reviewed By: markbt
Differential Revision: D15710672
fbshipit-source-id: 5c4cad08c0072c7dc711e5c1e65aa7552940699e
Summary:
This has the exact same behavior as `hg debugdatapack` but on
indexedlogdatastore. Since nodes can be added concurrently to the indexedlog,
we can't verify that nodes aren't duplicated, so disable that logic.
Reviewed By: kulshrax
Differential Revision: D16073582
fbshipit-source-id: b7bbefdca8062c8b797432cb76b0de9df850148e
Summary: All the users are now using the Rust code.
Reviewed By: kulshrax
Differential Revision: D16039092
fbshipit-source-id: 1341ed33ab9a17f30b541b02eaa577db56959e31
Summary:
This allows switching hg debughistorypack to using Rust historypack code
instead of Python. Similarly to the iterentries for datapack, we collect all
the data in memory instead of using an iterator.
Reviewed By: kulshrax
Differential Revision: D16039098
fbshipit-source-id: 8b61c04e43a319a4601cc1aaae2eb46922d4a851
Summary:
This has now been entierely replaced by Rust code.
Goodbye.
Reviewed By: kulshrax
Differential Revision: D16039096
fbshipit-source-id: 8c3ce2eb5595a5f90c3ecab413a242a3d0afcb89
Summary: This is the last use of Python datapack in treemanifest
Reviewed By: kulshrax
Differential Revision: D16039101
fbshipit-source-id: 205b6fed0811ae86542f72aeed95cec81a61d741
Summary: This is the last place in remotefilelog where the Python datapack is used.
Reviewed By: kulshrax
Differential Revision: D16039095
fbshipit-source-id: 848157111231d85ba9b74cb7b5dc0631e9997ef5
Summary:
This is needed to switch `hg debugdatapack` to the Rust datapack. One major
difference with the Python code is that the Rust code builds a Vec instead of
an iterator. Implementing iterators with cpython-rust is hard (is it actually
possible?), but since the stored content isn't returned the Vec shouldn't use
too much memory.
For now, the implemention is tied to a packfile, but it shouldn't be too hard
to implement for the IndexedLogDataStore too.
Reviewed By: kulshrax
Differential Revision: D16039097
fbshipit-source-id: 567809785263bbf8cd3e1a9c24ecbb989b5c2496
Summary:
Similar to what was done in D15992903 for bad certificates, this diff adds a new `edenapi.tlshelp` config parameter that allows the administrator to set up a custom error message on TLS exceptions. Given that such exceptions are often caused by user misconfiguration, the error message can be set to point the user to some relevant troubleshooting steps.
This diff also changes the existing `edenapi.badcertmessage` config parameter to `edenapi.authhelp` to bring it in line with the new parameter, as well as the convention used for such configurable help messages by Commit Cloud. (This won't break anything since we have not shipped any configs that set `edenapi.badcertmessage` yet.)
Reviewed By: xavierd
Differential Revision: D16086608
fbshipit-source-id: ead83eefbd5e9bc1bf06a3aeccdc1f0737720871
Summary: Proxy errors are often due to something going wrong between the proxy and backing server. These kinds of errors are transient and the request usually succeeds upon retrying. As such, let's retry such errors (up to 3 times) before falling back to SSH.
Reviewed By: xavierd
Differential Revision: D16086240
fbshipit-source-id: c585a27d3f387bade1ae498587774e2f61566f1c
Summary: do not override remote bookmrks with all available on remote, if selective pull is enabled
Reviewed By: mitrandir77
Differential Revision: D16084215
fbshipit-source-id: ab0f1b393c620bc57b41805c0633ae793785ce55
Summary:
always pull only accessed bookmarks and default, do not take remote books from `remotenames` file
also adding handling of `CorruptedState` error to delete `selectivepullaccessedbookmarks` file in that case. it will help mercurial not to crash in case the file was corrupted and self-fix next time.
Reviewed By: mitrandir77
Differential Revision: D16079548
fbshipit-source-id: a2983ee63fb17d2922f3c230c3d6e36aa6591b62
Summary:
# RFC, not sure if I know what I am doing but I know my goal as follow
In case of the entire `.hg/svn` missing, in order to get the subdir and uuid information,
hgsubversion needs to talk to svn server. In our cases, it's polluting our logging and
it's hard to know who is really using svn server.
In this diff I want to disable this by writing these 2 information directly in config
and hgsubversion will just read from there.
Reviewed By: mitrandir77
Differential Revision: D15989449
fbshipit-source-id: fd9cead094245b3c0baf6e7e1099cc302899668f
Summary: We need to log (and in debug mode print out) Eden API exceptions in several places in the code. Let's factor out this logic into a function in the edenapi module.
Reviewed By: xavierd
Differential Revision: D16027147
fbshipit-source-id: 76c8e97dcaaa114e0c22448d117caae948fb60f4
Summary:
The `notice` arg should be provided to `ui.warn`, not the translation `_`
function.
Reviewed By: xavierd
Differential Revision: D16055534
fbshipit-source-id: bb1e0213147fc591671c2ef4c79aab3f9e4a2987
Summary: I added remotebookmarks field to the References object earlier, but forgot to change object constructions in couple of places
Reviewed By: markbt
Differential Revision: D16052722
fbshipit-source-id: b0b46739a89c911541eababd35b5f43812e66c4a
Summary:
By default, simplecache puts its local cache in a global location that persists
across test runs. That can cause surprises. Change the location to be inside
$TESTTMP under tests.
Reviewed By: xavierd
Differential Revision: D16036296
fbshipit-source-id: c73b573b87d49798f4ad146a1c0a559f5d94caf1
Summary:
On Windows, fp.tell() might return unreliable data for files with mixed reads
and writes.
Reviewed By: xavierd
Differential Revision: D16035564
fbshipit-source-id: d6cbb018e2664dd1e4323c52e7b0c73df07d7796
Summary:
`util.fdopen` now adds workarounds for read+write+seek files on Windows.
This should solve issues we have seen on Windows behaviors.
See https://www.mercurial-scm.org/repo/hg/rev/3686fa2b8eee for the Windows weirdness.
Here is a minimal program to reproduce the weirdness:
```
import os
f = open("a.txt", "wb+")
# Write 12 bytes
f.write(b"b" * 12)
# Read byte slice 2..5
f.seek(2, os.SEEK_SET)
data = f.read(3)
# Try SEEK_END
f.seek(0, os.SEEK_END)
print("%d (expect 12)" % f.tell()) # got 5 using some python.exe
```
Reviewed By: xavierd
Differential Revision: D16033678
fbshipit-source-id: 4f17c463d9bfcc0cdd38d1b15f2a9e38e5b4c132
Summary:
Add more shortcuts to common modules.
This make it easier to test changed code paths.
Reviewed By: markbt
Differential Revision: D15952263
fbshipit-source-id: c0eca6a61902d36a26a99f85e29dc70f431eca59
Summary: Rename `ApiErrorKind::BadCreds` to `ApiErrorKind::BadCertificate`. This makes it clearer to users who see the error name in log output that the problem is with a TLS client certificate, since "credentials" is otherwise pretty broad.
Reviewed By: xavierd
Differential Revision: D16024751
fbshipit-source-id: 1dea342036519a33dc48abaa41ab891be1a3637d
Summary:
The changes from commit cloud sync are supposed to be applied in-memory and committed with cloudsync-transaction to the disk. For example, for bookmarks the changes first are applied to the existing dict of bookmarks and with the transaction are being written to the disk into the `bookmarks` file.
Ideally I need to have similar thing for remote bookmarks. However, the current implementation provides of remotenames only read-only store with only possibilty to change something on the disk (`remotenames` file) and upload it to memory.
We're planning to rewrite the whole extension at some point, but currently to move forward I added a new method to the store, which will apply changes to the `remotenames` file and reload the remotenames store.
It's not transactional and uses existing function `saveremotenames`, which I refactored a little bit, so it could apply changes for multiple remotes at the same time. I also removed deletion of `remotedistance` file as it is deprecated and not used a long time.
Reviewed By: markbt
Differential Revision: D15921983
fbshipit-source-id: d6b2638db689c0c1b66a6291fc9f0f2c9dac978c
Summary:
Added implementation of the remote bookmarks merging.
1. If remote books changed in cloud since the last sync and local didn't, then local remote books will be updated with the cloud changes.
2. If otherwise, cloud remote books are the same and local are different from the last sync, then the local changes will be submitted to the cloud.
3. If both local and cloud remote books changed from the last sync, the merging becomes more difficult. There are several cases for what particular changes could be. The current implementation is the first and simpliest, so some of the decisions I made here may be improved later.
3.1. If both changed and not deleted, then the local remote book will be updated to the newest node between local and cloud's.
This is not always a correct state. For example, the local remote bookmark's commit is newer (further in the stack), but the bookmark was moved back in the stack on the server and the cloud now has this bookmark on the older commit. After syncing the local book will be kept as it was before and it's node will be submitted to the cloud.
However it's not that bad, as after `hg pull` it will point back to the right revision.
3.2. If one changed and another was deleted. There are two case when it could happen: (a) the bookmark was deleted on the server, (b) the client unsubscribed from the remote bookmark (functionality is not implemented yet). The current implmentation will just keep local remote bookmark as it is if it was deleted in the cloud and submit changes back to cloud.
Reviewed By: markbt
Differential Revision: D15899944
fbshipit-source-id: b17ca4d01fd1a1230d74588e2b9ca5c6d58df751
Summary:
Made changes to the commit cloud sync to support remote bookmarks. Here I only declared steps of future remote bookmarks sync and will add the implementation further in the stack.
Processing a merge state between cloud remote bookmarks state, last sync and local state is done earlier in the workflow, than, for example, for local bookmarks, because to update remote bookmarks we may need to fetch new public heads from the server and so need to get them first.
Reviewed By: mitrandir77
Differential Revision: D15898740
fbshipit-source-id: ca77f66b0f1244a811b4b549e5b189c5f1772708
Summary: Adding support of remote bookmarks to the get/set references of Commit Cloud client service.
Reviewed By: mitrandir77
Differential Revision: D15852515
fbshipit-source-id: 9d5331955f5fc95ecb4bfe8e060d3b1d67952a41
Summary: When Eden API falls back to SSH, log the exception type and message (in addition to the traceback) to the `hg_errors` table. Note that previously, we were attempting to log the error message but failing because we were using the wrong column name. (`msg` instead of `exception_msg`).
Reviewed By: xavierd
Differential Revision: D16014833
fbshipit-source-id: 06460574f66999b1293ea31b43b5c7ec89737144
Summary: Now that Eden API returns specific exception types rather than always raising a `RuntimeError` when things go wrong, the fallback path needs to be updated to catch all exception types, not just `RuntimeError`.
Reviewed By: xavierd
Differential Revision: D16013115
fbshipit-source-id: d5c2d88acede7e70519eda8915401bb8ee394038
Summary:
`--fixup` is not really a great flag name and it came from a legacy
implementation. Suggest `restack` instead.
Reviewed By: kulshrax
Differential Revision: D15978952
fbshipit-source-id: 4e8c2e79be01f6d8f13628409a446d42ae22c0af
Summary:
See the test change for what this change is about.
The new code has side effect on linearizing things for `<rebase_src>::`. This
is reflected on the second last test change. The new behavior is arguably more
desiable - The new code can fail if rebasing D onto C2 causes conflicts. If
that happens, and rebasing C+C2+D onto the new B does not cause conflicts, then
the old behavior is better. We can add fallback code to the old behavior if
there are conflicts later, by calling a plain `rebase` instead of `restack`.
Reviewed By: kulshrax
Differential Revision: D15978953
fbshipit-source-id: 20b9d62c6125ce2faf8e21bd86f6aad31ac38a0c
Summary: This allows the callsite to control what revs to rebase.
Reviewed By: kulshrax
Differential Revision: D15978954
fbshipit-source-id: 73174475411c3986f3a8e93a1a61ce5b857f454b
Summary: When the Eden API Rust client raises a CredsError exception, print out a configurable error message to the user (defined by the `edenapi.badcertmessage` config option). This allows us to provide specific instructions on how the user should renew their certificate.
Reviewed By: xavierd
Differential Revision: D15992903
fbshipit-source-id: 8316e33c3a5d86da272deea6402271d4a65548f4
Summary:
We've had these 2 options turned on for a while now. Let's stop pretending
we'll switch them off, and just hardcode them over the codebase.
Some places are still using either the C code, or the Python one explicitely,
future changes will switch them to the Rust code.
Reviewed By: kulshrax
Differential Revision: D15981631
fbshipit-source-id: c23e474a84e887a2b92c9a304d63ef8d680cdf2d
Summary:
The requirements field is only interesting for non-remotefilelog or
generaldelta repos. Nowadays it's probably safe to assume clients are
remotefilelog and "requirements" is less useful.
Reviewed By: xavierd
Differential Revision: D15710676
fbshipit-source-id: 65f04b64760c652432471c4a8dda7acc4cf45466
Summary:
The in-core blackbox command displays blackbox entries in the given time range.
The `blackbox` command provided by the blackbox extension was removed. They
can still be accessed via `.hg/blackbox.log`, though.
Tests are updated. Most `| grep` patterns were changed to use structured pattern
matching `--pattern` instead. Tests that are not interesting (ex. bundlebackup,
since we are moving away from bundle files slowly) are just removed.
Reviewed By: markbt
Differential Revision: D15640718
fbshipit-source-id: 7e5da60ca2b15ae9495d0242b340a066979d5a4f
Summary:
Add a `mercurial.blackbox` module which just delegates to the Rust
binding. This means blackbox is no longer optional.
Change `ui.log` to log to the native blackbox as `LegacyLog` event.
The plan is to slowly migrate users from `ui.log` to `blackbox.log`,
which supports well-defined event types (instead of `LegacyLog`).
This does not change the `blackbox` command, which still uses the
legacy blackbox implementation. That will be changed in a later diff.
Reviewed By: markbt
Differential Revision: D15640720
fbshipit-source-id: de171f46e1430060083c9b7aee0a96dde315d021
Summary: This diff adds wrapper class around the Rust `edenapi.client` class defined in the bindings crate whose purpose is to catch any exceptions raised by the Rust bindings. For now, the wrapper class simply re-raises all exceptions; later diffs will use this to add special-case handling for certain exception types. (For example, for showing helpful messages when an error is due to a client certificate issue.)
Reviewed By: quark-zju
Differential Revision: D15975435
fbshipit-source-id: 21cef44dac84d5f468cb0e1d9eaddc2a1e93bfda
Summary:
The getpackv1 API doesn't support LFS blobs as no metadata flags are sent over
the wire. This means that repositories like ovrsource can't use it, which
prevents these repos from moving away from loosefiles...
The getpackv2 is similar to getpackv1, except that it also sends the metadata
alongside the data. This means that the LFS flag is properly sent over the
wire.
Reviewed By: DurhamG
Differential Revision: D15905425
fbshipit-source-id: dab8dcb75c8d9db8c661f11f87318feca7d0f6a9
Summary:
The getpackv1 is never sending delta chains, as the full content is always
requested. We don't have plans to change this, so let's simplify the code a
bit.
Reviewed By: DurhamG
Differential Revision: D15905431
fbshipit-source-id: f853da08391185b525d8b3da2382519b5f8d76b2
Summary:
Use _n for plurals of strings, rather than hard-coding the suffix, and don't
retranslate the string after is has already been translated.
Reviewed By: kulshrax
Differential Revision: D15940244
fbshipit-source-id: 26ef175fe201bcc5211ad05c8b3894fbc22d61b6
Summary:
In S179901, we've seen a couple of instances of `hg status` consuming as much
memory as they could until getting OOM killed. Looking at backtraces for these
processes (P67246356) the code spins trying to read data from memcache. The
connection is however closed and therefore `recv` returns an empty string,
which is then added to a list and the `recv` is retried again.
Reviewed By: kulshrax
Differential Revision: D15947154
fbshipit-source-id: f1216e26b0ce159377ff8e82bedc13a7cc3d0444
Summary:
This just moves things around. So native and pure Python modules are split to
different Python packages. This makes it possible to use the standard zip
importer without hacks (ex. `hgdemandimport/embeddedimport`).
This diff is mostly about moving things. While `make local` still works,
it does break nupkg build, which will be fixed in a later diff.
Reviewed By: kulshrax
Differential Revision: D15798642
fbshipit-source-id: 5d83f17099aa198df0acd5b7a99667e2f35fe7b4
Summary:
When cloud sync pulls in new commits from commitcloud, it also adds them to the
backedup heads, so that the next backup doesn't try to push them to the server
again. However, the pull happens in a transaction, which may fail. If it does
fail, the backedup heads are still recorded.
Update the write to the backedup heads file to take place inside the
transaction in this case, so the update only happens if the transaction
succeeds.
If any unavailable heads have been recorded, ignore them.
Reviewed By: mitrandir77
Differential Revision: D15898056
fbshipit-source-id: e7affce1b10f507e2ee924d9bc78000c835ccd65
Summary: Previously, there was a single config option (`edenapi.streaming`) that would turn on streaming responses for all Eden API operations. It turns out that not all operations benefit from streaming, and in some cases, streaming actually hurts performance. As such, we need more fine-grained control of whether streaming is enabled. This diff adds options for each major operation (fetching data, history, and trees) to toggle streaming as desired.
Reviewed By: xavierd
Differential Revision: D15838619
fbshipit-source-id: ce5eb70f8b391788405af1c90de9009876e9f1f9
Summary:
Verify that the behavior of the Rust history store is the same as the Python
one. A handful of changes needed to be made to both to account for small
difference in behavior regarding getancestors, and empty copyfrom paths.
Reviewed By: kulshrax
Differential Revision: D15721524
fbshipit-source-id: e03b7e8dcbc6f9f1941c64536b8de39b77f637c9
Summary:
When format.userustmutablestore is turned on, all the history data will be
written into a Rust mutable history store.
Reviewed By: kulshrax
Differential Revision: D15717844
fbshipit-source-id: 417ec44f4b6f9566ef4c8d46c35b1e2dca7e583a
Summary:
Now that linkrev are no longer added to mutablehistorypack, let's remove the
code that deals with it. This will enable using a Rust based historystore in
the future.
Reviewed By: kulshrax
Differential Revision: D15684623
fbshipit-source-id: d305e01cdfc0366c9432e0418854fd3b43a35539
Summary:
In order to provide fast "hg blame" for a file, Mercurial keeps an auxiliary
data attached to every filenode: the linknode. This linknode is a backpointer
to the revision that introduced the linknode.
While the linknode allows for very fast blame, it also has a big drawback: the
commit DAG has cycles. This makes for an awkward commit logic where the data
needed by a commit also needs the commit hash.
The solution that was taken up to now was to delay the write to the historypack
until it's closed and use "linkrev" until then. By the time the historypack is
closed the changelog will have been updated and the linkrev can be resolved to
a valid linknode. The drawback of this solution is that the historypack (and
other future stores) needs to have an understanding of what the changelog is to
be able to interpret linkrevs. It also prevents the store from being able to
blindly store data on disk as it is received.
For now, let's solve this by moving the handling of linkrev one layer up, and
recognizing when a linkrev is used and simply waiting until the changelog is
updated before adding the resolved linknode to the historypack.
A better solution to this problem is to simply not have a linknode in the
historypack, and instead have an auxiliary file where linknodes are stored.
This will remove the chicken-egg issue as this auxiliary file can be written
after the changelog is updated.
Most of the logic in this patch was inspired by a similar hack in
remotefilelog.
Reviewed By: DurhamG
Differential Revision: D15684624
fbshipit-source-id: 578c42349d6c6f572ff578de0495f2e762814cd2
Summary:
Enabled selective pull for myself and found issue with the passing positional args in the remotename wrapper of `exchange.py:pull(..)`.
`hg pull` failed because `heads` were passing uboth as positional args and as kwaargs. Fixed the issue explicitly specified args of the function.
Reviewed By: mitrandir77
Differential Revision: D15855502
fbshipit-source-id: 91b9bcfc122db069f9c6382e50f3185736103202
Summary:
This adds a new queue to replay scratch bookmark changes into Mononoke, which will allow us to replay them there.
There's not a lot going on in Mercurial (i.e. in this diff) to achieve this: we simply record bookmark changes as they happen in infinitepush, and allow for excluding some bookmarks. Specifically, we'll want to exclude backup branches, which we don't want to copy, since a) there's way too many of them and b) they're deprecated in favor of Commit Cloud.
Currently, this does not allow for replaying deletions. That would require further rework of how we delete things, since right now we do it my matching on bookmark names in the DB, which means the Python side of things is not aware of which bookmarks exactly were deleted. I'm not aware of how much use this is currently getting, but I'll research that and add it in if necessary.
Finally, one thing that's worth calling out here is the `bookmark_hash` column in this table. This is here in case we need to scale out the replication of bookmarks across multiple workers.
Indeed, we'll always want the replication of any given bookmark to happen sequentially, so we should perform it in a single worker. However, if we have too many bookmarks to replicate, then that could become a bottleneck. If that happens, we'll want to scale out workers, which we can do by having each worker operate on separate bookmarks.
The `bookmark_hash` column allows us to evenly divide up the space of bookmarks across workers if that becomes necessary (e.g. we could have 16 workers: one for each first hex digit of the hash). We won't use `bookmark_hash` immediately, but since it's very cheap to add (just compute one hash in Mercurial and put it in the table), I'm adding it in this diff now in case we need it later to avoid the friction of having to re-redeploy hg servers for that.
Reviewed By: StanislavGlebik
Differential Revision: D15778665
fbshipit-source-id: c34898c1a66e5bec08663a0887adca263222300d
Summary:
There is no commit associated with the Subversion revision 0 and therefore, we
can bail out early in this case.
Reviewed By: kulshrax
Differential Revision: D15838376
fbshipit-source-id: bed3bfb8af4ac177a67f33078778beccc83f0573
Summary: This diff adds an `edenapi.streaming` config option; when set, Mercurial will request streaming responses from the API server for Eden API operations, and will handle the results as a stream of individually-serialized CBOR values.
Reviewed By: xavierd
Differential Revision: D15788328
fbshipit-source-id: e5eeb573bc6b4f0382ae545547670e8308306bca
Summary: The hg pasterage is the command that people run to share debug info when something is wrong with their repository. Having the information about CommitCloud related stuff will help with all the future debugging of that.
Reviewed By: mitrandir77
Differential Revision: D15744065
fbshipit-source-id: 094ccdf79c38fed78f5106a1617a5af09e1870e8
Summary: The `osutil` module is re-exported by `util`. Use `util` instead.
Reviewed By: xavierd
Differential Revision: D15798643
fbshipit-source-id: 3b5ec4ce737664d70859889b1aaa18d09c2feb19
Summary:
The multiplexing has been on for hg_dev hosts for a while, with no issues,
let's use it exclusively when the indexedlogdatastore config is set.
Reviewed By: kulshrax
Differential Revision: D15763914
fbshipit-source-id: 9d33c215f700c57bf3cf0d8e55117b1020bb422a
Summary:
We've had this on for the hg_dev servers for a while, with no notable issues.
Let's move it out of the experimental folder.
Reviewed By: kulshrax
Differential Revision: D15763915
fbshipit-source-id: ff1059d0b77fd8b564f4d02753606f2800146949
Summary:
Let's avoid the copy/paste of code. This will also allow to move it out of the
experimental path more easily.
Reviewed By: kulshrax
Differential Revision: D15763917
fbshipit-source-id: 4f61bfa8562cdd836d61efc96c180908f07ea74f
Summary: This adds (behing a feature flag) the option for Mercurial servers to record bundles they ingest into a `forwardfillerqueue`. This queue is consumed by the commit cloud forwardfiller (see stack rooted at D15712911 for that) to replay those commits into Mononoke.
Reviewed By: farnz
Differential Revision: D15759638
fbshipit-source-id: 70660343408bbaa865fc8f51f49f590eaf525cce
Summary:
With listserverbookmarks in hg, we have to wait for an hg release to be able to run our monitoring, which isn't ideal.
Furthermore, the listserverbookmarks method command is unlikely to be used by anyone else, so we might as well provide it directly for review-bookmarks (like we do in e.g. the commit cloud backfiller for, though if we change our minds later we can always just revert this diff and the corresponding fbpkg change).
Reviewed By: farnz
Differential Revision: D15778340
fbshipit-source-id: d8fd6c9e1d33a420f95bdb4fadb582b864c51d5f
Summary:
After asking hg_memcache_client to fetch some data, we need to refresh the
underlying store. This refresh needs to be done after hg_memcache_client wrote
to it, not after we do the request.
Reviewed By: quark-zju
Differential Revision: D15763916
fbshipit-source-id: 99e88ffa13dba4394344c9340eba3f12fcfe0b76
Summary:
This adds a listserverbookmarks command to list bookmarks at a path without a
repository.
Reviewed By: ikostia
Differential Revision: D15737661
fbshipit-source-id: e2cd5dbcfe5c24868dd1d2b6685ebe13b1762676
Summary: Update the messages in the debug output for HTTP data fetching to specify whether we're downloading file content or file history.
Reviewed By: xavierd
Differential Revision: D15749426
fbshipit-source-id: e6fdff87d2cdc439549773c85a83341650652755
Summary: Print out download stats for HTTP tree fetching when `edenapi.debug` is set.
Reviewed By: xavierd
Differential Revision: D15746069
fbshipit-source-id: ab7f5bc67f4af64116a3095dd4424cfef9198f80
Summary:
In order to not have to deal with linkrev in mutablehistorypack, let's split
_addtopack in 2. This will allow _addtreeentry to be wrapped later to deal with
linkrev.
Reviewed By: kulshrax
Differential Revision: D15684625
fbshipit-source-id: 772ffa25ce7b72f95f057d0ae00c89c7dc91bf18
Summary:
Selective pull, when just enabled, will filter only remote bookmark for the active remote repo at that time. When user will try to pull next time from the different remote, bookmarks for that remote won't be filtered, because the selective pull will think that it's already enabled and the remotenames file already has only interesting bookmarks.
I changed `selectivepullenabled` file to keep track of the remotes which selectivepull has been enabled for.
Reviewed By: markbt
Differential Revision: D15583506
fbshipit-source-id: dec09baf1e1a0c80d1d5987ac6f85fe1202b2dab
Summary: Now slective pull, when enabled, will keep not only configured remote bookmarks, but accessed, that were recoreded earlier, as well.
Reviewed By: markbt
Differential Revision: D15583505
fbshipit-source-id: 8c8d300afc333a94427513d9844c1c1af3cf7f32
Summary:
If selective pull is enabled, the old implementation of `--remote` doesn't show all available remote bookmarks, but only locally available ones.
New implementation uses `listkeys` API to fetch remote bookmarks from the server.
However because `remote-path` flag belongs to the infinitepush extension, if the extension is not enabled, than `--remote` will show only bookmarks from the default remote.
Reviewed By: markbt
Differential Revision: D15547513
fbshipit-source-id: b7c79a0cc7237bbf086c64a71d6ed313aa520582
Summary:
With selective pull feature we need to have ability to list locally available remote bookmarks - the bookmarks the user is subscribed to.
As current implementation of `--remote` does exactly that, pointing subscription to the same implementation.
Reviewed By: markbt
Differential Revision: D15547515
fbshipit-source-id: d3ac7295fee3391076855fbe1bed00124dab973f
Summary: When selective pull will be enabled, `hg log` won't be able to show any information about the remote bookmarks that are not in subscriptions for particular user. So we need to hint the user that they may want to explicitly pull the remote bookmark first, if hg log fails to find it.
Reviewed By: quark-zju
Differential Revision: D15516462
fbshipit-source-id: 5be77b0048d8e175a737f76a8e89768f4c837f60
Summary:
As we're working towards moving exclusively to Rust based mutable stores, let's
modify the existing datapack unit tests to test them.
Reviewed By: kulshrax
Differential Revision: D15673688
fbshipit-source-id: 847ec0f5f5316e6f8ba72f27a7665dd3bf46c5fc
Summary: Now that EdenAPI's methods take in a MutableDeltaStore to perform writes, EdenAPI no longer needs to know about the cache path, as this is an implementation detail of the store. This field was no longer used anywhere (except by the constructor code which created the cache directory if it didn't exist -- this behavior is now handled by the MutableDeltaStore), so let's remove it.
Reviewed By: quark-zju
Differential Revision: D15715777
fbshipit-source-id: 11908c6be49f8846e1b7ba5890200d08529441e8
Summary: Add a new command, `hg debugserialgetfiles`, which works just like `hg debuggetfiles` except that each file is fetched serially via a separate call to `edneapi`'s `get_files` method. This is designed to simulate a pathological case where Mercurial performs a series of individual file fetches, depriving the Eden API client of the opportunity to use batching, etc. In particular, this command is intended to be used to test TCP connection reuse inside of libcurl.
Reviewed By: quark-zju
Differential Revision: D15707993
fbshipit-source-id: d57550639839ed597347228059f51ed343151867
Summary: Use a formatting function to pretty print byte counts in progress bars for HTTP data fetching. Also correctly pluralize the progress bar message. (i.e., treat "1 file" as a special case.)
Reviewed By: quark-zju
Differential Revision: D15700946
fbshipit-source-id: d2baaa757787ebeeb94d4b987128d8fafd12f19b
Summary: This diff adds a new `treemanifest.usehttp` config option. When set, Mercurial will attempt to prefetch trees from the API server, falling back to SSH if HTTP fetching fails.
Reviewed By: markbt
Differential Revision: D15506807
fbshipit-source-id: a5c5fcf14f29aee083c9fa0daeb0aedb5c1ec9a9
Summary: D13838772 introduce validation of hashes when fetching data from mercurial/memcache. We'd like to get insights in to how often this would happen, and potentially attach alarms to notify.
Differential Revision: D15472456
fbshipit-source-id: ad2f28e91824e4bbfc02284f230111e2d4cb3f00
Summary:
The clients don't have any mapping from `globalrev->hash` locally but
they can do this lookup quickly using ScmQuery. We do not want to invest too
much effort on supporting globalrev based lookups on the clients as this is a
workflow we want to discourage anyway. But, for now, existing workflows may
break if this lookup is slow on the clients. Therefore, for the interim, lets
just have this lookup backed by ScmQuery. We can later improve or completely
discard this based on the future direction for globalrevs.
Reviewed By: quark-zju
Differential Revision: D15588420
fbshipit-source-id: 61f91414248ca1defe6eac4311243ee8029a92cf
Summary:
We will eventually get rid of the `hgsubversion` extension and
therefore, we want to remove any dependency on it in the long term. This commit
removes any dependency on the `hgsubversion` extension for resolving the revset
string corresponding to the globalrev.
Reviewed By: quark-zju
Differential Revision: D15586218
fbshipit-source-id: 79685ce96075a162d40ce2ce48d347333d768d7a
Summary:
Support bundlerepos with mutation data.
Infinitepush is currently the only place where bundles may include mutation
data. This is used when rebundling infinitepush bundles to send back to the
client.
Reviewed By: mitrandir77
Differential Revision: D15604475
fbshipit-source-id: 111a3211044c742d3d1c75d06c6e6114f754e396
Summary:
The tempfile crate creates temporary files with the mode 0600 and thus when one
is persisted, it keeps these permissions. This effectively prevents a shared
indexedlog to be used by anybody but the owner.
Reviewed By: quark-zju
Differential Revision: D15580364
fbshipit-source-id: 06ca3872287b618bfbab28327644a143db7282c9
Summary:
Update `hg githelp -- git branch -d` suggestions to suggest `hg hide -B`.
Also fix up the code so that trailing `-B` options are not produced.
Reviewed By: farnz
Differential Revision: D15575864
fbshipit-source-id: bd0d0b23f2a509e02554bcbb453718bebe8ea11a
Summary:
Add support for `hg hide -B bookmarkname`. This hides all commits that are
uniquely reachable by the provided bookmarks. This means if a bookmark is at
the head of a stack, `hg hide -B bookmark` will hide the stack.
Reviewed By: quark-zju
Differential Revision: D15562715
fbshipit-source-id: 9fdb3383b534faea982396a4f4782c03e4910dc3
Summary:
Make the code for calculating which revisions the `-B` options to `hg prune`
and `hg debugstrip` affect common to both commands.
The remotenames modification of `repair.stripbmrevset` is wrong. It should
exclude all ancestors of commits that have remotenames, not just those that
don't share a bookmark with the remotename. Fix it.
Reviewed By: quark-zju
Differential Revision: D15562716
fbshipit-source-id: 507002505100e7d60c395f242cc8e1062b91cc20
Summary:
Update all rust crates that compile on Rust 2018 to use the 2018 edition.
The `commitcloudsubscriber` crate only compiles with Rust 2015, so make that
explicit in `Cargo.toml`.
Reviewed By: farnz
Differential Revision: D15601648
fbshipit-source-id: 7380e6e695fc3049913af91fcbde105dfe1be4bc
Summary:
This will allow us to efficiently identify scratch pushes that are not properly being pushed to both Mercurial and Mononoke.
When we roll out `infinitepush-other`, this will be helpful to make sure all repositories are properly configured to push to both destinations.
Reviewed By: farnz
Differential Revision: D15577201
fbshipit-source-id: 17912b90a934f65ed21851170402ae498b127c14
Summary:
Through Scuba, we can identify quite a few users that are doing `hg push default` for an infinitepush: https://fburl.com/scuba/6vz3ien7.
This is typically happening in repositories where the `default-push` destination is not the hg repository.
In fact, we actually prompt users to use hg push default if they try to perform a scratch push to a svn repository a few lines below!
markbt suggested we should just replicate the push in this case as well, which I think is a good idea.
Simply replicating the push in this case is not quite enough, however!
Indeed, for our infinitepushes to go to both Mercurial and Mononoke, we would have to maintain an number of consistency rules between `default`, `infinitepush`, and `infinitepush-other`:
- `default` and `infinitepush-other` must point to opposite repositories (i.e. one Mercurial, one Mononoke), to serve users using `hg push default`.
- `infinitepush` and `infinitepush-other` must point to opposite repositories, to serve users using `hg push` without a path.
This effectively means `default` and `infinitepush` must be the same, and `infinitepush-other` must be the other. We could do this with path markers, but this is getting so complicated that I don't foresee us getting it right.
At the end of the day, the root cause of this complexity stems from the fact that we're using a destination that is now effectively a read path (`default`) for infinitepush writes. I think this was perfectly valid when default meant "Mercurial, not subversion", but not so much now that we have Mononoke in the mix.
Of course, we could just update the message and ask our users to use `hg push infinitepush` instead, but this would mess with their muscle memory as well as their shell history (not to mention that `hg push default` would silently do the wrong thing).
So, this patch updates the code to use the infinitepush "write" destination for writes when that is what the user intended.
---
As a side note, note that the current behavior is actually a little broken. Indeed, if one were to do `hg push default --to scratch/$FOO` while Mononoke is serving reads, the scratch would go to Mononoke. In this scenario, the scratch push would in theory have been accepted but would have had its bookmarks discarded (at least until we support htose in Mononoke :) ).
This probably never happened in practice, however, for two reasons:
- Most users and systems that actively push scratch bookmarks are actually pushing to a specific `hg.vip` path instead of `default` (one notable exception is On Demand WWW).
- `hg push` to Mononoke for a scratch bookmark doesn't actually work if you have the pushrebase extension code loaded in some way (either by enabling the extension, or enabling an extension that uses it). See D15576199 for the details.
Reviewed By: farnz
Differential Revision: D15576545
fbshipit-source-id: c28b808632505bb8e8f4d114029f7d8c17c9749e
Summary:
We will eventually get rid of the `hgsubversion` extension and
therefore, we want to remove any dependency on it in the long term. This commit
removes any dependency on the `hgsubversion` extension for retrieving the
`globalrev` corresponding to a commit. Note that the `globalrev` for the older
commits is just the `svnrev`.
Reviewed By: quark-zju
Differential Revision: D15579401
fbshipit-source-id: 021e6d93d008b07591e1b7801062d9c5fbfc7651
Summary:
Infinitepush is not a pushrebase, so we should not send the b2x:commonheads part when we do an infinitepush.
Mercurial doesn't actually care about this, but Mononoke does (because it peeks at the stream to know if we're dealing with a pushrebase or a regular push).
Reviewed By: ikostia
Differential Revision: D15576199
fbshipit-source-id: a90adb5884b00ad65208697fa17983535aacb9c6
Summary:
In D15468485 calls to read the backup state started taking the repo lock to
ensure a consistent state. This had the unintended side-effect of making
revset predicates like `backedup` take the lock on what would normally be a
read-only path.
It's only necessary that sync and backup get a consistent view - it's ok if
logging functions see a slightly stale view of the repo. Move the lock out of
the `BackupState` constructor to the places it is called in sync and backup.
Reviewed By: mitrandir77
Differential Revision: D15575458
fbshipit-source-id: c4f2a62839e5f22a64ff6d6a4e78ef4c65cb8b15
Summary:
This commit just introduces a method for retrieving a mirrored
revision using the relevant ScmQuery API. The eventual objective is to use this
method to also be able to get the `hash` corresponding to a `globalrev` on the
Mercurial clients.
Reviewed By: quark-zju
Differential Revision: D15571470
fbshipit-source-id: cc0506356d27450594a690aa29a2a2a608aac5c0
Summary:
The former will always succeed on Windows, while the later raise an Exception
when the file is opened elsewhere.
Reviewed By: quark-zju
Differential Revision: D15571023
fbshipit-source-id: b7ee739dce49f9a9ba5087ec785a6dc332eb88c8
Summary:
On Windows, keeping the files open prevent them from being deleted.
Make sure we close them temporary packfiles on abort so they can be
removed properly.
Differential Revision: D15550431
fbshipit-source-id: eca557ffd852ef0255fe991c50ce35a6a8ee7fa0
Summary: This updates `hg push` to replicate pushes to the `infinitepush-other` path. This ensures that we replicate pushes to scratch branches to Mononoke once we enable this mechanism.
Reviewed By: markbt
Differential Revision: D15538790
fbshipit-source-id: 0baa31f26f516cf5d2f6ec5a14d8006c912766c2
Summary:
When writing a blackbox log entry, only include the current commit if the repo
has already loaded the changelog. Otherwise, finding out what the current
commit is will load the changelog, which is wasteful, and may cause a
re-entrant call back into blackbox logging.
Reviewed By: xavierd
Differential Revision: D15539413
fbshipit-source-id: 48c27b053b9db7ce5c2d4b1991020adaea720c21
Summary:
When computing which of the lastsyncstate heads should be available in the repo,
use the omittedheads from the lastsyncstate, not the new omitted heads.
Reviewed By: mitrandir77
Differential Revision: D15535780
fbshipit-source-id: 215f037215b20d1188999abfa42e7a45391b7723
Summary:
We have seen reports of this file getting corrupt on Windows when
peoples' machines are hard-rebooted. Handle that corruption with a
more helpful error message.
Reviewed By: quark-zju
Differential Revision: D15506345
fbshipit-source-id: 66d998a156bc11953b5afb440270ff77af4677fc
Summary:
The Rust code doesn't have a close method, but it has a flush one. This has the
drawback of removing the ledger argument from the close method which is used to
guarantee that data gets into the packfile before the rename operation.
For now, let's always sync the file content before closing the mutable
packfile.
Reviewed By: kulshrax
Differential Revision: D15504680
fbshipit-source-id: 09dbda8820351b128038d6c3ea1843fef9c5bc48
Summary:
By using a wrapper over them, it becomes easier to switch the underlying
implementation to be a Rust one.
Reviewed By: kulshrax
Differential Revision: D15504679
fbshipit-source-id: ac87311e015bd4c3baafb8e8e77ab6ed4890cdcc
Summary:
We have to use globs in tests everywhere they output durations. It's
annoying when you have to update many tests, so let's force the output to 0 when
in tests.
Reviewed By: quark-zju
Differential Revision: D15382069
fbshipit-source-id: eaed82e2ba685d594267433494d98fce5de3663e
Summary:
Now that all our repos are treemanifest, let's enable the extension by
default in tests. Once we're certain no one needs it in production we'll also
make it the default in core Mercurial.
This diff includes a minor fix in treemanifest to be aware of always-enabled
extensions. It won't matter until we actually add treemanifest to the list of
default enabled extensions, but I caught this while testing things.
Reviewed By: ikostia
Differential Revision: D15030253
fbshipit-source-id: d8361f915928b6ad90665e6ed330c1df5c8d8d86
Summary:
Cloud sync backs up commits before it takes the lock. If, during this backup
process, another transaction adds new commits to the repository, the cloud sync
process will not be able to sync properly, as some of the commits won't have
been backed up.
Furthermore, because the backup state may have been updated between opening the
changelog and reading the backup state, which means the backup state may
reference commits that are not available in the changelog to the current
process.
Serialize these operations by:
* Reading the backup state file under the repo lock. This, unfortunately, means
`hg cloud backup` needs to take the lock, but it should only be for a very
brief period while a file is read.
* When cloud sync has completed the backup and is about to start syncing, check
that the repository hasn't changed while the backup was running. If it has,
abandon this sync attempt. Another background sync will have been scheduled
that will backup the new commits and then sync everything.
Reviewed By: quark-zju
Differential Revision: D15468485
fbshipit-source-id: 706ada05dfb1f539638104722a044493c7e3e62f
Summary:
Ensure cloud sync gets a consistent view of the repository, and that the sync
operation is a single update, by taking in the lock and using a transaction for
the whole cloud sync.
The commits are backed up before we take the lock, so for syncs that don't pull
in any new commits, this should still be relatively fast: we will only have the
lock for the duration of the request to the commit cloud service. In the case
that we do need to pull new commits, we were already taking the lock anyway.
Reviewed By: quark-zju
Differential Revision: D15468487
fbshipit-source-id: 37abeac85dad8d2cfb4ad160c5f417dde087d74c
Summary:
If a ui object is not available, then callers can't make blackbox logs. However,
the blackbox extension has `lastui()` which lets it find a ui object.
Add `util.log`, a hook point for blackbox that lets callers log to blackbox without
having a ui object to hand.
Reviewed By: quark-zju
Differential Revision: D15495215
fbshipit-source-id: b468647ccacc6992f71b9a8be2552db860d16ebe
Summary: Use `util.timer` to time commands, to make these times stable in tests.
Reviewed By: quark-zju
Differential Revision: D15468482
fbshipit-source-id: 7d15d725ba436915dff5da514398514dd0b113f7
Summary:
This extention will return 0 if the bookmark points to an expected hash and
non-0 otherwise. We plan to use it in Mononoke.
Reviewed By: StanislavGlebik
Differential Revision: D15450801
fbshipit-source-id: 36d938fb6912046ada229a4e35b76e9cf13f0a74
Summary:
Now that all our repos are treeonly, let's default the extension to
using treeonly. This opens the doors for moving all the tests to operating on
treeonly repos in a future diff.
Reviewed By: quark-zju
Differential Revision: D15030251
fbshipit-source-id: cd289775174bbacd6a6cdc7a5b07fe7e41238584
Summary:
Previously readfast was an optimization that sometimes returned a delta
against p1, but other times returned the full manifest. This was weird and
caused certain algorithms to sometimes be O(changes) but other times O(repo).
In a tree world we no longer need this optimization, so let's drop it.
readdelta is similar in that it would read the difference between a manifest and
it's delta base. This has no relationship to the delta between a manifest and
it's parent, so it's weird and we should get rid of it.
There is a legitimate use case for wanting to know what entries are new in a
manifest, like when deciding what to send over the wire. So let's add a new
readnew() function that is explicitly for reading what entries were introduced
by this tree. The implementation is basically the same as readdelta, but
deltaing against p1 instead of the deltabase.
Reviewed By: markbt
Differential Revision: D15344434
fbshipit-source-id: dc8dca326f66b2fc55cc76f93c7ce48aa7efedf3
Summary:
This makes it easier for a wrapping job to associate output with a given command it ran.
Note that creating a SSH peer creates its own copy of the `ui` object, which is why we have to push a buffer to both `ui` instances.
Reviewed By: ikostia
Differential Revision: D15468456
fbshipit-source-id: c6f1937749447e27332801577538d9874eb18898
Summary: Add progress bars to remotefilelog's debug commands for HTTP data fetching. This makes it obvious whether there is activity occurring or whether Mercurial is stuck.
Reviewed By: quark-zju
Differential Revision: D15480683
fbshipit-source-id: 8c67c306488e5ec82a32143eb1748879474e3ec9
Summary:
While mutable datapack are used in several places, most of the writes are
coming through the pendingmutablepack class, so let's start switching these
to use the Rust code.
Reviewed By: kulshrax
Differential Revision: D15482770
fbshipit-source-id: d0e4aae38a5ef2ba625a40ff907420623e6efd11
Summary:
Both the Rust and Python mutable stores support a flush method, but while
Python support an abort method, Rust will simply cleanup temporary files when
the object is destroyed. We can implement a similar scheme in Python by
implementing __del__.
Reviewed By: kulshrax
Differential Revision: D15482771
fbshipit-source-id: 4c9647ab7dd9fa078a8fc3c8b00962fbc0823dfd
Summary:
Opening all the revlogs to validate them is showing up as a performance
bottleneck for hgsql transactions. Let's switch them to using mmap to avoid
reading a bunch of data from disk.
Actually making this have an effect on our servers will also require setting the
experimental.mmapindexthreshold config.
Reviewed By: quark-zju
Differential Revision: D15441867
fbshipit-source-id: 4edde0bc3419ef75f82a4234c9dfc6604c6db9f4
Summary: Print out download stats when `edenapi.debug` is set. Pretty printing of the stats has been improved to make these numbers more useful to humans.
Reviewed By: xavierd
Differential Revision: D15466702
fbshipit-source-id: 6fac9ca5976b98874fc7a5f9b89d42975e1520ce
Summary:
When EdenAPI encounters an error, the current behavior is to print an error message and then fall back to SSH. In addition to the fallback notice, we were previously printing the exception message using `ui.develwarn`, with the understanding that this information is only really useful to Mercurial developers.
Instead of using `develwarn`, let's use `ui.warn` but gate the call so that only users with `edenapi.debug` set to `True` see the error message.
I initially tried printing the entire backtrace with `ui.traceback`, but this proved too verbose to print out in the middle of normal command output. The full traceback is logged to Scuba, and can be obtained from the `hg_errors` Scuba table, so just logging the exception message should be sufficient.
Reviewed By: quark-zju
Differential Revision: D15466332
fbshipit-source-id: 43d40eff47e4f68fe860383e7bd519fbd0052830
Summary: On the Source Control team, we've been referring to the project to replace parts of the Mercurial wire protocol with the Mononoke API server as "HTTP data fetching". This has caused confusion for other teams in the past (particularly teams with a security-related focus), as this nomenclature makes it seem as if we're using unsecured HTTP. (This is not true; we are always using HTTPS.) In light of this, let's make sure that user-facing messages always use the term "HTTPS" instead of "HTTP" to avoid this confusion.
Reviewed By: quark-zju
Differential Revision: D15465390
fbshipit-source-id: b556ed222dc01979ceaa0d4a7aa921cb1c38d75e
Summary: Log metrics about HTTP data fetching to `dev_command_timers`, and log HTTP fetching errors to `hg_errors`.
Reviewed By: quark-zju
Differential Revision: D15464791
fbshipit-source-id: 1027383b6aa0dc6915351332bfbc2d20d540cc4e
Summary: Check that the user's configured TLS credentials exist during client setup and disable HTTP if they aren't present.
Reviewed By: quark-zju
Differential Revision: D15459917
fbshipit-source-id: f20664c6522e47f2960cec1f02ef1a5f4c7e2c8c
Summary: Add several methods to the DownloadStats FFI wrapper to allow Python code to easily access various metrics.
Reviewed By: xavierd
Differential Revision: D15458802
fbshipit-source-id: 9e19d2a9b3fcb6e3a066f040fd110510a2f0d63e
Summary: Make all of the EdenAPI data fetching methods return an object containing metrics. These can then be logged by the Python code.
Reviewed By: xavierd
Differential Revision: D15440641
fbshipit-source-id: 4a5fd090066a9020ae32986ab45ee8fb70c8de53
Summary:
Previously we sorted the trees topologically before inserting them. On
a revlog-backed server, this may mean that trees are written in a different
order from the actual commits. hgsql-backed servers rely on the data being
written in linkrev order so they can be replayed in linkrev order on other
machines, so this broke hgsql replication.
Let's instead sort by linkrev, which will be both topological and satisfy
hgsql's requirements.
Reviewed By: quark-zju
Differential Revision: D15437953
fbshipit-source-id: d4aaaa03b392a6cb6cf1be478aed2583ecb757c5
Summary:
We had this disabled in a config we ship in rpms, but if we want the
tests to work in treeonly mode we want this disabled in all tree cases.
Reviewed By: xavierd
Differential Revision: D15296199
fbshipit-source-id: 0f9751583eefa10c275bd499bb5998adfbe644a4
Summary:
`hg pushbackup` and `hg isbackedup` can be called from scripts with `HGPLAIN`
set, and so can't be provided by aliases. Instead, provide deprecated wrapper
commands.
Reviewed By: mitrandir77
Differential Revision: D15436286
fbshipit-source-id: 3fbbf9a5fb4d0e8de2026a17c41ee11a139d645f
Summary: Improve debugability of cloud sync by logging updated sync state to the blackbox logs.
Reviewed By: mitrandir77
Differential Revision: D15434890
fbshipit-source-id: c5065455985a48777a855997a99e32ce0b31cc72
Summary:
When syncing, if a locally-available bookmark is synced to a new commit that
has been omitted, remove the local bookmark to ensure that the next cloud sync
doesn't move the bookmark back to where it used to be.
Reviewed By: mitrandir77
Differential Revision: D15414172
fbshipit-source-id: 71aaa2d89f734e4c575c24da2c0ef6b59ca4deaa
Summary:
A new flush method is added to mutablebasepack that just close and re-init
self.
Reviewed By: kulshrax
Differential Revision: D15416708
fbshipit-source-id: 79cdcb20b51b9688a5e95402057c7da27883003c
Summary:
This hook fires for every commit that is introduced in a pull. When
doing pulls with hundreds of thousands of commits, this introduces a noticable
delay. We don't use this hook anywhere, and it's not particularly scalable, so
let's delete it.
Reviewed By: singhsrb
Differential Revision: D15424697
fbshipit-source-id: 98d76bca703e625adf5be8f6234436befd260fc4
Summary:
Let's move hgsubversion to absolute_import, just to be consistent with the rest
of Mercurial codebase.
Reviewed By: markbt
Differential Revision: D15392154
fbshipit-source-id: e4c32939aff0616790828da508f3feea158669e1
Summary: As of D15154509, the data fetching functions in remotefilelog write to shared mutable packs rather than opening new packs. As such, there is no need to return pack paths. In fact, the code has already been updated so that the returned paths are always `None`, so the code removed in this diff is already dead.
Reviewed By: xavierd
Differential Revision: D15419765
fbshipit-source-id: c999d5388042b429a8bda9f72a06569364d8e2e1
Summary: This diff adds an `hg debuggettrees` command that downloads trees from the API server and stores them in a datapack.
Reviewed By: xavierd
Differential Revision: D15301607
fbshipit-source-id: 7820d82d7d021c420e911a6a2e9bfce62b69fa2e
Summary:
Improve logging of background backup commands by including the command that was
run and the time it was started in the background backup logs.
Reviewed By: mitrandir77
Differential Revision: D15334879
fbshipit-source-id: 932e91a43033c5cb06c79ede7b5224da2e34eb7d
Summary:
When pulling heads from commit cloud during sync, pull them in small groups
of heads from around the same time, to prevent overloading the server when
pulling a large number of heads.
Reviewed By: mitrandir77
Differential Revision: D15317184
fbshipit-source-id: 5e69eb970b18292a4f5d643b25fac80c90c5d537
Summary:
Lift determination of the correct remote path to connect to up to the top-level
command. This prevents the need to pass around the command-line `**opts` in
all of the commit cloud functions.
Differential Revision: D15295811
fbshipit-source-id: 0e14c1643bad96022c7a01126b447b2a6fcabaed
Summary:
Refactor how commit cloud sync works.
Sync is simplified by delegating backup processing to the existing backup code.
This happens first, which means the user's work is backed up earlier, and the
sync processing can assume that all backed up commits are available in the
cloud storage.
Sync no longer attempts to handle the case where cloud storage has changed.
Instead, backup processing should ensure that all local commits are backed up
to the current cloud storage.
If a commit can't be backed up, then treat this as a normal failure to
sync and ignore that commit for this sync attempt. If a commit can't be
downloaded from the server then the sync fails.
Reviewed By: mitrandir77
Differential Revision: D15295499
fbshipit-source-id: d371c5bf0daedbbe42e8c7d4a0c3d1a40c21a36f
Summary:
Move token location into the `token` module.
Move subscription management into the `subscription` module.
Move obsmarker management into the `obsmarkers` module.
Move everything else to a `util` module which can be disambiguated at import time.
Differential Revision: D15282859
fbshipit-source-id: 7f20c449fd79ffc33b069236a05fc73fac0e7d63
Summary:
The name is too long. We can disambiguate with `edenscm.mercurial.commands` at
import time if required.
Differential Revision: D15282860
fbshipit-source-id: e55357a121b583d9fd659f27dd5e2adc8a3d4d2f
