Summary:
Avoid transient partially synced stated by updating the bookmarks and
obsmarkers as part of the pull command, if there is one. Additionally,
prevent remotenames bookmarks from being updated in cloud-sync pulls that
happen to include public commits. That's preferable, as usually this will only
be a subset of remote bookmarks, and this can give confusing results.
Reviewed By: DurhamG
Differential Revision: D8401117
fbshipit-source-id: 5ee646d8c7e49ea70d3ec5fd7df5c9009de7bcdd
Summary:
We get the list of obsmarkers to sync after we have pushed the commits. In
that time, new obsmarkers could have shown up, which shouldn't be synced until
the next sync operation.
Improve this by getting the obsmarkers just after we get the heads and
bookmarks. This still has a race, but the window is now much smaller.
It also means we can ensure we sync even when all we have is some extra
obsmarkers.
Reviewed By: DurhamG
Differential Revision: D8397739
fbshipit-source-id: e29890aa250d58bfc3c3c2a056a01d287c053605
Summary:
This fixes how DatapackStore::refresh() computes if it needs to perform a
refresh.
Previously this code was using `clock()`, which returns the amount of CPU time
used by the program. This seems like an undesirable metric to use: the amount
of CPU used by the current program doesn't seem like it should be relevant to
whether the files on disk need to be re-examined or not. If the program has
slept for an hour we probably still want to refresh the files even though no
CPU time has been consumed.
As a result of how the computation was performed, markForRefresh() also would
not trigger a refresh if the program has used less than 100ms of CPU time so
far.
This was breaking some of the eden tests which create a repository pull some
packs into it, and then immediately refresh the store. If all of this happens
in less than 100ms the DatapackStore code would not actually refresh the
store.
Reviewed By: chadaustin
Differential Revision: D8410266
fbshipit-source-id: f08d317b261ccfe89cff45bf3e511a2b87cf9fb5
Summary: This allows us to dogfood the new treestate.
Reviewed By: markbt
Differential Revision: D7912559
fbshipit-source-id: bfb30cd9989460b0eae859aeed402bd92776ab0a
Summary: This makes it easier to support format version 2 - treestate.
Reviewed By: markbt
Differential Revision: D8394003
fbshipit-source-id: ef341fc6d5dd6ba34ad57a98dc3ed63e03d5c1c4
Summary:
Previously, there are 2 configs: `treedirstate.useinnewrepos` and
`format.usetreestate`. They are both related to dirstate format and conflict
with each other. This patch unifies them into a single config
`format.dirstate`.
As we're here, merge `test-fb-hgext-treedirstate-x.t` to `test-dirstate-x.t`
if they were previously copied from `test-dirstate-x.t`
Reviewed By: markbt
Differential Revision: D8393878
fbshipit-source-id: 57abeea22ce732d93205e4d4308923afa90693f4
Summary:
When requesting trees from the server we try to provide a base tree
that we already have. In the hybrid manifest days, when we were fetching
manifest X, we would look up the linknode for X in the flat manifest revlog,
then scan the changelog up and down from that linknode. In the treeonly world we
don't have that revlog anymore though, so the current algorithm just searches from
the tip. This ended up being a really bad algorithm, since if you have tip and you
request tip~1000, then tip~2000, then tip~3000, you end up basing them all
against tip and redownloading a lot of data. This makes hg blame and hg log -p
super slow.
Let's change the algorithm to base off of a hint linknode, and let's change
changectx to set that hint every time it attempts to read a manifest. In manual
testing this significantly sped up hg log -p
Reviewed By: phillco
Differential Revision: D8399613
fbshipit-source-id: 771a94ee8b82be682ea0091b8d6c0fcd5f4e6646
Summary: as titled, so that we can `hg log -T {reviewers} --rev .`
Reviewed By: quark-zju
Differential Revision: D8389944
fbshipit-source-id: b3b194b7027bc831d5eca42dfcb4300745d6eb12
Summary: sunshowers hit an exception (either an upgrade race or a module deinitialization) and while it was spurious, it can't hurt to run this logic in a try/catch anyway.
Reviewed By: ikostia
Differential Revision: D8360101
fbshipit-source-id: e581bdeb69e22571f547633b7219d6886cab1aa7
Summary:
If the repo has treemanifests, we can use this to do the case check faster.
Note that with treemanifests we only find the first path component that
conflicts (subsequent conflicts won't be found), as these conflicts occur in
different trees.
Reviewed By: farnz
Differential Revision: D8332426
fbshipit-source-id: 243dab9bf711c6fdb62a0e0ba4346faea3b6b0ad
Summary:
Add a new function to `cstore.treemanifest` to list the contents of a directory
in the tree manifest.
Reviewed By: ryanmce
Differential Revision: D8332425
fbshipit-source-id: db74f11a691cd6b8ac7d51aec6b7da8149c22027
Summary: This will allow us to have json output
Reviewed By: quark-zju
Differential Revision: D8332945
fbshipit-source-id: 3ad39c4df4fe0fbe038d277eabeb18fed10a4bb1
Summary:
Detect the case when we're writing an commit that already exists and skip
writing the history entries.
Reviewed By: DurhamG
Differential Revision: D8363935
fbshipit-source-id: d8db6233eb0261b49e3903c0345c898f2abffa12
Summary:
It's legit for a file name to contain `%s` or `%r` strings. Previously,
blackbox always expect the first argument to be the "format string". That's
inconvenient and it's easy to just pass a string containing the troublesome
`%s` to `ui.log`.
Let's just do not even try formatting strings if there is only one argument,
and if there are multiple arguments, fallback to concatenate them if they
cannot be formatted.
Reviewed By: DurhamG
Differential Revision: D8364130
fbshipit-source-id: 75b2d0e0a460b9a86d4ecd6ecfbb77c0c0fbe98c
Summary:
Add a config option `format.usetreestate`, which would turn on the new
treestate for new repos.
Treestate does not need manually maintaining "copymap", "otherparentset",
"nonnormalset". The one single tree is the source of truth of everything.
So some code like `copymap.pop` etc. are disabled on treestate path.
Reviewed By: markbt
Differential Revision: D7909175
fbshipit-source-id: 96a405fccd4099cefc152ec17204cb5407efbe7e
Summary: return code check was not implemented before but it is important
Reviewed By: markbt
Differential Revision: D8348689
fbshipit-source-id: 69948091832d4c954284e0c999da05c4980d8228
Summary:
Let's roll this out gradually with opsfiles instead.
Also fix an inverted logic test.
Reviewed By: singhsrb
Differential Revision: D8339739
fbshipit-source-id: 2700d4a16ee38280d127e2a4ee7458bdff32f38c
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
This fixes the issue where every merge conflict situation immediately took 30s - 1m due to the "scanning working copy" full repo scan.
This happens because sparse matcher with two working parents is different from with one working parent - It switches from returning one matcher for the gitignore to returning both parent's matchers in a unionmatcher -- even if they are identical, the hash changes, triggering a full repo can.
After discussing with quark-zju, he mentioned the fsmonitor gitignore change detection has been mostly-broken for a long time ([1], [2]), and is no longer worth maintaining. The perf costs outweigh correctness and implementation of "correctness" costs, which are esoteric (you update to master with a dirty file that master happens to have unignored -- but unignores are rare) and easily fixed by the user (saving any of the files fixes it). Lastly, quark-zju's treestate stack already does this, so this effectively brings those perf benefits early.
In the longer team, the correct fix would be tracking ignored files (including those outside sparse) in treestate, and examining them.
Wrapped in a config in case it goes wrong.
[1] 6dea1701f1 (.hgignore only invallidated if the toplevel .hgignore changed, which was rare -- at Facebook usually the subdirectory ignore was changed)
[2] In `matcher.py`, the `gitignorematcher.__repr__` returns a static string.
Reviewed By: quark-zju
Differential Revision: D8337682
fbshipit-source-id: 725b3eb8fba06940fd2db0903bcecf7416e64e0d
Summary:
this is needed because `hg cloud sync` can be triggered by external serviced like scm_daemon on behalf of the user,
so it should just fail rather than expect user to type the password, so we change ui ssh option to the bgssh (background ssh) that is defined in infinitepush section
Reviewed By: markbt
Differential Revision: D8331723
fbshipit-source-id: 28f9d007702e4f6ed5216114921375b76def3f93
Summary:
In a previous diff we added logic to delete corrupt packs in the cache
directories, but the logic only caught issues after the pack was loaded. We also
need to catch issues while loading the pack. This happens when a pack file is 0
length.
Reviewed By: quark-zju
Differential Revision: D8324248
fbshipit-source-id: c2fcb4f69c4627cce316d626e91f752ef562a0d9
Summary:
An updated version of D8174246 with a test. Max memory usage is very useful to us to understand which commands are using too much memory, and the existing wrapper metrics don't work when chg is used (they measure chg's usage).
Windows support will come later as it is a bit more involved.
Reviewed By: DurhamG
Differential Revision: D8318584
fbshipit-source-id: 323450bc7ab376014d70106beb5d4fdcc7fba0c8
Summary:
In the new corruption fix strategy, we're only going to strip the
entries from the root treemanifest revlog. This will leave all the
still-referenced sub-trees available for fetching if clients still have old
references to them, while preventing new references from being taken from the
root nodes.
Reviewed By: phillco
Differential Revision: D8284287
fbshipit-source-id: 4cc20bfdbf7ae5d32074dae22da4d37717ee5728
Summary:
Use the command-level template and the `nodechanges` template keyword.
Drop the use of formatter.
`-Tjson` is not treated specially so it will just print `json`. If we'd like
to make it output `{"nodehcnages": ...}` (probably not a list with one
element), it's doable by having some mapping from `json` to a default
template.
Reviewed By: mitrandir77
Differential Revision: D8221076
fbshipit-source-id: 75575a98324d0d069789e0e915d5aa282ca0d4bb
Summary:
According to scuba chart https://fburl.com/scuba/mn5nfdna, these are
not used by any critical client, don't seem useful and can be removed.
Reviewed By: quark-zju
Differential Revision: D8232616
fbshipit-source-id: ae07cfb00bfe45cd6f8c31f3fa0871c9ba1fdadf
Summary:
It could affect pushrebase behavior - pushrebase server might have a large
set of "missing" nodes, and take that as the commits to rebase. Then fail
with "cannot rebase public commits".
Maybe there is a better way to fix it so it only affects pull but not push.
But for now it seems safer to disable it for automation.
Reviewed By: DurhamG
Differential Revision: D8049229
fbshipit-source-id: 3612e2fb2c1c825e8974156c27b03966ee745c98
Summary:
They're actively fighting against the clang-format config
and don't have an auto-fix.
Reviewed By: quark-zju
Differential Revision: D8283622
fbshipit-source-id: 2de45f50e6370a5ed14915c6ff23dc843ff14e8a
Summary:
Previously, when adding temporary pack files to the local data store,
we added them at the end. This causes problems with the recently added ondemand
generating datastores, since when we go down the line of stores we will hit the
generating store (which will create a tree from flat manifest, which is slow)
before we hit the bundle store that actually contains the data.
Let's reorder the stores so the bundle stores go at the front, since they are
likely to be used.
Reviewed By: phillco
Differential Revision: D8291415
fbshipit-source-id: 4f9799cf755b2b1f7fe8d8d1c041a3b3f2c7e463
Summary:
Optimized builds were giving a warning about shadowed variables, which
broke some of our continuous builds. Let's fix those.
Reviewed By: phillco
Differential Revision: D8287865
fbshipit-source-id: 8a2d023be93499ed6458a35ba8e8cb4d83d59e4a
Summary:
We want to better understand why people are getting "scanning working copy", so this is a first step towards that -- log every time we call `invalidate()` with the reason.
If one reason ends up dominating the others (e.g., "exception") we can drill down later.
Reviewed By: quark-zju
Differential Revision: D8272450
fbshipit-source-id: 46cb963eefa5ca60df81bdcd8f1e232855c106b7
Summary:
Previously, if a client sent a gettreepack request and specified a
non-root directory, the server would go back to the root of the tree and send
the whole tree. Now that treemanifest has a walksubdirtrees function for
iterating over a specific subset of a tree, we can make it return just the
subset.
This diff does delete the treememoizer class. This was previously used to keep
trees in memory if they would be needed again (for instance, as a base node for
a future tree). We aren't able to provide this functionality when using
walksubdirtrees, because python doesn't have access to the underlying sub
directory tree objects. This class was probably unnecessary anyway, since we
usually only request a small number of trees at once.
Reviewed By: quark-zju
Differential Revision: D8217360
fbshipit-source-id: 1db970659eabe2829acc6d49a15eeb9345ce4a5b
Summary:
In a future diff we'll be adding the ability to request just a
subsection of a tree. In order to do that, we need to expose an api that can
iterate of just a subsection of the tree.
Since the treemanifest object itself acts mostly like a dictionary, and doesn't
expose sub-tree functionality, we expose sub-tree walks as a limited use class
level function.
Reviewed By: quark-zju
Differential Revision: D8216726
fbshipit-source-id: 0d72ef041fd6a973387f975b21e55c82f7ec789c
Summary:
A recent change renamed the methods assuming they were rust methods
but these methods are used in the python code and therefore, already had the
correct naming. This broke the one of the python tests calling the methods
which will pass now.
Reviewed By: DurhamG, quark-zju
Differential Revision: D8279742
fbshipit-source-id: 01761ad033c6669a0ae54dc1beb853a0d56df27d
Summary:
just refactoring to improve the code quality
the main improvement is that I separated TcpReceiver to a different service,
any other services can register callbacks with TcpReceiver service.
For WorkspaceSubscriberService callbacks are implemented using mpsc channel to notify the main WorkspaceSubscriberService thread and single atomic flag that allows running subscriptions to join.
Another improvement is that I added logic to run cloud sync on the first keep alive after connection errors
Reviewed By: markbt
Differential Revision: D8226109
fbshipit-source-id: 3fe513da9273b28b2262948ecdf620821e7ab313
Summary:
Use the new stack-by-stack commit backup for infinitepushbackup as well as
commitcloud.
Reviewed By: DurhamG
Differential Revision: D8241380
fbshipit-source-id: 0c1481ad1ab5759ab642d0b02c7fe08523056292
Summary:
Rather than pushing a bundle containing all unpushed draft commits, instead
push stack by stack. This reduces the individual bundle size, and means
failure to push some commits means we can still make progress.
Reviewed By: DurhamG
Differential Revision: D8241383
fbshipit-source-id: 6f1528613fa228627fe156a8e4720dd46929ebe2
Summary:
Use the repo connectionpool to connect to the server. This allows sensible
re-use of connections whilst gracefully handling errors.
Reviewed By: DurhamG
Differential Revision: D8241382
fbshipit-source-id: f43669a32e4b1ff57193ea0959c57ff14526a998
Summary:
Split the pushing of commitcloud commits and infinitepush bookmarks into
separate functions, in preparation for pushing commits stack-by-stack.
Reviewed By: DurhamG
Differential Revision: D8241385
fbshipit-source-id: dc82c3470259b0ff738ebcdf6901ee8052405379
Summary:
Refactor pushing of draft heads into infinitepush; we want to call this from
infinitepushbackup, too.
Reviewed By: DurhamG
Differential Revision: D8241381
fbshipit-source-id: d5eb06efd6cd9382e016b0738d9b3e89e524de53
Summary:
Remove the cleanup from pushbackupbundle. As we now wait for the bundle
response, this is no longer necessary, and it prevents re-use of the
connection.
Reviewed By: DurhamG
Differential Revision: D8241379
fbshipit-source-id: a885198c49dce1beb1385a61a209d299ed3d2d02
Summary:
Split the infinitepush backups code into a separate module.
The infinitepush extension is responsible for providing scratch pushes, as well
as the server-side implementation for storing scratch bundles.
The infinitepushbackup extension uses this feature to provide automatic backup
support. It is an alternative to the commitcloud extension, and the two will
be mutually exclusive.
Reviewed By: DurhamG
Differential Revision: D8204418
fbshipit-source-id: 9279c426da35d451a5e7b39a72f2c91d364e3fd4
Summary:
There was a bug where if we tried to convert a flat manifest into a
tree while the flat manifest commit was being committed, the fast path
conversion would produce incorrect results because it tried to access the commit
data. If len(changelog) == 5, then changelog.revision(5) returns the nullid
data, which caused the bug.
This is absurb behavior from the changelog, so I will change that in a future
diff.
Reviewed By: phillco, singhsrb
Differential Revision: D8270372
fbshipit-source-id: 27bbfb2e54de35553b880954f06e76fe4fc0d47b
Summary:
In a future diff we'll be adding the ability to iterate over just a
subsection of a tree. In order for the resulting paths to be correct, we need to
be able to prime the start path. To do so, let's add a path argument to the
SubTreeIterator, and set it to empty string everywhere it's currently used.
Reviewed By: quark-zju
Differential Revision: D8216727
fbshipit-source-id: 1969f380f7ad13f54a7dbcd2283b0428c4480ab4
Summary:
In a future diff we'll be adding a function that also needs to turn a
python store into a ManifestFetcher. So let's move that logic to a separate
function.
Reviewed By: quark-zju
Differential Revision: D8216728
fbshipit-source-id: f6d2e9577eccfcb015797a519264bd3b241ebae5
Summary:
Now that the subtree iterator doesn't contain a reference to the
treemanifest directly (it instead holds a ManifestPtr reference), we no longer
need to persist a reference to it at the python layer.
Reviewed By: quark-zju
Differential Revision: D8206301
fbshipit-source-id: 531d4399f0f54a47e0eb741f419e6242188a83ad
