Summary:
commitcloud: do not use filtered set of heads when update infinitepush
state
hg cloud sync command can be interrupted when pulling
some of commits will be pulled, but not written to the commit cloud and
infinitepush states
the next run of cloud sync is smart, it will not pull what has been already pulled in but will fix the
commit cloud state
but because we used the filtered list of heads for infinitepush state, it will not fix the infinitepush
state
So, we shouldn't use the filtered set
Reviewed By: markbt
Differential Revision: D7972200
fbshipit-source-id: 94c01694d4ac77beeed647f77cbdb30fe3f7a404
Summary:
`DETACHED_PROCESS` simply means that child process won't inherit
current process' console (it does get one allocated by the OS though). `CREATE_NO_WINDOW` means that child
process does not get an allocated console altogether.
For details: https://msdn.microsoft.com/en-us/library/windows/desktop/ms684863(v=vs.85).aspx
Reviewed By: quark-zju
Differential Revision: D7969621
fbshipit-source-id: 58d5e69808fb1064bcc1101c971f52088a7fd2de
Summary:
The infinitepushbackupstate file must be written using the vfs of the source
repo when the share extension is in use.
Reviewed By: mitrandir77
Differential Revision: D7936097
fbshipit-source-id: a57e241ca969632ced65029cb5ccf61373bd8aeb
Summary:
In order to prevent newly added obsmarkers from being lost while we are syncing
them, we separate out the obsmarkers that we are in the process of syncing into
a separate `syncing` file. New obsmarkers that are created during sync go into
the `pending` file and are synced on the next sync, which means it's safe to
delete the `syncing` file when sync completes.
Differential Revision: D7932299
fbshipit-source-id: bd86e836ded10e75790b87ca6734a29f068f3571
Summary:
They got randomly printed out and broke tests. Let's limit it to
"debugtreedirstate" command.
Reviewed By: singhsrb
Differential Revision: D7904376
fbshipit-source-id: 4f0b3708fdc55ed207c3c3baa2926f4b2374730f
Summary:
Previously we only enabled on demand tree generation when treeonly was
set, but we also need it when sendtrees is set since we'll need to generate
trees for sending.
Reviewed By: quark-zju
Differential Revision: D7927856
fbshipit-source-id: a69d6c7920a92e4f90bdcd1d04aad9ef59e9c778
Summary:
hgsql depends on repositories being byte for byte identical, but the
current pull after a streaming clone can cause the repository to be different
(like if different delta decisions were made, or the commits were ordered
slightly differently). Let's disable that pull when the repository is an hgsql
repo.
Reviewed By: ryanmce
Differential Revision: D7925300
fbshipit-source-id: 6eba7ad4ccdd37f6d7c5090522867d1a54f722b7
Summary:
The destination revset could contain `SRC` that will break tweakdefaults.
Change the ordering so it could work.
Reviewed By: singhsrb
Differential Revision: D7924127
fbshipit-source-id: cb3eb1379425303607e4cb6f57f534133d457dea
Summary:
The "sensitive config sections" was used because "hg serve" loading
different extensions are incompatible with each other. Now we neither load
extensions nor run their uisetups, and just use one chg server. So
sensitive config sections can be removed.
Reviewed By: singhsrb
Differential Revision: D7847149
fbshipit-source-id: 758c1df21d280bf0f88d91432e1201c8417df532
Summary:
This is an attempt to fix the wrong status issue. A formal fix would be
D7909172, which merges fsmonitor.state into treestate. See the comment in
`__init__.py` change for why status can be wrong.
Although it fixes the "status" path, I'm not sure whether other code paths
writing dirstates could be problematic or not. Anyway, D7909172 should be the
preferred final fix.
Reviewed By: r4-in
Differential Revision: D7916344
fbshipit-source-id: beab1c825b970bb47ffe03617a14eb6f203feafa
Summary:
If `hg pushbackup` or `hg cloudsync` timed out waiting on the backup lock it
would still exit successfully. This updates the code to exit unsuccessfully so
that callers can tell that their commits were not actually backed up.
Reviewed By: quark-zju
Differential Revision: D7874624
fbshipit-source-id: 501b76d7c98d16f0128a77a20c881bf55d11a1d4
Summary:
This diff prepares the importer to cleanly handle moving files in perforce while
keeping the structure in hg untouched.
It introduces a magic string that has to be present in the changelist description:
`IMPORTER_IGNORE_REORG@`
When that is present, move/add move/delete actions where hg path is the same are
ignored.
Differential Revision: D7832814
fbshipit-source-id: e8323d0c3cc79ee81cb819bee63435f345069861
Summary:
Rework the commit cloud commands:
* Arrange them under a top-level `cloud` command.
* Make `join` call `register` if necessary and `sync`. This gives one-step
on-boarding.
* Add help text to all commands; clean up existing help text.
* Simplify output a little.
Differential Revision: D7876295
fbshipit-source-id: 1f7c0cabfa3d426ced34b90bdca64bef78d78211
Summary: This lets you find profiles that'll include a given filename.
Reviewed By: markbt
Differential Revision: D7876496
fbshipit-source-id: 3b375e5d8257a80853f854a6be27c74f805687e3
Summary:
Replace the monkey-patched subcommands in fbsparse with the new subcommand
infra.
Reviewed By: mjpieters
Differential Revision: D7849195
fbshipit-source-id: 7f1fc21f555dc2bea49d3a80efdbfd42a6e70cb5
Summary:
Add generalised support for subcommands. This is similar to the monkey-patched
version in `fbsparse`, but fully supported by the command infrastructure.
Subcommands are the same structure as normal commands, but are attached to a
table in the `subcommands` attribute of the main command. Normally, if no
subcommand is provided, the normal command function is called. This can be
made into an error by setting `subonly` on the top-level command.
In order to make `fbsparse` continue to work, I've temporarily hacked how it
handles help text. This will be fixed in a later diff that switches fbsparse
to use this infrastructure.
Reviewed By: mjpieters
Differential Revision: D7849476
fbshipit-source-id: b988edabb17da77bf91a278e0faa2feecd0c1db9
Summary:
WindowsError is only defined on Windows platforms. On non-Windows platforms
this could cause rage failures like this one: P59496908
This updates the code to just catch `OSError` since `WindowsError` derives from
`OSError` on Windows.
Reviewed By: phillco
Differential Revision: D7856903
fbshipit-source-id: cf56c24eb52bb1da5dcd94ef56a3f8eb5fd849e1
Summary:
The logic was added by D2594568 and is intentional for "repairing" dirstate.
It's O(working copy). That makes absorb super slow. Therefore let's use the
new "exact" flag to skip the slow path.
Reviewed By: mitrandir77
Differential Revision: D7818728
fbshipit-source-id: a3a7d6074bd0240b9e1919e18d3e0b95daf74a64
Summary:
Previously, absorb patches fsmonitor to skip invalidating fsmonitor state
during dirstate.rebuild. Now let's use the added "exact" parameter to avoid
the monkey patching.
Reviewed By: mitrandir77
Differential Revision: D7818729
fbshipit-source-id: 0db41c7609277acd4823101e5569cbbe80f7580e
Summary: This is a hint for performing certain fast paths.
Reviewed By: mitrandir77
Differential Revision: D7818730
fbshipit-source-id: 4adcf8724b462d8d652e8e580d6a36eebc46a0f8
Summary:
Previously it is not actually used.
`test-hgext-repogenerator.t` changed because treedirstate uses random
number to generate file names.
`fakedirstatewritetime.py` was updated to be treedirstate-aware. This
makes test-revert.t test-merge-tools.t test-merge1.t pass.
Reviewed By: singhsrb
Differential Revision: D7844960
fbshipit-source-id: 33a1d0d4a8e22ea5e6bb6454956884571fcf6bab
Summary:
There is a proper way [1] to skip the dict check-code check. Let's use it.
[1]: a61ed1c2d7
Reviewed By: DurhamG
Differential Revision: D7831336
fbshipit-source-id: a5e654e9e94cbfb1c5a07b047eb6e5451904c48e
Summary:
This would make tests run on treedirstate.
To avoid issues with Eden pulling from a non-eden treedirstate repo,
treedirstate is changed to be "always on" and disables itself on an eden repo.
The extension list is changed to a set for efficient `__contains__` test.
Reviewed By: phillco
Differential Revision: D7769804
fbshipit-source-id: d328fe51ef67c4730cfc53f43bdfc48c2765c541
Summary: This lets you select on substrings in fields and paths.
Reviewed By: quark-zju
Differential Revision: D7788826
fbshipit-source-id: f92b8cc646fd36f4cb3b8a4dc6116576db80eb42
Summary: Calculation on how many profiles were hidden is only done when the hint is actually shown.
Reviewed By: quark-zju
Differential Revision: D7774192
fbshipit-source-id: a7196b2cc5640d0a7cb9c4d572e1a31ebfa41598
Summary:
This lets us produce a list of profiles with the title set, or find all profiles that are missing a description.
Convert the 'hidden' filter to use this feature, rather than special-case it in `_discover()`.
Reviewed By: quark-zju
Differential Revision: D7774029
fbshipit-source-id: 3bcba75e6da97bf0e560e11ce1ae7cbcee49ee45
Summary:
Infinitepush stores bundles uncompressed, which can make them quite large.
Enable zstd compression of bundles, both when sending them to the server during
backups, and when storing the bundle in the bundle store on the server.
Reviewed By: quark-zju
Differential Revision: D7777371
fbshipit-source-id: ef9360bf05bf2a6d2b76e26cd40ad1e4ee8ae076
Summary:
check-seclevel.py complains about using section markers inside a container.
The help text about sparse file format and config options can be put at
extension-level.
Reviewed By: mjpieters
Differential Revision: D7776355
fbshipit-source-id: 0b7d813c5eee352a054b21897682f6064f384829
Summary:
We currently call "p4 where" on all files in the new client spec, which is roughly O(all files in ovrsource), and it's only going to get worse as ovrsource grows :(
This diff makes the number of calls O(files added & removed between two client specs) by filtering them in the following way:
- get a full list of files from old client (L1)
- get a full list of files from new client (L2)
- files_added = L2 - L1
- files_removed = L1 - L2
- `p4 where` on (files_added + files_removed)
How much speedup do we get?
from
[2018-04-25 16:07:27,725]INFO:root: Start sync import in ovrsource-master
[2018-04-25 17:35:35,873]INFO:root: Finish sync import in ovrsource-master after 5288.147963762283 seconds
to
[2018-04-26 13:43:28,819]INFO:root: Start sync import in ovrsource-master
[2018-04-26 13:51:35,575]INFO:root: Finish sync import in ovrsource-master after 486.7560772895813 seconds
on this change D7722798
Differential Revision: D7772403
fbshipit-source-id: 05a16343264007ee3ee466621da9da888c2368d7
Summary:
We want to allow blocking full repo streaming clones in certain
repositories (since they can take the lock and take a very long time) unless the
client has explicitly asked for it. The existing stream_out wire protocol has no
way of passing an option, so let's create a new endpoint.
Reviewed By: quark-zju
Differential Revision: D7763717
fbshipit-source-id: eace47143f8fdcc4c6e302b5c26678ccf56ca5d4
Summary: This is needed to be able to list profiles via SCMQuery, where there is a working copy at the null revision.
Reviewed By: quark-zju
Differential Revision: D7745434
fbshipit-source-id: b5ac236c36f9bafc9e0f305bc0892e0cd8bec628
Summary:
This makes it easier to modify and test the core logic without coupling with
the Python logic.
Reviewed By: markbt
Differential Revision: D7734012
fbshipit-source-id: 0d7b19198d85f6ca7314611256e9271be60070d1
Summary: See the comments for more.
Reviewed By: quark-zju
Differential Revision: D7738340
fbshipit-source-id: 20660c2ff7fdba1850c0fb9f34548db0084abf3d
Summary: If there is a non-existing sparse profile active in .hg/sparse, skip it rather than bail out.
Reviewed By: quark-zju
Differential Revision: D7744957
fbshipit-source-id: 4e6a046c95f5fe6cf49a7093d3443340f70e8e2a
Summary: Curernt sync importer modifies mercurial storage data structures directly, which is error prone and hard to debug. This reimplements the sync importer with a higher level mercurial API (what `p4seqmport` uses).
Differential Revision: D7714322
fbshipit-source-id: 0269839b5ee3a4b45f166dce74dfd29c8ec5135a
Summary:
D7676378 would break this as it changes how parse_where behaves. This is the only
callsite left behind. AFAICT it actually wouldn't work, as it passes the hg path
to p4 where.
This change removes the option and related code.
Differential Revision: D7696956
fbshipit-source-id: a27077a9540369b5c77692185f317cf5395789ba
Summary:
This logs timing data for:
- update
- merge.update
- mergedriver
- applyupdates
- workers
- fetches
- pulls
- remotefilelog prefetches
- treemanifest prefetches
- status
- dirstate walks
- fsmonitor walks
- watchman queries
Hopefully this will let us narrow down where time is going in a number of cases
where the profile data is ambigiuous or hard to come by.
Reviewed By: quark-zju
Differential Revision: D7681026
fbshipit-source-id: e6fe65c9a4d2f4e128f62ccb767a7cbe73b2649a
Summary:
They cause a lot of churn when updating vendored crates. It's unnecessary to
check them in because the vendored crates basically pinned versions of all
dependencies.
Also clean up .gitignore, since it does not support "syntax:re".
Reviewed By: DurhamG
Differential Revision: D7562533
fbshipit-source-id: b9fcade6f38c0652a1b19ac4f90827dac7eed181
Summary:
when a user is joined to commitcloud workspace, we should provide
commit cloud related messages in the smartlog
Reviewed By: markbt, StanislavGlebik
Differential Revision: D7705709
fbshipit-source-id: ba2c246f115542e4f49446284cd831d8c266245f
Summary:
Make `hg getavailablebackups` limit the list to 5 (configurable) backups by
default. Add a new `--all` option to list them all.
Reviewed By: quark-zju
Differential Revision: D7671210
fbshipit-source-id: 743e0032e11b5d21b1544b695d28e8818c44085e
Summary: Add the `hg backupdelete` commands, that lets users delete their old backups.
Differential Revision: D7670524
fbshipit-source-id: d0a12e1aa7c770cf9313e792d859c9a76ff43d9f
Summary:
If the infinitepush backup state is empty (indicating this is probably a new
clone of a repository), and there are no commits or bookmarks to be backed up,
don't push a backupbundle that clears all the heads and bookmarks of any
backups that have the same servername and path. This might be a reclone of the
repository, and the user might want to restore the backup.
When the first commit or bookmark is made in the new repository, the old
backup will be cleared at that point.
Differential Revision: D7669728
fbshipit-source-id: bec859bbbbee6b631efb4483a8fedbad8f3bcbac
Summary:
We currently call p4 where with one path at a time, but it accepts a list.
This change takes advantage of that, batching p4 where calls, which speeds up
importing.
Differential Revision: D7676378
fbshipit-source-id: 4a6747458555a60dd5f385604f2a25d595af947d
Summary:
The changes in this diff changes string comparisons using the compare method to
using equality operators.
Motivation:
- readability, simplifies code
- compare method is intended for sorting functions
This is clang check used: [Link Here](https://clang.llvm.org/extra/clang-tidy/checks/readability-string-compare.html)
Reviewed By: yfeldblum
Differential Revision: D7674416
fbshipit-source-id: dce375fb4b401f0e012ad4fc99b5dc4babe07f89
Summary:
The merge process could be interactive, and tools like Hack wants to
scan changed files when the user is resolving conflicts. There is a
"hg.update" watchman state and "state-enter", "state-exit" events that help
detect the "update" process. However, the interactive "ui.merge" process is
inside the "update" process and cannot be easily decoupled confidently (i.e.
hard to end "update" state before doing the merge).
It's also hard to have a general "merge" state due to the current watchman
design. Because a merge might be across multiple hg processes (ex.
non-interactive ui.merge, hg exits, returns to the shell, and the user invokes
`hg resolve --mark` to end the merge state), and the current watchman
implementation will force end all states associated with a watchman socket
connection. If a hg process exits, it will disconnect from the watchman socket
and the merge state will be forced terminated, although there is still
unresolved conflicts. Previous attempt on this is D7286181.
This patch adds a "hg.filemerge" watchman state that could make watchman
users aware of ongoing interactive merge resolutions. The event fires when an
external merge tool is called, per conflicted file. It is strictly a sub
interval of an "update" state, which is within a single process. So it does not
have the problem that a "merge" state has.
Differential Revision: D7648929
fbshipit-source-id: 858beba1bff261e8da5bd4d8b08a388dcbc8a9ef
Summary:
Currently, the sync importer only reads files from perforce to ensure that the contents are up to date. Connecting to perforce and use `p4 print` is pretty slow, and we would like to use alternatives like `rcs` and `gzip` to speed things up, which are also what seqimporter is using.
This diff changes the reading behavior of sync importer so that it can also read from `rcs` and `gzip`.
A brief test on reading files from `Tools/...` shows that this works and the speedup is pretty good.
Differential Revision: D7619304
fbshipit-source-id: 563d0e40bcf7fa9a187cd3ede70878dccef4f9e9
