2019-11-01 23:24:22 +03:00
|
|
|
# Copyright (c) Facebook, Inc. and its affiliates.
|
2015-10-31 01:37:17 +03:00
|
|
|
#
|
|
|
|
# This software may be used and distributed according to the terms of the
|
2019-11-01 23:24:22 +03:00
|
|
|
# GNU General Public License version 2.
|
|
|
|
|
|
|
|
# dirsync.py - keep two directories synchronized at commit time
|
2015-10-31 01:37:17 +03:00
|
|
|
"""
|
2017-09-22 01:57:11 +03:00
|
|
|
keep directories in a repo synchronized (DEPRECATED)
|
2015-10-31 01:37:17 +03:00
|
|
|
|
2017-09-22 01:57:11 +03:00
|
|
|
Configure it by adding the following config options to your .hg/hgrc or
|
|
|
|
.hgdirsync in the root of the repo::
|
2015-10-31 01:37:17 +03:00
|
|
|
|
2017-09-22 01:57:11 +03:00
|
|
|
[dirsync]
|
|
|
|
projectX.dir1 = path/to/dir1
|
|
|
|
projectX.dir2 = path/dir2
|
2015-10-31 01:37:17 +03:00
|
|
|
|
|
|
|
The configs are of the form "group.name = path-to-dir". Every config entry with
|
|
|
|
the same `group` will be mirrored amongst each other. The `name` is just used to
|
|
|
|
separate them and is not used anywhere. The `path` is the path to the directory
|
|
|
|
from the repo root. It must be a directory, but it doesn't matter if you specify
|
|
|
|
the trailing '/' or not.
|
|
|
|
|
|
|
|
Multiple mirror groups can be specified at once, and you can mirror between an
|
2017-09-22 01:57:11 +03:00
|
|
|
arbitrary number of directories::
|
|
|
|
|
|
|
|
[dirsync]
|
|
|
|
projectX.dir1 = path/to/dir1
|
|
|
|
projectX.dir2 = path/dir2
|
|
|
|
projectY.dir1 = otherpath/dir1
|
|
|
|
projectY.dir2 = foo/bar
|
|
|
|
projectY.dir3 = foo/goo/hoo
|
2019-01-15 18:11:49 +03:00
|
|
|
|
|
|
|
If you wish to exclude a subdirectory from being synced, for every rule in a group,
|
|
|
|
create a rule for the subdirectory and prefix it with "exclude":
|
|
|
|
|
|
|
|
[dirsync]
|
|
|
|
projectX.dir1 = dir1/foo
|
|
|
|
exclude.projectX.dir1 = dir1/foo/bar
|
|
|
|
projectX.dir2 = dir2/dir1/foo
|
|
|
|
exclude.projectX.dir2 = dir2/dir1/foo/bar
|
2015-10-31 01:37:17 +03:00
|
|
|
"""
|
|
|
|
|
2017-03-18 02:35:08 +03:00
|
|
|
from __future__ import absolute_import
|
|
|
|
|
dirsync: Behave better with already-inconsistent mirror
Summary:
If a mirror of a file is missing, dirsync now behaves better:
- When deleting the source, it allows the commit and does no mirroring since the two paths are already in sync (previously, it raised an exception when trying to delete the mirror)
- When updating the source, it creates a file in the mirrored location to match
- When updating (creating) the mirrored file, the source (which already exists) is updated too
Fixes https://www.facebook.com/groups/sourcecontrol/permalink/929877003728587/.
Test Plan: `../../hg/tests/run-tests.py test-dirsync.t`
Reviewers: #sourcecontrol, durham, lcharignon, ttung
Reviewed By: durham
Subscribers: poshannessy, mjpieters, durham
Differential Revision: https://phabricator.fb.com/D2794928
Signature: t1:2794928:1456446576:c18d06a42eb89b5168c25d048ba4fa1fa3f0c6d3
2016-02-26 03:50:43 +03:00
|
|
|
import errno
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
dirsync: improve performance for codemods
Summary:
Use the Rust tree matcher to rule out files that do not need dirsync quickly.
This should make codemod faster to commit.
I created 5000 files outside dirsync config in fbsource (with 494 lines of
dirsync config), and `hg add` them:
$ mkdir unused
$ cd unused
$ for f in `seq 5000`; do touch $f; done
$ hg add .
Baseline "status":
In [3]: %time repo.status();1
CPU times: user 111 ms, sys: 10.2 ms, total: 122 ms
Wall time: 148 ms
Before, dirsync overhead is ~8x "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.37 s, sys: 28.8 ms, total: 1.4 s
Wall time: 1.79 s
Out[1]: (<workingctx f23d7c84c5a7+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.07 s, sys: 8.41 ms, total: 1.08 s
Wall time: 1.11 s
Out[2]: (<workingctx f23d7c84c5a7+>, set())
After, dirsync overhead is ~1/2 of "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 203 ms, sys: 18.9 ms, total: 222 ms
Wall time: 245 ms
Out[1]: (<workingctx 8ff14e46c9d8+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 154 ms, sys: 24.1 ms, total: 178 ms
Wall time: 202 ms
Out[2]: (<workingctx 8ff14e46c9d8+>, set())
Reviewed By: DurhamG
Differential Revision: D26726476
fbshipit-source-id: e34218769c779c9a4ee64c654c75298b7c79f213
2021-03-03 05:09:48 +03:00
|
|
|
import bindings
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
from edenscm import tracing
|
2019-01-30 03:25:33 +03:00
|
|
|
from edenscm.mercurial import (
|
2017-09-08 20:01:05 +03:00
|
|
|
cmdutil,
|
2018-07-30 17:55:50 +03:00
|
|
|
config,
|
2017-03-18 02:35:08 +03:00
|
|
|
error,
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
context,
|
2017-03-18 02:35:08 +03:00
|
|
|
extensions,
|
|
|
|
localrepo,
|
|
|
|
match as matchmod,
|
2020-01-29 03:25:29 +03:00
|
|
|
pycompat,
|
2017-09-08 20:01:05 +03:00
|
|
|
scmutil,
|
2017-03-18 02:35:08 +03:00
|
|
|
util,
|
|
|
|
)
|
2019-01-30 03:25:33 +03:00
|
|
|
from edenscm.mercurial.i18n import _
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
from edenscm.mercurial.node import bin
|
2015-10-31 01:37:17 +03:00
|
|
|
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
|
|
|
testedwith = "ships-with-fb-hgext"
|
2019-01-15 18:11:49 +03:00
|
|
|
EXCLUDE_PATHS = "exclude"
|
2015-10-31 01:37:17 +03:00
|
|
|
|
2017-03-18 05:42:50 +03:00
|
|
|
_disabled = [False]
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
_nodemirrored = {} # {node: {path}}, for syncing from commit to wvfs
|
2017-03-18 05:42:50 +03:00
|
|
|
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
2015-10-31 01:37:17 +03:00
|
|
|
def extsetup(ui):
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
extensions.wrapfunction(localrepo.localrepository, "commitctx", _commitctx)
|
2017-09-08 20:01:05 +03:00
|
|
|
|
dirsync: disable dirsync during shelve/unshelve
Summary:
Shelve and unshelve use commits under the hood to bundle up data. dirsync sees
these commits happening and performs mirrors, which then shows up at unbundle
time, making it impossible to mirror the change again. Example: if I change X,
and it gets mirrored to Y, if I go back and change X later after a unshelve, hg
commit now fails because Y has pending changes and differs from X.
Test Plan: Added a test
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Subscribers: quark, wez, mjpieters, frantic, wluh
Differential Revision: https://phabricator.intern.facebook.com/D3266602
Signature: t1:3266602:1462484017:738f67c0ab4b5af999819d3855c1f4ba6b2ea338
2016-05-06 00:51:59 +03:00
|
|
|
def wrapshelve(loaded=False):
|
|
|
|
try:
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
shelvemod = extensions.find("shelve")
|
|
|
|
extensions.wrapcommand(shelvemod.cmdtable, "shelve", _bypassdirsync)
|
|
|
|
extensions.wrapcommand(shelvemod.cmdtable, "unshelve", _bypassdirsync)
|
dirsync: disable dirsync during shelve/unshelve
Summary:
Shelve and unshelve use commits under the hood to bundle up data. dirsync sees
these commits happening and performs mirrors, which then shows up at unbundle
time, making it impossible to mirror the change again. Example: if I change X,
and it gets mirrored to Y, if I go back and change X later after a unshelve, hg
commit now fails because Y has pending changes and differs from X.
Test Plan: Added a test
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Subscribers: quark, wez, mjpieters, frantic, wluh
Differential Revision: https://phabricator.intern.facebook.com/D3266602
Signature: t1:3266602:1462484017:738f67c0ab4b5af999819d3855c1f4ba6b2ea338
2016-05-06 00:51:59 +03:00
|
|
|
except KeyError:
|
|
|
|
pass
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
|
|
|
extensions.afterloaded("shelve", wrapshelve)
|
|
|
|
|
dirsync: disable dirsync during shelve/unshelve
Summary:
Shelve and unshelve use commits under the hood to bundle up data. dirsync sees
these commits happening and performs mirrors, which then shows up at unbundle
time, making it impossible to mirror the change again. Example: if I change X,
and it gets mirrored to Y, if I go back and change X later after a unshelve, hg
commit now fails because Y has pending changes and differs from X.
Test Plan: Added a test
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Subscribers: quark, wez, mjpieters, frantic, wluh
Differential Revision: https://phabricator.intern.facebook.com/D3266602
Signature: t1:3266602:1462484017:738f67c0ab4b5af999819d3855c1f4ba6b2ea338
2016-05-06 00:51:59 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
def reposetup(ui, repo):
|
2021-03-04 22:32:34 +03:00
|
|
|
if not repo.local() or not util.safehasattr(repo, "dirstate"):
|
|
|
|
return
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
|
|
|
|
# If dirstate is updated to a commit that has 'mirrored' paths without
|
|
|
|
# going though regular checkout code path, write the mirrored files to disk
|
|
|
|
# and mark them as clean (or remove mirrored deletions).
|
|
|
|
#
|
|
|
|
# Note: This is not needed for a regular checkout that updates dirstate,
|
|
|
|
# but there are code paths (ex. reset, amend, absorb) that updates
|
|
|
|
# dirstate parents directly for performance reasons. This fixup is
|
|
|
|
# necessary for them, because dirsync breaks their assumptions about what
|
|
|
|
# files have changed (dirsync introduces more changed files without their
|
|
|
|
# consent).
|
|
|
|
#
|
|
|
|
# Similar to run `hg revert <mirrored paths>`.
|
|
|
|
|
|
|
|
def dirsyncfixup(dirstate, old, new, repo=repo):
|
|
|
|
p1, p2 = new
|
|
|
|
mirrored = _nodemirrored.get(p1, None)
|
|
|
|
if not mirrored:
|
|
|
|
return
|
|
|
|
paths = sorted(mirrored)
|
|
|
|
wctx = repo[None]
|
|
|
|
ctx = repo[p1]
|
|
|
|
for path in paths:
|
|
|
|
wf = wctx[path]
|
|
|
|
if path in ctx:
|
|
|
|
# Modified.
|
|
|
|
f = ctx[path]
|
|
|
|
wf.write(f.data(), f.flags())
|
|
|
|
dirstate.normal(path)
|
|
|
|
tracing.debug("rewrite mirrored %s" % path)
|
|
|
|
else:
|
|
|
|
# Deleted.
|
|
|
|
if wf.exists():
|
|
|
|
wf.remove()
|
|
|
|
dirstate.delete(path)
|
2021-03-03 05:12:44 +03:00
|
|
|
tracing.debug("remove mirrored %s" % path)
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
# The working copy is in sync. No need to fixup again.
|
|
|
|
_nodemirrored.clear()
|
|
|
|
|
|
|
|
repo.dirstate.addparentchangecallback("dirsync", dirsyncfixup)
|
|
|
|
|
|
|
|
|
dirsync: disable dirsync during shelve/unshelve
Summary:
Shelve and unshelve use commits under the hood to bundle up data. dirsync sees
these commits happening and performs mirrors, which then shows up at unbundle
time, making it impossible to mirror the change again. Example: if I change X,
and it gets mirrored to Y, if I go back and change X later after a unshelve, hg
commit now fails because Y has pending changes and differs from X.
Test Plan: Added a test
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Subscribers: quark, wez, mjpieters, frantic, wluh
Differential Revision: https://phabricator.intern.facebook.com/D3266602
Signature: t1:3266602:1462484017:738f67c0ab4b5af999819d3855c1f4ba6b2ea338
2016-05-06 00:51:59 +03:00
|
|
|
def _bypassdirsync(orig, ui, repo, *args, **kwargs):
|
2017-03-18 05:42:50 +03:00
|
|
|
_disabled[0] = True
|
dirsync: disable dirsync during shelve/unshelve
Summary:
Shelve and unshelve use commits under the hood to bundle up data. dirsync sees
these commits happening and performs mirrors, which then shows up at unbundle
time, making it impossible to mirror the change again. Example: if I change X,
and it gets mirrored to Y, if I go back and change X later after a unshelve, hg
commit now fails because Y has pending changes and differs from X.
Test Plan: Added a test
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Subscribers: quark, wez, mjpieters, frantic, wluh
Differential Revision: https://phabricator.intern.facebook.com/D3266602
Signature: t1:3266602:1462484017:738f67c0ab4b5af999819d3855c1f4ba6b2ea338
2016-05-06 00:51:59 +03:00
|
|
|
try:
|
|
|
|
return orig(ui, repo, *args, **kwargs)
|
|
|
|
finally:
|
2017-03-18 05:42:50 +03:00
|
|
|
_disabled[0] = False
|
2015-10-31 01:37:17 +03:00
|
|
|
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
def getconfigs(wctx):
|
|
|
|
"""returns {name: [path]}.
|
|
|
|
[path] under a same name are synced. name is not useful.
|
|
|
|
"""
|
|
|
|
# read from .hgdirsync in repo
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
filename = ".hgdirsync"
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
try:
|
|
|
|
content = pycompat.decodeutf8(wctx[filename].data())
|
|
|
|
except (error.ManifestLookupError, IOError, AttributeError, KeyError):
|
|
|
|
content = ""
|
2018-07-30 17:55:50 +03:00
|
|
|
cfg = config.config()
|
2017-03-18 02:35:08 +03:00
|
|
|
if content:
|
2018-07-30 17:55:50 +03:00
|
|
|
cfg.parse(filename, "[dirsync]\n%s" % content, ["dirsync"])
|
2017-03-18 02:35:08 +03:00
|
|
|
|
2017-09-22 01:57:11 +03:00
|
|
|
maps = util.sortdict()
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
repo = wctx.repo()
|
2018-07-30 17:55:50 +03:00
|
|
|
for key, value in repo.ui.configitems("dirsync") + cfg.items("dirsync"):
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
if "." not in key:
|
2015-10-31 01:37:17 +03:00
|
|
|
continue
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
name, disambig = key.split(".", 1)
|
2015-10-31 01:37:17 +03:00
|
|
|
# Normalize paths to have / at the end. For easy concatenation later.
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
if value[-1] != "/":
|
|
|
|
value = value + "/"
|
2017-09-22 01:57:11 +03:00
|
|
|
if name not in maps:
|
|
|
|
maps[name] = []
|
2015-10-31 01:37:17 +03:00
|
|
|
maps[name].append(value)
|
|
|
|
return maps
|
|
|
|
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
dirsync: improve performance for codemods
Summary:
Use the Rust tree matcher to rule out files that do not need dirsync quickly.
This should make codemod faster to commit.
I created 5000 files outside dirsync config in fbsource (with 494 lines of
dirsync config), and `hg add` them:
$ mkdir unused
$ cd unused
$ for f in `seq 5000`; do touch $f; done
$ hg add .
Baseline "status":
In [3]: %time repo.status();1
CPU times: user 111 ms, sys: 10.2 ms, total: 122 ms
Wall time: 148 ms
Before, dirsync overhead is ~8x "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.37 s, sys: 28.8 ms, total: 1.4 s
Wall time: 1.79 s
Out[1]: (<workingctx f23d7c84c5a7+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.07 s, sys: 8.41 ms, total: 1.08 s
Wall time: 1.11 s
Out[2]: (<workingctx f23d7c84c5a7+>, set())
After, dirsync overhead is ~1/2 of "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 203 ms, sys: 18.9 ms, total: 222 ms
Wall time: 245 ms
Out[1]: (<workingctx 8ff14e46c9d8+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 154 ms, sys: 24.1 ms, total: 178 ms
Wall time: 202 ms
Out[2]: (<workingctx 8ff14e46c9d8+>, set())
Reviewed By: DurhamG
Differential Revision: D26726476
fbshipit-source-id: e34218769c779c9a4ee64c654c75298b7c79f213
2021-03-03 05:09:48 +03:00
|
|
|
def configstomatcher(configs):
|
|
|
|
"""returns a matcher matching files need to be dirsynced
|
|
|
|
|
|
|
|
configs is the return value of getconfigs()
|
|
|
|
"""
|
|
|
|
rules = set()
|
|
|
|
for mirrors in configs.values():
|
|
|
|
for mirror in mirrors:
|
|
|
|
assert mirror.endswith("/"), "getconfigs() ensures this"
|
|
|
|
rules.add("%s**" % mirror)
|
|
|
|
m = bindings.pathmatcher.treematcher(sorted(rules))
|
|
|
|
return m
|
|
|
|
|
|
|
|
|
2015-10-31 01:37:17 +03:00
|
|
|
def getmirrors(maps, filename):
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
"""Returns (srcmirror, mirrors)
|
|
|
|
|
|
|
|
srcmirror is the mirror path the filename is in.
|
|
|
|
mirrors is a list of mirror paths, including srcmirror.
|
|
|
|
"""
|
2021-02-12 02:52:56 +03:00
|
|
|
# The getconfigs() code above adds "/" to the end of all of the entries.
|
|
|
|
# This makes it easy to check that we are only matching full directory path name
|
|
|
|
# components (e.g., so we don't incorrectly treat "foobar/test.txt" as matching a
|
|
|
|
# rule for "foo").
|
|
|
|
#
|
|
|
|
# However, we do want to allow rules to match exact file names too, and not just
|
|
|
|
# directory prefixes. Therefore when looking matches append "/" to the end of the
|
|
|
|
# filename that we are checking.
|
|
|
|
checkpath = filename + "/"
|
|
|
|
|
2019-01-15 18:11:49 +03:00
|
|
|
if EXCLUDE_PATHS in maps:
|
|
|
|
for subdir in maps[EXCLUDE_PATHS]:
|
2021-02-12 02:52:56 +03:00
|
|
|
if checkpath.startswith(subdir):
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
return None, []
|
2019-01-15 18:11:49 +03:00
|
|
|
|
2021-02-12 02:52:56 +03:00
|
|
|
for key, mirrordirs in maps.items():
|
2015-10-31 01:37:17 +03:00
|
|
|
for subdir in mirrordirs:
|
2021-02-12 02:52:56 +03:00
|
|
|
if checkpath.startswith(subdir):
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
return subdir, mirrordirs
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
return None, []
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
2017-09-08 20:01:05 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
def _mctxstatus(ctx, matcher=None):
|
|
|
|
"""Figure out what has changed that need to be synced
|
2015-10-31 01:37:17 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
Return (mctx, status).
|
|
|
|
- mctx: mirrored ctx
|
|
|
|
- status: 'scmutil.status' struct for changes that need to be
|
|
|
|
considered for dirsync.
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
There are different cases. For example:
|
2015-11-25 22:21:47 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
Commit: compare with p1:
|
|
|
|
|
|
|
|
o ctx
|
|
|
|
|
|
|
|
|
o ctx.p1
|
|
|
|
|
|
|
|
Rebase (and other parent-change mutations): compare with p1 (re-sync),
|
|
|
|
because comparing with pred can be slow due to potential long distance
|
|
|
|
|
|
|
|
o pred -----> o ctx
|
|
|
|
| |
|
|
|
|
o pred.p1 o ctx.p1
|
|
|
|
|
|
|
|
Amend (and other content-change mutations): compare with pred, because
|
|
|
|
comparing with p1 might cause sync conflicts:
|
|
|
|
|
|
|
|
o ctx
|
|
|
|
|
|
|
|
|
| o pred
|
|
|
|
|/
|
|
|
|
o pred.p1, ctx.p1
|
|
|
|
|
|
|
|
Conflict example: dirsync dir1/ dir2/
|
|
|
|
|
|
|
|
ctx.p1:
|
|
|
|
dir1/A = 1
|
|
|
|
dir2/A = 1
|
|
|
|
pred:
|
|
|
|
dir1/A = 2
|
|
|
|
dir2/A = 2
|
|
|
|
ctx:
|
|
|
|
dir1/A = 3
|
|
|
|
dir2/A = 2 <- should be considered as "unchanged"
|
|
|
|
|
|
|
|
Stack amend (ex. absorb): compare with pred to avoid conflicts and pick up
|
|
|
|
needed changes:
|
|
|
|
|
|
|
|
o ctx
|
|
|
|
|
|
|
|
|
o ctx.p1
|
|
|
|
|
|
|
|
|
| o pred(ctx)
|
|
|
|
| |
|
|
|
|
| o pred(ctx.p1), pred(ctx).p1
|
|
|
|
|/
|
|
|
|
o ctx.p1.p1, pred(ctx.p1).p1, pred(ctx).p1.p1
|
|
|
|
|
|
|
|
"""
|
|
|
|
repo = ctx.repo()
|
|
|
|
mctx = context.memctx.mirror(ctx)
|
|
|
|
mutinfo = ctx.mutinfo()
|
|
|
|
|
|
|
|
# mutpred looks like:
|
|
|
|
# hg/2dc0850429134cc0d21d84d6e5a3960faa9aadce,hg/ccfb9effa811b10fdc40e314524f9340c31085d7
|
|
|
|
# The first one is the "top" of the stack that we care about.
|
|
|
|
mutpredhex = mutinfo and mutinfo["mutpred"].split(",", 1)[0].lstrip("hg/")
|
|
|
|
mutpred = mutpredhex and bin(mutpredhex)
|
|
|
|
predctx = mutpred and mutpred in repo and repo[mutpred] or None
|
|
|
|
|
|
|
|
if predctx and (predctx.p1() == ctx.p1() or ctx.p1().node() in _nodemirrored):
|
|
|
|
# "Amend" or stack amend case. Compare with with pred, not p1
|
|
|
|
status = _status(predctx, ctx)
|
|
|
|
else:
|
|
|
|
# Other cases. Compare with mctx.p1.
|
|
|
|
# _status is a fast path.
|
|
|
|
status = mctx._status
|
|
|
|
|
|
|
|
return mctx, status
|
|
|
|
|
|
|
|
|
|
|
|
def dirsyncctx(ctx, matcher=None):
|
|
|
|
"""for changes in ctx that matches matcher, apply dirsync rules
|
|
|
|
|
|
|
|
Return:
|
|
|
|
|
|
|
|
(newctx, {path})
|
|
|
|
|
|
|
|
This function does not change working copy or dirstate.
|
|
|
|
"""
|
|
|
|
maps = getconfigs(ctx)
|
|
|
|
resultmirrored = set()
|
|
|
|
resultctx = ctx
|
|
|
|
if not maps:
|
|
|
|
return resultctx, resultmirrored
|
|
|
|
|
dirsync: improve performance for codemods
Summary:
Use the Rust tree matcher to rule out files that do not need dirsync quickly.
This should make codemod faster to commit.
I created 5000 files outside dirsync config in fbsource (with 494 lines of
dirsync config), and `hg add` them:
$ mkdir unused
$ cd unused
$ for f in `seq 5000`; do touch $f; done
$ hg add .
Baseline "status":
In [3]: %time repo.status();1
CPU times: user 111 ms, sys: 10.2 ms, total: 122 ms
Wall time: 148 ms
Before, dirsync overhead is ~8x "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.37 s, sys: 28.8 ms, total: 1.4 s
Wall time: 1.79 s
Out[1]: (<workingctx f23d7c84c5a7+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.07 s, sys: 8.41 ms, total: 1.08 s
Wall time: 1.11 s
Out[2]: (<workingctx f23d7c84c5a7+>, set())
After, dirsync overhead is ~1/2 of "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 203 ms, sys: 18.9 ms, total: 222 ms
Wall time: 245 ms
Out[1]: (<workingctx 8ff14e46c9d8+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 154 ms, sys: 24.1 ms, total: 178 ms
Wall time: 202 ms
Out[2]: (<workingctx 8ff14e46c9d8+>, set())
Reviewed By: DurhamG
Differential Revision: D26726476
fbshipit-source-id: e34218769c779c9a4ee64c654c75298b7c79f213
2021-03-03 05:09:48 +03:00
|
|
|
needsync = configstomatcher(maps)
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
repo = ctx.repo()
|
|
|
|
mctx, status = _mctxstatus(ctx)
|
|
|
|
|
|
|
|
added = set(status.added)
|
|
|
|
modified = set(status.modified)
|
|
|
|
removed = set(status.removed)
|
|
|
|
|
|
|
|
if matcher is None:
|
|
|
|
matcher = lambda path: True
|
|
|
|
|
|
|
|
for action, paths in (
|
|
|
|
("a", status.added),
|
|
|
|
("m", status.modified),
|
|
|
|
("r", status.removed),
|
|
|
|
):
|
|
|
|
for src in paths:
|
dirsync: improve performance for codemods
Summary:
Use the Rust tree matcher to rule out files that do not need dirsync quickly.
This should make codemod faster to commit.
I created 5000 files outside dirsync config in fbsource (with 494 lines of
dirsync config), and `hg add` them:
$ mkdir unused
$ cd unused
$ for f in `seq 5000`; do touch $f; done
$ hg add .
Baseline "status":
In [3]: %time repo.status();1
CPU times: user 111 ms, sys: 10.2 ms, total: 122 ms
Wall time: 148 ms
Before, dirsync overhead is ~8x "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.37 s, sys: 28.8 ms, total: 1.4 s
Wall time: 1.79 s
Out[1]: (<workingctx f23d7c84c5a7+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 1.07 s, sys: 8.41 ms, total: 1.08 s
Wall time: 1.11 s
Out[2]: (<workingctx f23d7c84c5a7+>, set())
After, dirsync overhead is ~1/2 of "status":
In [1]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 203 ms, sys: 18.9 ms, total: 222 ms
Wall time: 245 ms
Out[1]: (<workingctx 8ff14e46c9d8+>, set())
In [2]: %time x.dirsync.dirsyncctx(repo[None])
CPU times: user 154 ms, sys: 24.1 ms, total: 178 ms
Wall time: 202 ms
Out[2]: (<workingctx 8ff14e46c9d8+>, set())
Reviewed By: DurhamG
Differential Revision: D26726476
fbshipit-source-id: e34218769c779c9a4ee64c654c75298b7c79f213
2021-03-03 05:09:48 +03:00
|
|
|
if not needsync.matches(src) or not matcher(src):
|
2015-10-31 01:37:17 +03:00
|
|
|
continue
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
srcmirror, mirrors = getmirrors(maps, src)
|
|
|
|
if not mirrors:
|
2015-10-31 01:37:17 +03:00
|
|
|
continue
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
|
|
|
|
dstpaths = [] # [(dstpath, dstmirror)]
|
|
|
|
for dstmirror in (m for m in mirrors if m != srcmirror):
|
|
|
|
dst = _mirrorpath(srcmirror, dstmirror, src)
|
|
|
|
dstpaths.append((dst, dstmirror))
|
|
|
|
|
|
|
|
if action == "r":
|
|
|
|
fsrc = None
|
2015-10-31 01:37:17 +03:00
|
|
|
else:
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
fsrc = ctx[src]
|
|
|
|
for dst, dstmirror in dstpaths:
|
|
|
|
# changed: whether ctx[dst] is changed, according to status.
|
|
|
|
# conflict: whether the dst change conflicts with src change.
|
|
|
|
if dst in removed:
|
|
|
|
conflict, changed = (action != "r"), True
|
|
|
|
elif dst in modified or dst in added:
|
|
|
|
conflict, changed = (fsrc is None or ctx[dst].cmp(fsrc)), True
|
|
|
|
else:
|
|
|
|
conflict = changed = False
|
|
|
|
if conflict:
|
|
|
|
raise error.Abort(
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
_(
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
"path '%s' needs to be mirrored to '%s', but "
|
|
|
|
"the target already has pending changes"
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
)
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
% (src, dst)
|
codemod: join the auto-formatter party
Summary:
Turned on the auto formatter. Ran `arc lint --apply-patches --take BLACK **/*.py`.
Then run `arc lint` again so some other autofixers like spellchecker etc. looked
at the code base. Manually accept the changes whenever they make sense, or use
a workaround (ex. changing "dict()" to "dict constructor") where autofix is false
positive. Disabled linters on files that are hard (i18n/polib.py) to fix, or less
interesting to fix (hgsubversion tests), or cannot be fixed without breaking
OSS build (FBPYTHON4).
Conflicted linters (test-check-module-imports.t, part of test-check-code.t,
test-check-pyflakes.t) are removed or disabled.
Duplicated linters (test-check-pyflakes.t, test-check-pylint.t) are removed.
An issue of the auto-formatter is lines are no longer guarnateed to be <= 80
chars. But that seems less important comparing with the benefit auto-formatter
provides.
As we're here, also remove test-check-py3-compat.t, as it is currently broken
if `PYTHON3=/bin/python3` is set.
Reviewed By: wez, phillco, simpkins, pkaush, singhsrb
Differential Revision: D8173629
fbshipit-source-id: 90e248ae0c5e6eaadbe25520a6ee42d32005621b
2018-05-26 07:34:37 +03:00
|
|
|
)
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
if changed:
|
|
|
|
if action == "r":
|
|
|
|
fmt = _(
|
|
|
|
"not mirroring remove of '%s' to '%s'; it is already removed\n"
|
|
|
|
)
|
|
|
|
else:
|
|
|
|
fmt = _("not mirroring '%s' to '%s'; it already matches\n")
|
|
|
|
repo.ui.note(fmt % (src, dst))
|
|
|
|
continue
|
|
|
|
|
|
|
|
# Mirror copyfrom, too.
|
|
|
|
renamed = fsrc and fsrc.renamed()
|
|
|
|
fmirror = fsrc
|
|
|
|
msg = None
|
|
|
|
if renamed:
|
|
|
|
copyfrom, copynode = renamed
|
|
|
|
newcopyfrom = _mirrorpath(srcmirror, dstmirror, copyfrom)
|
|
|
|
if newcopyfrom:
|
|
|
|
if action == "a":
|
|
|
|
msg = _("mirrored copy '%s -> %s' to '%s -> %s'\n") % (
|
|
|
|
copyfrom,
|
|
|
|
src,
|
|
|
|
newcopyfrom,
|
|
|
|
dst,
|
|
|
|
)
|
|
|
|
fmirror = context.overlayfilectx(
|
|
|
|
fsrc, copied=(newcopyfrom, copynode)
|
|
|
|
)
|
|
|
|
|
|
|
|
mctx[dst] = fmirror
|
|
|
|
resultmirrored.add(dst)
|
|
|
|
|
|
|
|
if msg is None:
|
|
|
|
if action == "a":
|
|
|
|
fmt = _("mirrored adding '%s' to '%s'\n")
|
|
|
|
elif action == "m":
|
|
|
|
fmt = _("mirrored changes in '%s' to '%s'\n")
|
|
|
|
else:
|
|
|
|
fmt = _("mirrored remove of '%s' to '%s'\n")
|
|
|
|
msg = fmt % (src, dst)
|
|
|
|
repo.ui.status(msg)
|
|
|
|
|
|
|
|
if resultmirrored:
|
|
|
|
resultctx = mctx
|
|
|
|
|
|
|
|
return resultctx, resultmirrored
|
|
|
|
|
|
|
|
|
|
|
|
def _mirrorpath(srcdir, dstdir, src):
|
|
|
|
"""Mirror src path from srcdir to dstdir. Return None if src is not in srcdir."""
|
|
|
|
if src + "/" == srcdir:
|
|
|
|
# special case: src is a file to mirror
|
|
|
|
return dstdir.rstrip("/")
|
|
|
|
elif src.startswith(srcdir):
|
|
|
|
relsrc = src[len(srcdir) :]
|
|
|
|
return dstdir + relsrc
|
|
|
|
else:
|
|
|
|
return None
|
|
|
|
|
|
|
|
|
|
|
|
def _status(ctx1, ctx2):
|
|
|
|
"""Similar to ctx1.status(ctx2) but remove false positive modifies.
|
|
|
|
|
|
|
|
The false positive might happen if a file in a "commitablectx" changes
|
|
|
|
back to match its p1 content. Context like `memctx` currently does not
|
|
|
|
read file content in its `status` calculation.
|
|
|
|
|
|
|
|
This might belong to ctx.status(). However, the performance penalty might
|
|
|
|
be undesirable.
|
|
|
|
"""
|
|
|
|
status = ctx1.status(ctx2)
|
|
|
|
newmodified = []
|
|
|
|
newadded = []
|
|
|
|
for oldpaths, newpaths in [
|
|
|
|
(status.modified, newmodified),
|
|
|
|
(status.added, newadded),
|
|
|
|
]:
|
|
|
|
for path in oldpaths:
|
|
|
|
# No real changes?
|
|
|
|
if path in ctx1 and path in ctx2:
|
|
|
|
f1 = ctx1[path]
|
|
|
|
f2 = ctx2[path]
|
|
|
|
if f1.flags() == f2.flags() and not f1.cmp(f2):
|
|
|
|
continue
|
|
|
|
newpaths.append(path)
|
|
|
|
return scmutil.status(newmodified, newadded, status.removed, [], [], [], [])
|
|
|
|
|
|
|
|
|
|
|
|
def _commitctx(orig, self, ctx, *args, **kwargs):
|
|
|
|
if _disabled[0]:
|
|
|
|
return orig(self, ctx, *args, **kwargs)
|
|
|
|
|
|
|
|
ctx, mirrored = dirsyncctx(ctx)
|
|
|
|
node = orig(self, ctx, *args, **kwargs)
|
|
|
|
|
|
|
|
if mirrored:
|
|
|
|
# used by dirsyncfixup to write back from commit to disk
|
|
|
|
_nodemirrored[node] = mirrored
|
2015-11-25 22:21:47 +03:00
|
|
|
|
dirsync: improve in-memory commit compatibility
Summary:
Previously, dirsync wraps `repo.commit`, requires an on-disk working copy,
and dirstate to work properly. This diff updates dirsync to wrap
`repo.commitctx` instead, do commit edits purely in memory, then sync the
commit back to disk. It makes dirsync compatible with absorb and in-memory
rebase (and potentially other things like drawdag, if drawdag's context APIs
are improved).
To sync the changes made in-memory back to the filesystem, a dirstate callback
is added to write back mirrored files from commit to disk. This works for both
amend and absorb so the special wrapper about amend is dropped. It is
also optimal for absorb, because it only writes the mirrored files once for
the "final" commit, instead of writing the files for each commit in the stack.
Some `O(N^2)`s (N: len(status)) complexities were avoided:
- `applytomirrors` was called N times.
- `allchanges = set(status.modified + status.removed + status.added)` in
`applytomirrors` was O(N).
- `sourcepath in status.removed` in `applytomirrors` was O(N).
- `mirrorpath in status.removed` in `applytomirrors` was O(N).
Note there is still a suboptimal complexity of `getmirrors` called per changed
path, `O(N*M)` (N: len(status), M: len(dirsync_conig)). That will be addressed
in a later diff.
Reviewed By: DurhamG
Differential Revision: D26726479
fbshipit-source-id: 482c6c830ab65cc0d9cd569a51ec610a1dac49cc
2021-03-03 05:09:48 +03:00
|
|
|
return node
|