facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-08 15:57:43 +03:00
Code Issues Projects Releases Wiki Activity
6dce563cd3
sapling/mercurial/sparse.py

303 lines
9.4 KiB
Python
Raw Normal View History

sparse: move config parsing into core This patch marks the beginning of moving code from the sparse extension into core. The goal is to move as much of the functionality as possible into core, where it will be an experimental feature. The extension will likely continue to exist to enable the feature and provide UI elements. As part of the move, the repo method was converted to a module function. It doesn't need to exist on repos. An error message was also updated to reflect that an error isn't necessarily from the .hg/sparse file. The API should be updated later to pass in a filename so the error can be more descriptive. Copyright of the added file was copied from the sparse extension.
2017-07-06 22:14:03 +03:00
# sparse.py - functionality for sparse checkouts
#
# Copyright 2014 Facebook, Inc.
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import
sparse: move pruning of temporary includes into core This was our last method on the custom repo type, meaning we could remove that custom type and inline the 2 lines of code into reposetup(). As part of the move, instead of wrapping merge.update() from the sparse extension, we inline the function call. The ported function now no-ops if sparse isn't enabled, making it safe to always call. The call site in update() may not be the most appropriate. But it matches the previous behavior, which is the safest thing to do. It can be improved later.
2017-07-07 00:33:18 +03:00
import collections
sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.
2017-07-07 02:11:56 +03:00
import hashlib
sparse: move function for resolving sparse matcher into core As part of the move, the function arguments changed so revs are passed as a list instead of *args. This allows us to use keyword arguments properly. Since the plan is to integrate sparse into core and have it enabled by default, we need to prepare for a sparse matcher to always be obtained and operated on. As part of the move, we inserted code that returns an always matcher if sparse isn't enabled. Some callers in the sparse extension take this into account and conditionally perform matching depending on whether the special always matcher is seen. I /think/ this may have sped up some operations where the extension is installed but no sparse config is activated. One thing I'm ensure of in this code is whether os.path.dirname() is semantically correct. os.posixpath.dirname() (which is exported as pathutil.dirname) might be a better choise because all patterns should be using posix directory separators (/) instead of Windows (\). There's an inline comment that implies Windows was tested. So hopefully it won't be a problem. We can improve this in a follow-up. I've added a TODO to track it.
2017-07-07 03:41:45 +03:00
import os
sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.
2017-07-07 02:11:56 +03:00
sparse: move config parsing into core This patch marks the beginning of moving code from the sparse extension into core. The goal is to move as much of the functionality as possible into core, where it will be an experimental feature. The extension will likely continue to exist to enable the feature and provide UI elements. As part of the move, the repo method was converted to a module function. It doesn't need to exist on repos. An error message was also updated to reflect that an error isn't necessarily from the .hg/sparse file. The API should be updated later to pass in a filename so the error can be more descriptive. Copyright of the added file was copied from the sparse extension.
2017-07-06 22:14:03 +03:00
from .i18n import _
sparse: move active profiles function into core Also includes some light formatting changes.
2017-07-06 22:26:04 +03:00
from .node import nullid
sparse: move config parsing into core This patch marks the beginning of moving code from the sparse extension into core. The goal is to move as much of the functionality as possible into core, where it will be an experimental feature. The extension will likely continue to exist to enable the feature and provide UI elements. As part of the move, the repo method was converted to a module function. It doesn't need to exist on repos. An error message was also updated to reflect that an error isn't necessarily from the .hg/sparse file. The API should be updated later to pass in a filename so the error can be more descriptive. Copyright of the added file was copied from the sparse extension.
2017-07-06 22:14:03 +03:00
from . import (
error,
sparse: move function for resolving sparse matcher into core As part of the move, the function arguments changed so revs are passed as a list instead of *args. This allows us to use keyword arguments properly. Since the plan is to integrate sparse into core and have it enabled by default, we need to prepare for a sparse matcher to always be obtained and operated on. As part of the move, we inserted code that returns an always matcher if sparse isn't enabled. Some callers in the sparse extension take this into account and conditionally perform matching depending on whether the special always matcher is seen. I /think/ this may have sped up some operations where the extension is installed but no sparse config is activated. One thing I'm ensure of in this code is whether os.path.dirname() is semantically correct. os.posixpath.dirname() (which is exported as pathutil.dirname) might be a better choise because all patterns should be using posix directory separators (/) instead of Windows (\). There's an inline comment that implies Windows was tested. So hopefully it won't be a problem. We can improve this in a follow-up. I've added a TODO to track it.
2017-07-07 03:41:45 +03:00
match as matchmod,
sparse: move pruning of temporary includes into core This was our last method on the custom repo type, meaning we could remove that custom type and inline the 2 lines of code into reposetup(). As part of the move, instead of wrapping merge.update() from the sparse extension, we inline the function call. The ported function now no-ops if sparse isn't enabled, making it safe to always call. The call site in update() may not be the most appropriate. But it matches the previous behavior, which is the safest thing to do. It can be improved later.
2017-07-07 00:33:18 +03:00
merge as mergemod,
sparse: move function for resolving sparse matcher into core As part of the move, the function arguments changed so revs are passed as a list instead of *args. This allows us to use keyword arguments properly. Since the plan is to integrate sparse into core and have it enabled by default, we need to prepare for a sparse matcher to always be obtained and operated on. As part of the move, we inserted code that returns an always matcher if sparse isn't enabled. Some callers in the sparse extension take this into account and conditionally perform matching depending on whether the special always matcher is seen. I /think/ this may have sped up some operations where the extension is installed but no sparse config is activated. One thing I'm ensure of in this code is whether os.path.dirname() is semantically correct. os.posixpath.dirname() (which is exported as pathutil.dirname) might be a better choise because all patterns should be using posix directory separators (/) instead of Windows (\). There's an inline comment that implies Windows was tested. So hopefully it won't be a problem. We can improve this in a follow-up. I've added a TODO to track it.
2017-07-07 03:41:45 +03:00
pycompat,
sparse: move config parsing into core This patch marks the beginning of moving code from the sparse extension into core. The goal is to move as much of the functionality as possible into core, where it will be an experimental feature. The extension will likely continue to exist to enable the feature and provide UI elements. As part of the move, the repo method was converted to a module function. It doesn't need to exist on repos. An error message was also updated to reflect that an error isn't necessarily from the .hg/sparse file. The API should be updated later to pass in a filename so the error can be more descriptive. Copyright of the added file was copied from the sparse extension.
2017-07-06 22:14:03 +03:00
)
sparse: variable to track if sparse is enabled Currently, the sparse extension sniffs repo instances for attributes defined by the sparse extension to determine if sparse is enabled. As we move code away from repo instances, these checks will be a bit more brittle. We introduce a module-level variable to track whether sparse is enabled as a temporary workaround.
2017-07-06 22:06:37 +03:00
# Whether sparse features are enabled. This variable is intended to be
# temporary to facilitate porting sparse to core. It should eventually be
# a per-repo option, possibly a repo requirement.
enabled = False
sparse: move config parsing into core This patch marks the beginning of moving code from the sparse extension into core. The goal is to move as much of the functionality as possible into core, where it will be an experimental feature. The extension will likely continue to exist to enable the feature and provide UI elements. As part of the move, the repo method was converted to a module function. It doesn't need to exist on repos. An error message was also updated to reflect that an error isn't necessarily from the .hg/sparse file. The API should be updated later to pass in a filename so the error can be more descriptive. Copyright of the added file was copied from the sparse extension.
2017-07-06 22:14:03 +03:00
def parseconfig(ui, raw):
"""Parse sparse config file content.
Returns a tuple of includes, excludes, and profiles.
"""
includes = set()
excludes = set()
current = includes
profiles = []
for line in raw.split('\n'):
line = line.strip()
if not line or line.startswith('#'):
# empty or comment line, skip
continue
elif line.startswith('%include '):
line = line[9:].strip()
if line:
profiles.append(line)
elif line == '[include]':
if current != includes:
# TODO pass filename into this API so we can report it.
raise error.Abort(_('sparse config cannot have includes ' +
'after excludes'))
continue
elif line == '[exclude]':
current = excludes
elif line:
if line.strip().startswith('/'):
ui.warn(_('warning: sparse profile cannot use' +
' paths starting with /, ignoring %s\n') % line)
continue
current.add(line)
return includes, excludes, profiles
sparse: move profile reading into core One more step towards weaning off methods on repo instances and moving code to core. While this function is only used once and is simple, it needs to exist on its own so Facebook can monkeypatch it to enable simplecache integration.
2017-07-06 22:14:12 +03:00
# Exists as separate function to facilitate monkeypatching.
def readprofile(repo, profile, changeid):
"""Resolve the raw content of a sparse profile file."""
# TODO add some kind of cache here because this incurs a manifest
# resolve and can be slow.
return repo.filectx(profile, changeid=changeid).data()
sparse: move resolving of sparse patterns for rev into core This method is reasonably well-contained and simple to move. As part of the move, some light formatting was performed. A "working copy" reference in an error message was changed to "working directory." The biggest change was to _refreshoncommit() in sparse.py. It was previously checking for the existence of an attribute on the repo instance. Since the moved function now returns empty data if sparse isn't enabled, we unconditionally call the new function. However, we do have to protect another method call in that function. This will all be unhacked eventually.
2017-07-06 22:15:14 +03:00
def patternsforrev(repo, rev):
"""Obtain sparse checkout patterns for the given rev.
Returns a tuple of iterables representing includes, excludes, and
patterns.
"""
# Feature isn't enabled. No-op.
if not enabled:
return set(), set(), []
raw = repo.vfs.tryread('sparse')
if not raw:
return set(), set(), []
if rev is None:
raise error.Abort(_('cannot parse sparse patterns from working '
'directory'))
includes, excludes, profiles = parseconfig(repo.ui, raw)
ctx = repo[rev]
if profiles:
visited = set()
while profiles:
profile = profiles.pop()
if profile in visited:
continue
visited.add(profile)
try:
raw = readprofile(repo, profile, rev)
except error.ManifestLookupError:
msg = (
"warning: sparse profile '%s' not found "
"in rev %s - ignoring it\n" % (profile, ctx))
# experimental config: sparse.missingwarning
if repo.ui.configbool(
'sparse', 'missingwarning', True):
repo.ui.warn(msg)
else:
repo.ui.debug(msg)
continue
pincludes, pexcludes, subprofs = parseconfig(repo.ui, raw)
includes.update(pincludes)
excludes.update(pexcludes)
for subprofile in subprofs:
profiles.append(subprofile)
profiles = visited
if includes:
includes.add('.hg*')
return includes, excludes, profiles
sparse: move active profiles function into core Also includes some light formatting changes.
2017-07-06 22:26:04 +03:00
def activeprofiles(repo):
revs = [repo.changelog.rev(node) for node in
repo.dirstate.parents() if node != nullid]
profiles = set()
for rev in revs:
profiles.update(patternsforrev(repo, rev)[2])
return profiles
localrepo: add sparse caches The sparse extension maintains caches for the sparse files to a signature and a signature to a matcher. This allows the sparse matchers to be resolved quickly, which is apparently something that can occur in loops. This patch ports the sparse caches to the localrepo class pretty much as-is. There is potentially room to improve the caching mechanism. But that can be done as a follow-up. The default invalidatecaches() now clears the relevant sparse cache. invalidatesignaturecache() has been moved to sparse.py.
2017-07-06 22:20:53 +03:00
def invalidatesignaturecache(repo):
repo._sparsesignaturecache.clear()
sparse: move config file writing into core The code was refactored during the move to be more procedural instead of using string formatting. This has the benefit of not writing empty sections, which changed tests.
2017-07-06 22:24:55 +03:00
sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.
2017-07-07 02:11:56 +03:00
def configsignature(repo, includetemp=True):
"""Obtain the signature string for the current sparse configuration.
This is used to construct a cache key for matchers.
"""
cache = repo._sparsesignaturecache
signature = cache.get('signature')
if includetemp:
tempsignature = cache.get('tempsignature')
else:
sparse: clean up config signature code Before, 0 was being used as the default signature value and we cast the int to a string. We also handled I/O exceptions manually. The new code uses cfs.tryread() so we always feed data into the hasher. The empty string does hash and and should be suitable for input into a cache key. The changes made the code simple enough that the separate checksum function could be inlined.
2017-07-07 02:01:36 +03:00
tempsignature = '0'
sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.
2017-07-07 02:11:56 +03:00
if signature is None or (includetemp and tempsignature is None):
sparse: clean up config signature code Before, 0 was being used as the default signature value and we cast the int to a string. We also handled I/O exceptions manually. The new code uses cfs.tryread() so we always feed data into the hasher. The empty string does hash and and should be suitable for input into a cache key. The changes made the code simple enough that the separate checksum function could be inlined.
2017-07-07 02:01:36 +03:00
signature = hashlib.sha1(repo.vfs.tryread('sparse')).hexdigest()
sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.
2017-07-07 02:11:56 +03:00
cache['signature'] = signature
if includetemp:
sparse: clean up config signature code Before, 0 was being used as the default signature value and we cast the int to a string. We also handled I/O exceptions manually. The new code uses cfs.tryread() so we always feed data into the hasher. The empty string does hash and and should be suitable for input into a cache key. The changes made the code simple enough that the separate checksum function could be inlined.
2017-07-07 02:01:36 +03:00
raw = repo.vfs.tryread('tempsparse')
tempsignature = hashlib.sha1(raw).hexdigest()
sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.
2017-07-07 02:11:56 +03:00
cache['tempsignature'] = tempsignature
sparse: clean up config signature code Before, 0 was being used as the default signature value and we cast the int to a string. We also handled I/O exceptions manually. The new code uses cfs.tryread() so we always feed data into the hasher. The empty string does hash and and should be suitable for input into a cache key. The changes made the code simple enough that the separate checksum function could be inlined.
2017-07-07 02:01:36 +03:00
return '%s %s' % (signature, tempsignature)
sparse: move config signature logic into core This is a pretty straightforward port. It will be cleaned up in a subsequent commit.
2017-07-07 02:11:56 +03:00
sparse: move config file writing into core The code was refactored during the move to be more procedural instead of using string formatting. This has the benefit of not writing empty sections, which changed tests.
2017-07-06 22:24:55 +03:00
def writeconfig(repo, includes, excludes, profiles):
"""Write the sparse config file given a sparse configuration."""
with repo.vfs('sparse', 'wb') as fh:
for p in sorted(profiles):
fh.write('%%include %s\n' % p)
if includes:
fh.write('[include]\n')
for i in sorted(includes):
fh.write(i)
fh.write('\n')
if excludes:
fh.write('[exclude]\n')
for e in sorted(excludes):
fh.write(e)
fh.write('\n')
invalidatesignaturecache(repo)
sparse: move some temporary includes functions into core Functions for reading and writing the tempsparse file have been moved. prunetemporaryincludes() will be moved separately because it is non-trivial.
2017-07-07 00:48:16 +03:00
def readtemporaryincludes(repo):
raw = repo.vfs.tryread('tempsparse')
if not raw:
return set()
return set(raw.split('\n'))
def writetemporaryincludes(repo, includes):
repo.vfs.write('tempsparse', '\n'.join(sorted(includes)))
invalidatesignaturecache(repo)
def addtemporaryincludes(repo, additional):
includes = readtemporaryincludes(repo)
for i in additional:
includes.add(i)
writetemporaryincludes(repo, includes)
sparse: move function for resolving sparse matcher into core As part of the move, the function arguments changed so revs are passed as a list instead of *args. This allows us to use keyword arguments properly. Since the plan is to integrate sparse into core and have it enabled by default, we need to prepare for a sparse matcher to always be obtained and operated on. As part of the move, we inserted code that returns an always matcher if sparse isn't enabled. Some callers in the sparse extension take this into account and conditionally perform matching depending on whether the special always matcher is seen. I /think/ this may have sped up some operations where the extension is installed but no sparse config is activated. One thing I'm ensure of in this code is whether os.path.dirname() is semantically correct. os.posixpath.dirname() (which is exported as pathutil.dirname) might be a better choise because all patterns should be using posix directory separators (/) instead of Windows (\). There's an inline comment that implies Windows was tested. So hopefully it won't be a problem. We can improve this in a follow-up. I've added a TODO to track it.
2017-07-07 03:41:45 +03:00
sparse: move pruning of temporary includes into core This was our last method on the custom repo type, meaning we could remove that custom type and inline the 2 lines of code into reposetup(). As part of the move, instead of wrapping merge.update() from the sparse extension, we inline the function call. The ported function now no-ops if sparse isn't enabled, making it safe to always call. The call site in update() may not be the most appropriate. But it matches the previous behavior, which is the safest thing to do. It can be improved later.
2017-07-07 00:33:18 +03:00
def prunetemporaryincludes(repo):
if not enabled or not repo.vfs.exists('tempsparse'):
return
origstatus = repo.status()
modified, added, removed, deleted, a, b, c = origstatus
if modified or added or removed or deleted:
# Still have pending changes. Don't bother trying to prune.
return
sparsematch = matcher(repo, includetemp=False)
dirstate = repo.dirstate
actions = []
dropped = []
tempincludes = readtemporaryincludes(repo)
for file in tempincludes:
if file in dirstate and not sparsematch(file):
message = _('dropping temporarily included sparse files')
actions.append((file, None, message))
dropped.append(file)
typeactions = collections.defaultdict(list)
typeactions['r'] = actions
mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False)
# Fix dirstate
for file in dropped:
dirstate.drop(file)
repo.vfs.unlink('tempsparse')
invalidatesignaturecache(repo)
msg = _('cleaned up %d temporarily added file(s) from the '
'sparse checkout\n')
repo.ui.status(msg % len(tempincludes))
sparse: move function for resolving sparse matcher into core As part of the move, the function arguments changed so revs are passed as a list instead of *args. This allows us to use keyword arguments properly. Since the plan is to integrate sparse into core and have it enabled by default, we need to prepare for a sparse matcher to always be obtained and operated on. As part of the move, we inserted code that returns an always matcher if sparse isn't enabled. Some callers in the sparse extension take this into account and conditionally perform matching depending on whether the special always matcher is seen. I /think/ this may have sped up some operations where the extension is installed but no sparse config is activated. One thing I'm ensure of in this code is whether os.path.dirname() is semantically correct. os.posixpath.dirname() (which is exported as pathutil.dirname) might be a better choise because all patterns should be using posix directory separators (/) instead of Windows (\). There's an inline comment that implies Windows was tested. So hopefully it won't be a problem. We can improve this in a follow-up. I've added a TODO to track it.
2017-07-07 03:41:45 +03:00
def matcher(repo, revs=None, includetemp=True):
"""Obtain a matcher for sparse working directories for the given revs.
If multiple revisions are specified, the matcher is the union of all
revs.
``includetemp`` indicates whether to use the temporary sparse profile.
"""
# If sparse isn't enabled, sparse matcher matches everything.
if not enabled:
return matchmod.always(repo.root, '')
if not revs or revs == [None]:
revs = [repo.changelog.rev(node)
for node in repo.dirstate.parents() if node != nullid]
signature = configsignature(repo, includetemp=includetemp)
key = '%s %s' % (signature, ' '.join(map(pycompat.bytestr, revs)))
result = repo._sparsematchercache.get(key)
if result:
return result
matchers = []
for rev in revs:
try:
includes, excludes, profiles = patternsforrev(repo, rev)
if includes or excludes:
# Explicitly include subdirectories of includes so
# status will walk them down to the actual include.
subdirs = set()
for include in includes:
# TODO consider using posix path functions here so Windows
# \ directory separators don't come into play.
dirname = os.path.dirname(include)
# basename is used to avoid issues with absolute
# paths (which on Windows can include the drive).
while os.path.basename(dirname):
subdirs.add(dirname)
dirname = os.path.dirname(dirname)
matcher = matchmod.match(repo.root, '', [],
include=includes, exclude=excludes,
default='relpath')
if subdirs:
matcher = matchmod.forceincludematcher(matcher, subdirs)
matchers.append(matcher)
except IOError:
pass
if not matchers:
result = matchmod.always(repo.root, '')
elif len(matchers) == 1:
result = matchers[0]
else:
result = matchmod.unionmatcher(matchers)
if includetemp:
tempincludes = readtemporaryincludes(repo)
result = matchmod.forceincludematcher(result, tempincludes)
repo._sparsematchercache[key] = result
return result
Reference in New Issue Copy Permalink