Now that the 'vfs' classes moved in their own module, lets use the new module
directly. We update code iteratively to help with possible bisect needs in the
future.
Similar to the previous patch, the .hg/store/meta/ directory does not
get copied when when using "hg clone --uncompressed". Fix by including
"meta/" in store.datafiles(). This seems safe to do, as there are only
a few users of this method. "hg manifest" already filters the paths by
"data/" prefix. The calls from largefiles also seem safe. The use in
verify needs updating to prevent it from mistaking dirlogs for
orphaned filelogs. That change is included in this patch.
Since the dirlogs will now be in the fncache when using fncachestore,
let's also update debugrebuildfncache(). That will also allow any
existing treemanifest repos to get their dirlogs into the fncache.
Also update test-treemanifest.t to use an a directory name that
requires dot-encoding and uppercase-encoding so we test that the path
encoding works.
When doing a local clone with treemanifests, the .hg/store/meta/
directory currently does not get copied. To fix it, all we need to do
is to add it to the list of directories to copy.
The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be
confused about that and gives all the credit to 'util' instead of the
hardworking 'error'. In a spirit of equity, we break the cycle of injustice and
give back to 'error' the respect it deserves. And screw that 'util' poser.
For great justice.
Python 2.6 introduced the "except type as instance" syntax, replacing
the "except type, instance" syntax that came before. Python 3 dropped
support for the latter syntax. Since we no longer support Python 2.4 or
2.5, we have no need to continue supporting the "except type, instance".
This patch mass rewrites the exception syntax to be Python 2.6+ and
Python 3 compatible.
This patch was produced by running `2to3 -f except -w -n .`.
Python 2.6 introduced a new octal syntax: "0oXXX", replacing "0XXX". The
old syntax is not recognized in Python 3 and will result in a parse
error.
Mass rewrite all instances of the old octal syntax to the new syntax.
This patch was generated by `2to3 -f numliterals -w -n .` and the diff
was selectively recorded to exclude changes to "<N>l" syntax conversion,
which will be handled separately.
With this change, when tree manifests are enabled (in .hg/requires),
commits will be written with one manifest revlog per directory. The
manifest revlogs are stored in
.hg/store/meta/$dir/00manifest.[id].
Flat manifests can still be read and interacted with as usual (they
are also read into treemanifest instances). The functionality for
writing treemanifest as a flat manifest to disk is still left in the
code; tests still pass with '_treeinmem=True' hardcoded.
Exchange is not yet implemented.
When encoding long paths, the pure Python code strips the first
directory from the path, while the native code currently strips the
first 5 characters. This discrepancy has not been a problem so far,
since we have not stored anything in directories other than
data/. However, we will soon be storing submanifest revlogs in
metadata/, so the discrepancy will have to go [1]. Since file
collisions are avoided by the hashing alone (which is done on the full
unencoded path), it doesn't really matter whether we drop the first
dir, the first 5 characters, or special-case non-data/. To avoid
touching the C code, let's always strip the first 5 characters.
[1] Or maybe elsewhere, but the discrepancy is ugly either way.
Previously the fncache was cleaned up at read time by noticing when it was out
of sync. This caused writes to happen outside the scope of transactions and
could have caused race conditions. With this change, we'll keep the fncache
up-to-date as we go by removing old entries during repair.strip.
The fncache was not being properly invalidated each time the lock was taken, so
in theory it could contain old data from prior to the caller having the lock.
This changes it to be invalidated as soon as the lock is taken (same as all our
other caches).
Previously the fncache was written at lock.release time. This meant it was not
tracked by a transaction, and if an error occurred during the fncache write it
would fail to update the fncache, but would not rollback the transaction,
resulting in an fncache that was not in sync with the files on disk (which
causes verify to fail, and causes streaming clones to not copy all the revlogs).
This uses the new transaction backup mechanism to make the fncache transacted.
It also moves the fncache from being written at lock.release time, to being
written at transaction.close time.
The fncache could rewrite itself during a read operation if it noticed any
entries that were no longer on disk. This was problematic because it caused
Mercurial to perform write operations outside the scope of a lock or
transaction, which could interefere with any other pending writes.
This will be replaced in a future patch by logic that cleans up the fncache
as files are deleted during strips.
Some extensions find it useful to be able to walk the non-data files in the
repo, so I'm moving that part of the walk to a separate function.
In particular, this allows an extension to interact with only the non-filelog
store data (for instance, when cloning everything but filelogs).
The check pattern only checked for whitespace between keyword and operator.
Now it also warns:
> x = f(),7
missing whitespace after ,
> x = f()+7
missing whitespace in expression
This patch invokes "osutil.listdir()" via vfs object.
The function added newly to "abstractvfs" is named not as "listdir()"
but as "readdir()", because:
- "os.listdir()" seems to be more familiar as "listdir()" than
"osutil.listdir()"
- "osutil.listdir()" returns also type of each files like
"readdir()" POSIX API: even though "d_type" field of "dirent"
structure is defined mainly only on BSD/Linux
This patch invokes "osutil.listdir()" via "rawvfs" object to avoid
filename encoding, because the path passed to "osutil.listdir()"
shouldn't be encoded.
This patch also omits importing "osutil" module, because it is no
longer used.
Adds a __contains__ method to fncachestore to check for file/dir existence (using fncache.__contains__).
Also extends fncache.__contains__ to check for directories (by prefix matching)
This patch invokes "os.path.isdir()" via "rawvfs" object to avoid
filename encoding, because the path passed to "os.path.isdir()"
shouldn't be encoded.
This patch newly adds "self.rawvfs" field only to "basicstore" and
"encodedstore", because "fncachestore" has "self.rawvfs" already.
This patch replaces invocation of "getsize()", which calls "os.stat()"
internally, by "vfs.stat()".
The object referred by "self.rawvfs" is used internally by
"_fncachevfs" and doesn't encode filename for each file API invocation.
This patch invokes "os.stat()" via "self.rawvfs" to avoid redundant
filename encoding: invocation of "os.stat()" via "self.vfs" hides
filename encoding and encoding result from caller, so it is not
appropriate, when both encoded and non-encoded filenames should be
yield.
Even though changeset 89eacc6262af improved stream_out performance by
"self.pathsep + path", this patch replaces it by
"os.path.join(self.base, path)" of vfs. So, this may increase cost to
join path components.
But this shouldn't have large impact, because:
- such cost is much less than cost of "os.stat()" which causes
system call invocation
- "datafiles()" of store object is invoked only for "hg manifest
--all" or "hg verify" which are both heavy functions
This just replaces "os.stat()" invocation: refactoring around
"self.createmode" and "vfs.createmode" initialization is omitted.
This patch also newly adds "stat()" function to "abstractvfs".
