Revlog chunks can be stored uncompressed. If the first byte of the
raw data is \0, we store the data as is. Else we prefix it with 'u'.
Before, we performed a string slice to strip out the 'u' prefix.
With this patch, we use a buffer to avoid an extra memory copy and
associated garbage collection overhead.
I was unable to verify any performance impact of this patch. For both
mozilla-central and the hg repos, the number of manifest revisions
with 'u' prefixes is very small - under 1%. So this change likely
isn't called enough to have an impact on manifest reading. However,
the reasoning behind this change is solid, so it should be safe.
This is less portable than the C version, but PyPy can't load CPython
extensions. So for now, this will be used on PyPy.
I've tested it on Linux amd64 and Mac OS X.
It will be used to attach client's stdio files to a background chg command
server.
The socket module of Python 2.x doesn't provide recvmsg(). This could be
implemented by using ctypes, but it would be less portable than the C version
because the handling of socket ancillary data heavily depends on preprocessor.
Also, some length fields are wrongly typed in the Linux kernel.
As part of investigating performance improvements to revlog reading,
I needed a mechanism to measure every part of revlog reading so I knew
where time was spent and how effective optimizations were.
This patch implements a perf command for benchmarking the various
stages of reading a single revlog revision.
When executed against a manifest revision at the end of a 30,000+
long delta chain in mozilla-central, the command demonstrates that
~80% of time is spent in zlib decompression.
We have a nice API now. Use it.
This does mean we introduce an extra index lookup for each revision.
Considering this is a debug command, the overhead should be acceptable.
We could add the chain size to revlog._deltachain(). However, that
feels like avoidable overhead.
This code is already written in multiple locations.
While this code needs to be fast and extracting it to its own function
adds overhead, code paths reading delta chains typically read,
decompress, and do binary patching on revlog data from the delta chain.
This other work (especially zlib decompression) almost certainly
accounts for a lot more time than the overhead of introducing a Python
function call. So I'm not worried about the performance impact of this
change.
The old code only partially cleared the caches. Now that we have a
comprehensive method for wiping all caches, let's call it.
This appears to introduce a marginal regression in `hg perfmanifest`
on mozilla-central. This is good because the new result is more
accurate since caches aren't being used.
clearcaches() was added several years ago in 1e47437a1ca7 as part
of implementing a perf command. Since revlog instances have many caches
and since the spirit of this mostly unused method is to facilitate
performance testing, I think it's appropriate for all the revlog's
caches to get cleared when it is called.
A fileset predicate can invoke 'matchctx.existing()' successfully,
even if it isn't marked as "existing caller". It is aborted only in
some corner cases: e.g. there were one deleted file in the working
directory (see 2c5c0790cbcc for detail).
This patch makes 'matchctx.existing()' invocation abort if not
'_existingenabled', which is true only while "existing caller"
running.
After this changes, non-"existing caller" predicate function is
aborted immediately, whenever it invokes 'matchctx.existing()'. This
prevent developer from forgetting to mark a predicate as "existing
caller".
BTW, unintentional 'matchctx.status()' invocation can be detected
easily without any additional trick like this patch, because it
returns 'None' if a predicate isn't marked as "status caller", and
referring field (e.g. '.modified') of it is always aborted.
This reduces cost of examining whether given predicate calls
'matchctx.status()' or 'matchctx.existing()' in 'getfileset()' at
runtime.
This kind of examination is used also in subsequent patch, which
detects unintentional 'matchctx.existing()' invocation per each
predicate evaluation.
Before this patch, predicates calling 'matchctx.status()' are listed
up by immediate list value in 'getfileset()'.
This prevents 3rd party extensions from adding specific predicate
calling 'matchctx.status()'.
This uses decorator to mark a predicate as "status caller".
This can also localize changes for adding (or removing) a "status
caller" predicate function in source code.
Using decorator can localize changes for adding (or removing) a
fileset predicate function in source code.
It is also useful to pick predicates up for specific purpose. For
example, subsequent patches marks predicates as "call status" or "use
existing" via decorator.
To avoid (1) redundancy between "predicate name" and (the beginning
of) help document, and (2) accidental typo of help document, this
patch also makes decorator put predicate declration into the beginning
of help.
Before this patch, predicate function 'encoding' and 'eol' aren't
listed up in '_existingcallers', even though they invoke 'existing()'.
This causes unexpected failure of these predicate, if there is a
(manually) deleted file in the working directory.
2c5c0790cbcc and 12b403664548 seem to overlook putting already
existing 'encoding' or newly introduced 'eol' into '_existingcallers'.
This patch also changes order of fileset "eol(unix)" output in test,
because "existing caller" predicates show "A(dded)" files before
"C(lean)" ones.
Nested functions in Python are not able to assign to variables in the
outer scope without something like the list trick because assignments
refer to the inner scope. So, we formerly used a list to give an
object to assign into.
Now that error and warning are object members, the [0] hack is no
longer needed.
This will allow us to start moving some of the nested functions inside verify()
out onto the class.
This will allow extensions to hook into verify more easily.
In order to allow extensions to hook into the verification logic more easily, we
need to refactor it into multiple functions. The first step is to move it to a
class so the shared state can be more easily accessed.
This patch makes hg log <file|folder> faster by using changelog.readfiles
instead of changelog.read.
On our large repos for hg log <file|folder> -l5 operations that were taking:
- ~8s I see a 25% improvement
- ~15s, I see a 35% improvement
For recently modified folder/file, the difference is negligible as we don't
have to consider many revisions.
This patch adds a new method "readfiles" to get the files modified by a
changeset. It extracts some logic from "read" to only return the files modified
by a changeset as efficiently as possible. This is used in the next patch to
speed up hg log <file|folder>
Revlogs were recently refactored to open file handles in "a+" and use a
persistent file handle for reading and writing. This drastically
reduced the number of file handles being opened.
Unfortunately, it appears that some versions of Solaris lose the file
offset when performing a write after the handle has been seeked.
The simplest workaround is to seek to EOF on files opened in a+ mode
before writing to them, which is what this patch does.
Ideally, this code would exist in the vfs layer. However, this would
require creating a proxy class for file objects in order to provide a
custom implementation of write(). This would add overhead. Since
revlogs are the only files we open in a+ mode, the one-off workaround
in revlog.py should be sufficient.
This patch appears to have little to no impact on performance on my
Linux machine.
Without this, the test fails with:
$ hg -q commit -A -m 'add pushurl'
abort: file:// URLs can only refer to localhost
$ hg push
abort: file:// URLs can only refer to localhost
The variable $PWD causes check-code to complain, so avoid that.
