We have a convention of using -c|-m|FILE elsewhere for reading from
revlogs. Use it for `hg perfrevlog`.
While I was here, I also added a docstring to document what this
command does, as "perfrevlog" is ambiguous.
When demandimport is disabled, the test will fail because extroot/foo.py uses
import outside PYTHONPATH. This is bad since we have a PyPy migration plan and
it does not support demandimport. Fix by setting PYTHONPATH.
Revlog chunks can be stored uncompressed. If the first byte of the
raw data is \0, we store the data as is. Else we prefix it with 'u'.
Before, we performed a string slice to strip out the 'u' prefix.
With this patch, we use a buffer to avoid an extra memory copy and
associated garbage collection overhead.
I was unable to verify any performance impact of this patch. For both
mozilla-central and the hg repos, the number of manifest revisions
with 'u' prefixes is very small - under 1%. So this change likely
isn't called enough to have an impact on manifest reading. However,
the reasoning behind this change is solid, so it should be safe.
This is less portable than the C version, but PyPy can't load CPython
extensions. So for now, this will be used on PyPy.
I've tested it on Linux amd64 and Mac OS X.
It will be used to attach client's stdio files to a background chg command
server.
The socket module of Python 2.x doesn't provide recvmsg(). This could be
implemented by using ctypes, but it would be less portable than the C version
because the handling of socket ancillary data heavily depends on preprocessor.
Also, some length fields are wrongly typed in the Linux kernel.
As part of investigating performance improvements to revlog reading,
I needed a mechanism to measure every part of revlog reading so I knew
where time was spent and how effective optimizations were.
This patch implements a perf command for benchmarking the various
stages of reading a single revlog revision.
When executed against a manifest revision at the end of a 30,000+
long delta chain in mozilla-central, the command demonstrates that
~80% of time is spent in zlib decompression.
We have a nice API now. Use it.
This does mean we introduce an extra index lookup for each revision.
Considering this is a debug command, the overhead should be acceptable.
We could add the chain size to revlog._deltachain(). However, that
feels like avoidable overhead.
This code is already written in multiple locations.
While this code needs to be fast and extracting it to its own function
adds overhead, code paths reading delta chains typically read,
decompress, and do binary patching on revlog data from the delta chain.
This other work (especially zlib decompression) almost certainly
accounts for a lot more time than the overhead of introducing a Python
function call. So I'm not worried about the performance impact of this
change.
The old code only partially cleared the caches. Now that we have a
comprehensive method for wiping all caches, let's call it.
This appears to introduce a marginal regression in `hg perfmanifest`
on mozilla-central. This is good because the new result is more
accurate since caches aren't being used.
clearcaches() was added several years ago in 1e47437a1ca7 as part
of implementing a perf command. Since revlog instances have many caches
and since the spirit of this mostly unused method is to facilitate
performance testing, I think it's appropriate for all the revlog's
caches to get cleared when it is called.
A fileset predicate can invoke 'matchctx.existing()' successfully,
even if it isn't marked as "existing caller". It is aborted only in
some corner cases: e.g. there were one deleted file in the working
directory (see 2c5c0790cbcc for detail).
This patch makes 'matchctx.existing()' invocation abort if not
'_existingenabled', which is true only while "existing caller"
running.
After this changes, non-"existing caller" predicate function is
aborted immediately, whenever it invokes 'matchctx.existing()'. This
prevent developer from forgetting to mark a predicate as "existing
caller".
BTW, unintentional 'matchctx.status()' invocation can be detected
easily without any additional trick like this patch, because it
returns 'None' if a predicate isn't marked as "status caller", and
referring field (e.g. '.modified') of it is always aborted.
This reduces cost of examining whether given predicate calls
'matchctx.status()' or 'matchctx.existing()' in 'getfileset()' at
runtime.
This kind of examination is used also in subsequent patch, which
detects unintentional 'matchctx.existing()' invocation per each
predicate evaluation.
Before this patch, predicates calling 'matchctx.status()' are listed
up by immediate list value in 'getfileset()'.
This prevents 3rd party extensions from adding specific predicate
calling 'matchctx.status()'.
This uses decorator to mark a predicate as "status caller".
This can also localize changes for adding (or removing) a "status
caller" predicate function in source code.
Using decorator can localize changes for adding (or removing) a
fileset predicate function in source code.
It is also useful to pick predicates up for specific purpose. For
example, subsequent patches marks predicates as "call status" or "use
existing" via decorator.
To avoid (1) redundancy between "predicate name" and (the beginning
of) help document, and (2) accidental typo of help document, this
patch also makes decorator put predicate declration into the beginning
of help.
Before this patch, predicate function 'encoding' and 'eol' aren't
listed up in '_existingcallers', even though they invoke 'existing()'.
This causes unexpected failure of these predicate, if there is a
(manually) deleted file in the working directory.
2c5c0790cbcc and 12b403664548 seem to overlook putting already
existing 'encoding' or newly introduced 'eol' into '_existingcallers'.
This patch also changes order of fileset "eol(unix)" output in test,
because "existing caller" predicates show "A(dded)" files before
"C(lean)" ones.
Nested functions in Python are not able to assign to variables in the
outer scope without something like the list trick because assignments
refer to the inner scope. So, we formerly used a list to give an
object to assign into.
Now that error and warning are object members, the [0] hack is no
longer needed.
