Eliminates
mercurial/bdiff.c(383) : warning C4244: 'function' : conversion from '__int64'
to 'uint32_t', possible loss of data
mercurial/bdiff.c(384) : warning C4244: 'function' : conversion from '__int64'
to 'uint32_t', possible loss of data
mercurial/bdiff.c(385) : warning C4244: 'function' : conversion from
'Py_ssize_t' to 'uint32_t', possible loss of data
when compiling for Windows x64 target using the Microsoft compiler.
Reduces the conversion warnings
mercurial/bdiff.c(61) : warning C4244: '=' : conversion from '__int64' to
'int', possible loss of data
mercurial/bdiff.c(307) : warning C4244: 'function' : conversion from
'Py_ssize_t' to 'int', possible loss of data
mercurial/bdiff.c(308) : warning C4244: 'function' : conversion from
'Py_ssize_t' to 'int', possible loss of data
mercurial/bdiff.c(362) : warning C4244: '+=' : conversion from '__int64' to
'int', possible loss of data
mercurial/bdiff.c(380) : warning C4244: '=' : conversion from '__int64' to
'int', possible loss of data
mercurial/bdiff.c(381) : warning C4244: 'function' : conversion from '__int64'
to 'uint32_t', possible loss of data
mercurial/bdiff.c(382) : warning C4244: 'function' : conversion from '__int64'
to 'uint32_t', possible loss of data
mercurial/bdiff.c(416) : warning C4244: '=' : conversion from 'Py_ssize_t' to
'int', possible loss of data
to
mercurial/bdiff.c(383) : warning C4244: 'function' : conversion from '__int64'
to 'uint32_t', possible loss of data
mercurial/bdiff.c(384) : warning C4244: 'function' : conversion from '__int64'
to 'uint32_t', possible loss of data
mercurial/bdiff.c(385) : warning C4244: 'function' : conversion from
'Py_ssize_t' to 'uint32_t', possible loss of data
on the three putbe32() calls in the function bdiff
when compiling for Windows x64 target using the Microsoft compiler.
This means that threaded webservers will have more of a chance of
doing something useful while the C extension is busy computing a
delta. Not doing this was causing problems for Google Code with a 25
meg text file that takes O(7 minutes) to deltify.
If fixws() is called on a zero-length string, malloc(0) is called and
expected to return a pointer. Which it does on e.g. Linux. AIX returns
NULL, which it is also legal, but the malloc() is then assumed to have
failed. So ensure a valid pointer is always returned.
This patch adds support for py3k in bdiff.c. This is accomplished by including
a header file responsible for abstracting the API differences between python 2
and python 3.
Patch from Jason Orendorff
The lower the threshold, the stronger the popularity hack's
influence. So at 3999 lines, the hack is disabled; and at 4000 lines,
the hack is enabled at maximum strength (t=4).
No source file in mercurial/crew is over 4000 lines. But there are, oh,
a few such files in Mozilla. I can testify that this hack causes hg to
generate some correct but eyebrow-raising patches.
I think the hack should phase in gradually. The threshold should be high
for small files where we don't need it so much. Like this:
t = (bn < 31000) ? 1000000 / bn : bn / 1000;
That would leave the popularity hack disabled for small files, then
gradually phase it in:
bn < 1000 -- t > bn (popularity hack is completely disabled)
bn == 1000 -- t = 1000 (still effectively disabled)
bn == 2000 -- t = 500 (only hits unusual files)
bn == 10000 -- t = 100 (only hits especially common lines)
bn == 31000 -- t = 31 (hack is at maximum power)
bn == 32000 -- t = 32 (hack could backfire, ease off)
When the common part of a diff can be moved forward, move it forward.
Otherwise we don't get deterministic results (it would depends on the way we
split for the recursion).
That way we get identical hunks when doing the same change, it helps to solve
issue1295 (inconsistent diffs on different side during a merge).
- adjust the common line threshold to .1%
this speeds up a delta of 7M lines of source from 10m to 40s
- adjust the scaling of the hash array down a bit as it was raising the peak
memory usage significantly
lyhash is a very simple and fast hash function that had the fewest
hash collisions on a 3.9M line text corpus and 190k line binary corpus
and should have significantly fewer collisions than the current hash
function.
pretty easy to find after I recompiled the python interpreter and
mercurial for profiling.
In "bdiff.c" function "equatelines" allocates the minimum hash table
size, which can lead to tons of collisions. I introduced an
"overcommit" factor of 16, this is, I allocate 16 times more memory
than the minimum value. Overcommiting 128 times does not improve the
performance over the 16-times case.
bdiff.blocks() returns a dummy match at the end of both files; the
length of that chunk is never set, so it will sometimes contain random
heap garbage. There are apparently workarounds for this elsewhere:
# bdiff sometimes gives huge matches past eof, this check eats them,
Python 2.5 doesn't like it when we mix str objects and the "t#" format
in PyArg_ParseTuple. Change it to use "s#". Tested with python 2.3, 2.4
and 2.5.
manifest.add gives revlog.addrevision a buffer object, which may
be cached and used for a second call in the same session (as mq does
when pushing multiple patches). The other option would be to cast the
buffer to str when caching it.
on 5.8MB (244.000 lines) text file with similar lines, hash before
this change made diff against empty file take 75 seconds. this change
improves performance to 0.6 seconds. result is that clone of smallish
repo (137MB) with some files like this takes 1 minute instead of 10
minutes.
common case of diff is 10% slower now, probably because of worse cache
locality. but diff does not affect overall performance in common case
(less than 1% of runtime is in diff when it is working ok), so this
tradeoff looks good.
Many projects use inttypes.h, too. stdint.h isn't available everywhere, e.g.
on some versions of Solaris, while inttypes.h is available everywhere where
stdint.h is.
The compiling runs through without warning, but runnig the newly builded
hg emmits a message:
| ImportError: ld.so.1: python: fatal: relocation error:
| file /opt/local/lib/python2.3/site-packages/mercurial/bdiff.so:
| symbol cmp: referenced symbol not found
Removing the inline infront of cmp corrects this error message.
