This patch adds support for py3k in bdiff.c. This is accomplished by including
a header file responsible for abstracting the API differences between python 2
and python 3.
Patch from Jason Orendorff
The lower the threshold, the stronger the popularity hack's
influence. So at 3999 lines, the hack is disabled; and at 4000 lines,
the hack is enabled at maximum strength (t=4).
No source file in mercurial/crew is over 4000 lines. But there are, oh,
a few such files in Mozilla. I can testify that this hack causes hg to
generate some correct but eyebrow-raising patches.
I think the hack should phase in gradually. The threshold should be high
for small files where we don't need it so much. Like this:
t = (bn < 31000) ? 1000000 / bn : bn / 1000;
That would leave the popularity hack disabled for small files, then
gradually phase it in:
bn < 1000 -- t > bn (popularity hack is completely disabled)
bn == 1000 -- t = 1000 (still effectively disabled)
bn == 2000 -- t = 500 (only hits unusual files)
bn == 10000 -- t = 100 (only hits especially common lines)
bn == 31000 -- t = 31 (hack is at maximum power)
bn == 32000 -- t = 32 (hack could backfire, ease off)
When the common part of a diff can be moved forward, move it forward.
Otherwise we don't get deterministic results (it would depends on the way we
split for the recursion).
That way we get identical hunks when doing the same change, it helps to solve
issue1295 (inconsistent diffs on different side during a merge).
- adjust the common line threshold to .1%
this speeds up a delta of 7M lines of source from 10m to 40s
- adjust the scaling of the hash array down a bit as it was raising the peak
memory usage significantly
lyhash is a very simple and fast hash function that had the fewest
hash collisions on a 3.9M line text corpus and 190k line binary corpus
and should have significantly fewer collisions than the current hash
function.
pretty easy to find after I recompiled the python interpreter and
mercurial for profiling.
In "bdiff.c" function "equatelines" allocates the minimum hash table
size, which can lead to tons of collisions. I introduced an
"overcommit" factor of 16, this is, I allocate 16 times more memory
than the minimum value. Overcommiting 128 times does not improve the
performance over the 16-times case.
bdiff.blocks() returns a dummy match at the end of both files; the
length of that chunk is never set, so it will sometimes contain random
heap garbage. There are apparently workarounds for this elsewhere:
# bdiff sometimes gives huge matches past eof, this check eats them,
Python 2.5 doesn't like it when we mix str objects and the "t#" format
in PyArg_ParseTuple. Change it to use "s#". Tested with python 2.3, 2.4
and 2.5.
manifest.add gives revlog.addrevision a buffer object, which may
be cached and used for a second call in the same session (as mq does
when pushing multiple patches). The other option would be to cast the
buffer to str when caching it.
on 5.8MB (244.000 lines) text file with similar lines, hash before
this change made diff against empty file take 75 seconds. this change
improves performance to 0.6 seconds. result is that clone of smallish
repo (137MB) with some files like this takes 1 minute instead of 10
minutes.
common case of diff is 10% slower now, probably because of worse cache
locality. but diff does not affect overall performance in common case
(less than 1% of runtime is in diff when it is working ok), so this
tradeoff looks good.
Many projects use inttypes.h, too. stdint.h isn't available everywhere, e.g.
on some versions of Solaris, while inttypes.h is available everywhere where
stdint.h is.
The compiling runs through without warning, but runnig the newly builded
hg emmits a message:
| ImportError: ld.so.1: python: fatal: relocation error:
| file /opt/local/lib/python2.3/site-packages/mercurial/bdiff.so:
| symbol cmp: referenced symbol not found
Removing the inline infront of cmp corrects this error message.
Temporary fix to allow Mercurial to build on HP-UX 11, as the C
compiler on HP-UX 11 doesn't support 'inline' qualifier. The
'__inline' qualifier seemed to be supported, but not without
first resolving other associated issues.
I ran into a bug while importing a large repository into mercurial.
The diff algorithm does not allocate a big enough array of hunks
for some test cases. This results in memory corruption, and possibly,
as in my case, a seg fault.
You should be able to reproduce this problem with any case of more
than a few lines that follows this pattern:
a b
= =
1 1
2
2 3
4
3 5
.
4 .
.
5
.
.
.
I.e., "a" has blank lines on every other line that have been removed in
"b". In this case, the number of matching hunks is equal to the number
of lines in "b". This is more than ((an + bn)/4 + 2). I'm not sure what
motivates this formula, but when I changed it to the smaller of an or
bn (+ 1), it works.
[comment added by mpm]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
[PATCH] use <arpa/inet.h> instead of <netinet/in.h> for ntohl/htonl
From: Jed Davis <jdev@panix.com>
This fixes the Mac OS X build problem; hopefully it won't break any
other OSes, especially since SUSv3 says arpa/inet is the right header.
( http://www.opengroup.org/onlinepubs/009695399/functions/ntohl.html )
manifest hash: 2f06ff0cffefdb35e794131afcd1f34f9fdfa5cf
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
iD8DBQFCyEoFywK+sNU5EO8RAk6WAJ9v/pnr07zUXKM9EBQQGaKSZAlhxACdHrwS
XTLSL6pPGAwaRfExGF2A3DQ=
=Rtv9
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
[PATCH] bdiff/mpatch under MSVC
From: K Thananchayan <thananck@yahoo.com>
MSVC (6.0) environment does not have 'stdint.h' and does not provide
`inline' qualifier. The following patch is needed to make mecurial
installable under MSVC.
manifest hash: a5b64235acced16cb451faa698922559fec4e573
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
iD8DBQFCxPapywK+sNU5EO8RAmRnAKCt9cOASaIsYB6kNUDSIStR1DmY4gCgnXlL
Jf0nMmGEkoyXtB0eV+fLzJU=
=fKD5
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Remove stdint.h from mpatch and bdiff
It's only there for ntohl and htonl and should be pulled in by in.h.
manifest hash: 65954290279241ac92c9ce04c21cf1a3c9dd54e0
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
iD8DBQFCwD8KywK+sNU5EO8RAhv2AJ40R/T72XK63IbeEFqMLSRJbRJWdACcDa9r
dOL9XpyYxR09REbAHw0JrlE=
=8wkZ
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Minor speed improvements for bdiff
Consolidate the jpos/jlen arrays to improve cache locality.
Do the same for the hash head/length arrays.
manifest hash: e6d9ed36782741b1d6fcce8c2d00155a9540e81d
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
iD8DBQFCvzWxywK+sNU5EO8RAlTMAJ9+yl0dKIeWv4RegeLy7g6wcnoYwgCgk6la
ip6KEAyBb7ktsX14KyZ5+/s=
=utNJ
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
extensions: use stdint.h
Not sure why I didn't do this the first time around. Hopefully still
builds everywhere.
manifest hash: 965582286a190728f8cc0dfb8e11ee56628a59a5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)
iD8DBQFCvfRgywK+sNU5EO8RAg9SAJ4/ZVpQZcDY5xovLDTZK2txEegEgwCdF2b+
lzSIP109qq8D+KIdUWsbEPc=
=+0Yy
-----END PGP SIGNATURE-----
