We just need a hash table {fctx.data(): fctx} which doesn't keep fctx.data()
in memory. Let's simply use hash(fctx.data()) to put data out from memory,
and manage collided fctx objects by list.
This isn't significantly faster than using sha1, but is more correct as we
know SHA-1 collision attack is getting practical.
Benchmark with 50k added/removed files, on tmpfs:
$ hg addremove --dry-run --time -q
previous: real 12.420 secs (user 11.120+0.000 sys 1.280+0.000)
this patch: real 12.350 secs (user 11.210+0.000 sys 1.140+0.000)
Instead, build a set of files to be removed and recreate addedfiles
only if necessary.
Benchmark with 50k added/removed files, on tmpfs:
$ hg addremove --dry-run --time -q
original: real 16.550 secs (user 15.000+0.000 sys 1.540+0.000)
previous: real 16.730 secs (user 15.280+0.000 sys 1.440+0.000)
this patch: real 16.070 secs (user 14.470+0.000 sys 1.580+0.000)
Perhaps the original implementation would want to sort added/removed files
alphabetically, but actually it did sort fctx objects by memory location.
This patch removes the use of set()s in order to preserve the order of
added/removed files. addedfiles.remove() becomes quadratic, but its cost
appears not dominant. Anyway, the quadratic behavior will be eliminated by
the next patch.
Benchmark with 50k added/removed files, on tmpfs:
$ mkdir src
$ for n in `seq 0 49`; do
> mkdir `printf src/%02d $n`
> done
$ for n in `seq 0 49999`; do
> f=`printf src/%02d/%05d $(($n/1000)) $n`
> dd if=/dev/urandom of=$f bs=8k count=1 status=none
> done
$ hg ci -qAm 'add 50k files of random content'
$ mv src dest
$ hg addremove --dry-run --time -q
original: real 16.550 secs (user 15.000+0.000 sys 1.540+0.000)
this patch: real 16.730 secs (user 15.280+0.000 sys 1.440+0.000)
Before this patch, similarity detection logic (for addremove and
automv) depends entirely on SHA-1 digesting. But this causes incorrect
rename detection, if:
- removing file A and adding file B occur at same committing, and
- SHA-1 hash values of file A and B are same
This may prevent security experts from managing sample files for
SHAttered issue in Mercurial repository, for example.
https://security.googleblog.com/2017/02/announcing-first-sha1-collision.htmlhttps://shattered.it/
Hash collision itself isn't so serious for core repository
functionality of Mercurial, described by mpm as below, though.
https://www.mercurial-scm.org/wiki/mpm/SHA1
This patch compares between actual file contents after hash comparison
for exact identity.
Even after this patch, SHA-1 is still used, because it is reasonable
enough to quickly detect existence of "(almost) same" file.
- replacing SHA-1 causes decreasing performance, and
- replacement of it has ambiguity, yet
Getting content of removed file (= rfctx.data()) at each exact
comparison should be cheap enough, even though getting content of
added one costs much.
======= ============== =====================
file fctx data() reads from
======= ============== =====================
removed filectx in-memory revlog data
added workingfilectx storage
======= ============== =====================
All versions of Python we support or hope to support make the hash
functions available in the same way under the same name, so we may as
well drop the util forwards.
We speed up 'findrenames' for the usecase when a user specifies they
want a similarity of 100% by matching files by their exact SHA1 hash
value. This reduces the number of comparisons required to find exact
matches from O(n^2) to O(n).
While it would be nice if we could just use mercurial's pre-calculated
SHA1 hash for existing files, this hash includes the file's ancestor
information making it unsuitable for our purposes. Instead, we calculate
the hash of old content from scratch.
The following benchmarks were taken on the current head of crew:
addremove 100% similarity:
rm -rf *; hg up -C; mv tests tests.new
hg --time addremove -s100 --dry-run
before: real 176.350 secs (user 128.890+0.000 sys 47.430+0.000)
after: real 2.130 secs (user 1.890+0.000 sys 0.240+0.000)
addremove 75% similarity:
rm -rf *; hg up -C; mv tests tests.new; \
for i in tests.new/*; do echo x >> $i; done
hg --time addremove -s75 --dry-run
before: real 264.560 secs (user 215.130+0.000 sys 49.410+0.000)
after: real 218.710 secs (user 172.790+0.000 sys 45.870+0.000)
The next few patches will increase the size of the "findrenames"
functionality. This patch simply moves the function into its own
file to avoid clutter building up in 'cmdutil.py'.
