Summary:
Move the third-party sha1collisiondetection code from clib/sha1 to
third-party/sha1dc. This helps isolate third-party code from our own
internally developed code.
This also updates the code to use the same include paths and library names as
used by the sha1collisiondetection's upstream Makefile, which would be needed
to link against a version of sha1collisiondetection installed locally.
Test Plan:
Confirmed "make local" succeeds.
All of the tests pass, except for test-check-commit-hg, which complains about
the fact that some of this third-party code contains multiple empty lines in a
row. It doesn't seem straightforward to update test-check-commit to ignore
this third-party code, but these test failures shouldn't affect any future
commits.
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D281
Summary:
This is a first set of changes to help `cdatapack` compile on Windows. Second
set will include adding some way of using `mman` on Windows.
Test Plan:
- `make local` on Linux, `rt`
- with some intermediary solution for `mman` this also builds on Windows 10,
I was able to produce `cdatapack_get.exe` and `cdatapack_dump.exe`. Here's an
example:
```
PS C:\Code\fb-hg-rpms\fb-hgext\cdatapack> .\cdatapack_get.exe 3ba0b10b8d251743a2692e042b114c1204b19d74 88dadb363234ec4fec3df85810810d6073288350
xplat/third-party/yarn/offline-mirror/smoothscroll-polyfill-0.3.5.tgz
Node Delta Base Delta SHA1 Delta Length
88dadb363234ec4fec3df85810810d6073288350 0000000000000000000000000000000000000000 466e6039b51cb525d70e1a5077ef81e064678eae 26057
```
Reviewers: durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D106
Summary:
This patch removes all `#include "../` lines and use the shortest possible
include path.
Test Plan: `make clean build`
Reviewers: durham, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5113672
Signature: t1:5113672:1495565454:961fb6f2f57a81a95013e0b8f67b2917c2e4523e
Summary:
There are two `convert.h`. This patch unifies them and does cleanups so it's
a valid header file which could be included by multiple .c and .cpp files
and linker won't complain re-definition (by adding `static` to everything).
Besides, reformat the code so it could pass check-code. Also fix a compiler
warning about comparing an unsigned integer with a signed integer.
Test Plan: `make clean local`. It still builds.
Reviewers: durham, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5113550
Signature: t1:5113550:1495565413:e399f898ac513e64af37dab5daf55cedbabfc703
Summary:
It's a small C utility. It should belong to `clib` directory.
Also, `#include "../foo"` does not seem to be a good pattern. It makes
include files harder to follow and make code movement more difficult. Since
`clib` is already included in `-I` during compilation, remove `../`.
Test Plan: `make clean local`. It still builds.
Reviewers: ikostia, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters, vsutaria
Differential Revision: https://phabricator.intern.facebook.com/D5113236
Signature: t1:5113236:1495562436:e8d64083ab0417c67b63223a092470739f4c1176
Summary:
Previously, we used the length of the index file to determine the upper bounds
of the bisect. In a future patch we'll want to add more data to the end of the
index file, so we need to record how long the index portion of the index is.
This patch adds that information.
Test Plan: Ran the tests.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4983682
Signature: t1:4983682:1493693255:57ab9af2030847fedff05b6755113ba8ce0c933b
Summary:
The newly changed `LZ4_decompress_safe` is unhappy about empty buffer. So let's
check empty revision explicitly.
Test Plan: Added a test
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4986277
Signature: t1:4986277:1493743735:be2ad6047bb0b983425c6e49b7c5ddf2c94d1c1a
Summary:
D4960035 used `GET_DELTA_CHAIN_CORRUPT` at a place where it should be
`GET_DELTA_CHAIN_LINK_CORRUPT`. The error was not caught by gcc but clang.
Test Plan: Build on OS X
Reviewers: durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4981954
Signature: t1:4981954:1493678436:3cb4779f14e64951c4d489ca30766888e7b0386a
Summary:
This patch moves lz4decompress logic to a separate function from
getdeltachainlink.
This should solve a memory leak issue and speed up datapack entry iteration.
Practically, this means repack will be faster and take less memory.
Test Plan: `make clean local`, run `test-cstore.t`, and `test-remotefilelog-repack-fast.t`
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters, terrelln
Differential Revision: https://phabricator.intern.facebook.com/D4960035
Signature: t1:4960035:1493609520:a3c74bae92b8fff85ccadd9dd412a0c2b05573ac
Summary:
This diffs add a `getmeta` method to all content stores. The cdatapack code is
modified to pass the tests, it needs further change to support `getmeta`.
The datapack format is bumped to v1 from v0. For v1, we append a `metadata`
dict at the end of each revision. The dict is currently used to store revlog
flags and rawsize of raw revlog fulltext. In the future we can put more data
like a second hash etc, without changing API or format again.
This diff focuses on correctness. A datapack caching layer to speed up
`getmeta` will be added later.
Tests are updated since we write new v1 packfile now and the format change
leads to different content and packfile names.
`Makefile`, `ls-l.py` are added to make tests easier to maintain.
Test Plan: Updated existing tests.
Reviewers: #mercurial, rmcelroy, durham
Reviewed By: durham
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4903917
Signature: t1:4903917:1493255844:7ef5d487096cd2f78f2aaae672a68d49f33632ee
Summary:
This diff changes our code to use the new SHA1 library. See the previous diff
for why we do this.
Test Plan:
Run related tests manually:
```
$ make local PYTHON=python2
$ rt test-remotefilelog-*.t
.........................
# Ran 25 tests, 0 skipped, 0 warned, 0 failed.
$ rt test-treemanifest*.t
........
# Ran 8 tests, 0 skipped, 0 warned, 0 failed.
$ rt test-fastmanifest*.t
.........
# Ran 9 tests, 0 skipped, 0 warned, 0 failed.
```
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4945025
Signature: t1:4945025:1493154873:844e55a51ab250354fc08163e0949eed47b0a861
Summary:
I sync'd a copy of this code into the eden repository.
I had to adjust a couple of include paths to get the code to
compile correctly in the hermetic build environment that is
in use there.
In addition, our linter suite over there found a couple of C++ nits
to be fixed up.
Test Plan: make local
Reviewers: simpkins, ikostia, simonfar, durham
Reviewed By: durham
Subscribers: net-systems-diffs@fb.com, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4879285
Signature: t1:4879285:1492039044:8cb1e033e35ee568806de94dda3d2f6f8e78f5cb
Summary:
This is a s//g replacement of all the `return (type_name) {` with
`return COMPOUND_LITERAL(type_name) {`.
This is the command which produced the diff:
`egrep 'return \(\w*\) \{' -Ir . --exclude='*.py*' --exclude-dir=.hg -l | xargs sed 's/return (\(\w*\)) {/return COMPOUND_LITERAL(\1) {/g' -i `
After I've done this, I checked:
`egrep '\(\w+\) \{' -Ir . --exclude='*.py*' --exclude-dir=.hg | egrep -v '(switch|while)' | grep -v 'if (' | grep -v 'COMPOUND_LITERAL' | less`
and it looks like the only things of `(something) {` syntax are function definitions, adding space before `(` in search pattern yields no results.
This is needed to make this compile on Windows under MSVC2015.
Depends on: D4843230
Test Plan:
- run `python setup.py build -f`, see it compile
- run all the tests, see them pass
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4843240
Signature: t1:4843240:1491495690:a097bfab403805052d5ac25d1db7bb32af3bca28
Summary:
Proposed header (or its dir) is a single place to put MSVC/GCC hacks. So
far it only includes the COMPOUND_LITERAL macro which behaves differently
depending on MSVC mode.
When MSVC2015 is used in C++ mode, it does not support things like:
`(my_type) {initializers}`, but in C mode it does.
To clarify: I am not even sure whether we need to have the ability to compile in a purely C mode, but I did not want to figure out.
Test Plan: - on Linux, run `python setup.py build`, run all the tests, see them passing
Reviewers: #sourcecontrol, tja
Reviewed By: tja
Subscribers: tja, jsgf, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4843230
Signature: t1:4843230:1491496062:3fa10ae5a5aac850689991de1ca6ee1ac86d9dce
Summary:
This is an RFC kind of diff, so I am looking for comments. I am not very
familiar with either C or C++.
The goal is to make all this stuff compile on Windows under MSVC2015. MSVC
does not support variable-sized arrays in C++ out-of-the-box (not sure
whether there are tricks to make it do so), so my proposal is to use
explicit `malloc`s in place.
Test Plan:
Only tested on Linux for now. `python setup.py build` works,
tests pass, seems to make sense.
Reviewers: #sourcecontrol, jsgf
Reviewed By: jsgf
Subscribers: jsgf, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4839968
Signature: t1:4839968:1491492552:c80fac6990aaee78e6bb18522ff13e02eb9521e4
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
