This extension is a thin wrapper around the native `changelog.index` object
that allows us to incrementally replace its methods. Since some index
methods (ex. `nodemap.__getitem__`) are called very frequently, Cython
features are used intentionally to avoid overhead. This also makes it easier
to integrate logic with C interface in the future.
As a side effect, this patch enforce `index` to be conceptually separate
from `nodemap`. So `changelog.index[node]` could be made illegal in the
future, which seems to be a good thing.
Test Plan:
Run `hg sl` with and without the extension in a large repo. Check traceprof
outputs. Notice the performance difference around index methods are roughly
10%, which seems acceptable:
Without the extension:
25 \ node (4823 times) changelog.py:361
18 | node (4931 times) revlog.py:631
With the extension:
27 \ node (4823 times) changelog.py:361
19 | node (4931 times) revlog.py:631
Also run `rt --extra-config-opt=extensions.clindex=` from core hg and make
sure changes are all caused by having an extra extension enabled.
Differential Revision: https://phab.mercurial-scm.org/D1353
urlgrabber is not available on Windows machines; urllib3 is MIT licensed so can
safely be bundled.
Test Plan:
Run the tests, run hg ssl against a repository with valid arcanist config.
Differential Revision: https://phab.mercurial-scm.org/D1026
We no longer use this experiment and the upstream refactor breaks it. Let's remove it.
Test Plan:
all sqldirstate tests pass now
Differential Revision: https://phab.mercurial-scm.org/D944
Summary:
Now uniondatapackstore can also hold python data stores. PythonDataStore
wrapper simply passes function calls to underlying python objects and marshals
the output.
Test Plan:
* Added test case
* Tested on fbsource with treemanifest.usecunionstore=True
Reviewers: durham, simonfar, ryanmce, #fbhgext
Reviewed By: durham, simonfar, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D631
Summary:
Extended DeltaChain with CDeltaChain and PyDeltaChain which are wrappers around
c and python delta chains respectively. The declaration and implementation
of c and python delta chains as well as DeltaChainLink were put in a different
file.
Test Plan: * Ensure that unit tests pass
Reviewers: ryanmce, durham, simonfar, #fbhgext
Reviewed By: ryanmce, durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D630
Summary:
Define the ManifestPtr class in its own header file, to eliminate some of the
nasty circular dependencies between manifest.h, manifest_entry.h, and
manifest_fetcher.h
This makes the include process for these files much simpler, and prevents
manifest.h and manifest_entry.h from both trying to include each other at
different locations in their files.
This corresponds to Facebook diff D5588672.
Test Plan: Confirmed "make local" succeeds and the unit tests pass.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D506
Summary:
Convert the C and C++ files in cdatapack, clib, cstore, and ctreemanifest
to always include files from the root of fb-hgext. This simplifies the build
process by no long requiring a variety of separate include directories to be
specified on the compiler command line.
This will also make it easier to re-use these extensions in other projects
with different build systems.
This corresponds to the Facebook diff D5588676.
Test Plan: Confirmed that "make local" succeeds from a clean build.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D505
Summary:
Add a #define which allows selecting between the sha1collisiondetection and
openssl SHA-1 implementations at build time.
For now setup.py always uses sha1collisiondetection, but we have some other
internal build environments where we will likely default to OpenSSL's SHA-1
implementation instead.
Test Plan: Confirmed the code builds and passes tests.
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D285
Summary:
Add a clib/sha1.h file with SHA-1 utility functions that hide the details of
the underlying SHA-1 implementation being used. This will make it easier in
the future if we want to use the faster SHA-1 implementation from OpenSSL if it
is available, but fall back to the sha1collisiondetection library if it is not
available.
Test Plan: Confirmed the code builds and passes unit tests.
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D282
Summary:
Move the third-party sha1collisiondetection code from clib/sha1 to
third-party/sha1dc. This helps isolate third-party code from our own
internally developed code.
This also updates the code to use the same include paths and library names as
used by the sha1collisiondetection's upstream Makefile, which would be needed
to link against a version of sha1collisiondetection installed locally.
Test Plan:
Confirmed "make local" succeeds.
All of the tests pass, except for test-check-commit-hg, which complains about
the fact that some of this third-party code contains multiple empty lines in a
row. It doesn't seem straightforward to update test-check-commit to ignore
this third-party code, but these test failures shouldn't affect any future
commits.
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D281
Summary:
This removes Cython as a build dependency as requested by our users.
Added a test to prevent check-in files with `/home/`.
Modified check-code to skip checking Cython generated files.
Test Plan:
Run `make clean local` with Cython installed and uninstalled.
Also run `make clean local` with `USECYTHON=1` and `0`.
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D5252935
Signature: t1:5252935:1497525190:402798077d44e39e24fcb037535ec7ffd1af9c4b
Summary:
Previously, the tracer extracts file name, function name from Python code
objects and stores them as std::string. And frame the de-duplication logic
hashes those strings, which is actually quite expensive.
`hg id` takes 1.0 seconds without frame de-duplication, and 2.5 seconds with
de-duplication.
This diff assumes Python code object is unique (i.e. Python won't generate two
code objects for a single code segment). That seems reasonable and `lsprof`
seems to make a same assumption.
With that assumption, just store `PyCodeObject` (contains file, method name and
line number) instead of `std::string` (for file, method names) and line numbers
for code identity and use the raw address of `PyCodeObject` to do frame
de-duplication.
That is helpful during profiling and de-duplication:
- During profiling, no need to copy strings (convert to std::string),
therefore less overhead
- During frame de-duplication, no need to read long strings and hash them,
therefore much faster
Debug flag during compilation is added to make debugging easier as the code
starts to mix of C++ and Python objects, which means mistakes could be made
more easily.
Test Plan:
With this patch, `hg id` with frame de-duplication takes 1.0 seconds.
There is no visible overhead doing de-duplication. Cheers!
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5112636
Signature: t1:5112636:1495562232:dd0724198ebb7f2eb4e9fd3a83517fc9160fa01e
Summary:
statprof is now part of core hg. And core hg will import its own version
instead of the external one. So it's no longer to have a separate version.
I did a comparison. The version in fb-hgext is falling behind - it lacks of
Chrome support. The upstream version is also cleaner in terms of Python3 code
standard and error handling.
Test Plan: eyes
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5112666
Signature: t1:5112666:1495562274:7e1ca7d964bf617695dbefd59c3843a3ebf47f9e
Summary:
This diff disables or rewrites problematic parts in linelog code on Windows -
`mmap`, `ftruncate`, `sysconf`, `htonl` so the linelog module could be built
without on-disk linelog support. That means `absorb` can work. `fastannotate`
needs extra effort.
Test Plan:
`python setup.py install` and check in Python shell:
```
>>> import linelog
>>> linelog.linelog()
<open linelog (in-memory) at 0x3ab07bb5c8>
```
Also check the test pass:
```
fb-hgext\linelog\pyext>%PYTHONHOME%\python test-random-edits.py
```
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5093192
Signature: t1:5093192:1495161776:06e05544cff808b03546f753a553ecc66fafc79f
Summary:
This was reported on IRC. People building absorb got:
lz4.h: No such file or directory
Test Plan:
`make clean`, and try `setup.py build` with:
- no parameters
- `--component absorb`, make sure lz4 is not needed
- `--component remotefilelog`, make sure it builds
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5014663
Signature: t1:5014663:1494023360:31ededc49de2dda45dfea4f62d6759cb19d819ef
Summary:
I noticed that cdatapack.o was not cleaned with `make clean`. And `.c`
generated by Cython should also be cleaned to avoid surprise.
Since the latter cannot be done cleanly using short shell script, write the
clean logic in Python.
Test Plan: `make clean`
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4979676
Signature: t1:4979676:1493717290:a826da9e50bcd67362c790b27ff0d88714d71d49
Summary:
It's reported Cython 0.20.1post0 version shipped by Ubuntu works.
However, I cannot build linelog using Cython at git commit 0e6e38ec8,
which is supposed to be the version used by that package.
That said, let's remove the explicit version check but run a real build to
detect whether Cython works or not.
[1]: https://bitbucket.org/facebook/hg-experimental/pull-requests/21/build-only-requiring-cython-020-not-022
Test Plan: Run `make local`.
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4972024
Signature: t1:4972024:1493407514:c5e0d21f92e866955b40600d1a1144f780e90c6e
Summary:
This diff changes our code to use the new SHA1 library. See the previous diff
for why we do this.
Test Plan:
Run related tests manually:
```
$ make local PYTHON=python2
$ rt test-remotefilelog-*.t
.........................
# Ran 25 tests, 0 skipped, 0 warned, 0 failed.
$ rt test-treemanifest*.t
........
# Ran 8 tests, 0 skipped, 0 warned, 0 failed.
$ rt test-fastmanifest*.t
.........
# Ran 9 tests, 0 skipped, 0 warned, 0 failed.
```
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4945025
Signature: t1:4945025:1493154873:844e55a51ab250354fc08163e0949eed47b0a861
Summary:
I sync'd a copy of this code into the eden repository.
I had to adjust a couple of include paths to get the code to
compile correctly in the hermetic build environment that is
in use there.
In addition, our linter suite over there found a couple of C++ nits
to be fixed up.
Test Plan: make local
Reviewers: simpkins, ikostia, simonfar, durham
Reviewed By: durham
Subscribers: net-systems-diffs@fb.com, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4879285
Signature: t1:4879285:1492039044:8cb1e033e35ee568806de94dda3d2f6f8e78f5cb
Summary:
The portability/ subdirectory always needs to be listed in the compiler include
paths in order to successfully build. This fixes setup.py to always add it to
the list of include directories, rather than only adding it when INCLUDE_DIRS
was not specified through the environment.
Test Plan:
Successfully built the fb-hgext repository when specifying alternate system
header paths through the INCLUDE_DIRS environment variable.
Reviewers: tja, wez, ikostia
Reviewed By: ikostia
Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4865296
Signature: t1:4865296:1491903269:fc07edf01a89117e25805e9ee936b2e2eee22410
Summary:
Proposed header (or its dir) is a single place to put MSVC/GCC hacks. So
far it only includes the COMPOUND_LITERAL macro which behaves differently
depending on MSVC mode.
When MSVC2015 is used in C++ mode, it does not support things like:
`(my_type) {initializers}`, but in C mode it does.
To clarify: I am not even sure whether we need to have the ability to compile in a purely C mode, but I did not want to figure out.
Test Plan: - on Linux, run `python setup.py build`, run all the tests, see them passing
Reviewers: #sourcecontrol, tja
Reviewed By: tja
Subscribers: tja, jsgf, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4843230
Signature: t1:4843230:1491496062:3fa10ae5a5aac850689991de1ca6ee1ac86d9dce
Summary:
While this does not change any real behavior, it makes `python setup.py`
output a bit cleaner as it does not pass irrelevant options to `cl.exe`
and command lines are shorter.
Test Plan: - build on Linux, make sure it still builds
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4834980
Signature: t1:4834980:1491427834:3f5be0644b02ae488946366ff733d0bf1ffdce71
78d6c7763d48 changed how component is tested, but forgot to change the
default component. This patch fixes it so setup.py running without
"--component" will work as expected.
Summary:
The ideal interactive mode couldn't be implemented trivially. Move `absorb`
to a directory so we can add related, but decoupled components as separate
files.
Test Plan: `make local`
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4813909
Signature: t1:4813909:1491211561:e9c40b1242c9b74230c0b8937723a2d4548e22c3
Summary:
I accidently disabled fastmanifest and a bunch of other packages on non-win
platforms, instead of Windows itself. This should fix it.
Test Plan: - makes sense visually
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4761873
Signature: t1:4761873:1490267573:74e81c97652f15b06b7b7a56f96bf3d35385f54e
Summary:
We have encountered a kernel issue where `rmdir` a non-empty directory may race
with other things and hang in kernel for a long time.
This patch changes `os.rmdir` to avoid `rmdir` non-empty directories. It is
written in Cython calling the low-level `readdir` libc friends to make overhead
minimal.
Test Plan: Added a new test
Reviewers: #sourcecontrol, clm, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, simpkins, osandov, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4716711
Tasks: 16647532
Signature: t1:4716711:1489627923:7c7432748c1fd8c070ce257bd172feebd3807f65
Summary:
Now that ctreemanifest no longer depends on python.h, let's move pythonutil over
to cstore where all the python code is.
Test Plan: Ran the build and the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663988
Signature: t1:4663988:1488895919:652b3fc35a2dd12c51a9f70e32997c7b4d037c95
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
Linking native python extensions together is a huge pain and I'm encountering
bugs in the linker in centos6. To simplify everything, I'm going to merge all
the cstore related libraries into a single library.
Step one is to move the C libdatapack code to be a statically linked library.
distutils doesn't actually support using -l or extra args in it's static library
generator, so we need to monkey patch a fix for that here.
Test Plan:
Built using 'python setup.py build --component cdatapack'. Verified the static library was built:
```
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Iclib -I/usr/local/include -I/opt/local/include -I/opt/homebrew/include/ -c cdatapack/cdatapack.c -o build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o -std=c99 -Wall -Werror -Werror=strict-prototypes
ar rc build/temp.linux-x86_64-2.7/libdatapack.a build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o
```
And the resulting cdatapack.o library did not reference a libdatapack shared library any more by running 'ldd cdatapack.so'. On OSX verified the same thing using "otool -L cdatapack.so"
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581274
Signature: t1:4581274:1487842180:d2125ebec7bbb9533e02b0589f188251881ed569
Summary:
As part of a refactoring to move all the store code together, we need to break
the cdatapack extension into two parts, the actual c structure and the python
extension. In a future patch the python extension part will be moved into the
cstore with everything else, but the c structure needs to remain separate since
it's C code and not C++.
Test Plan: Ran the tests
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581268
Signature: t1:4581268:1487669407:9d97fcdfd04689c95494a76ead515043a71a7f38
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10