Summary:
This removes Cython as a build dependency as requested by our users.
Added a test to prevent check-in files with `/home/`.
Modified check-code to skip checking Cython generated files.
Test Plan:
Run `make clean local` with Cython installed and uninstalled.
Also run `make clean local` with `USECYTHON=1` and `0`.
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D5252935
Signature: t1:5252935:1497525190:402798077d44e39e24fcb037535ec7ffd1af9c4b
Summary:
Previously, the tracer extracts file name, function name from Python code
objects and stores them as std::string. And frame the de-duplication logic
hashes those strings, which is actually quite expensive.
`hg id` takes 1.0 seconds without frame de-duplication, and 2.5 seconds with
de-duplication.
This diff assumes Python code object is unique (i.e. Python won't generate two
code objects for a single code segment). That seems reasonable and `lsprof`
seems to make a same assumption.
With that assumption, just store `PyCodeObject` (contains file, method name and
line number) instead of `std::string` (for file, method names) and line numbers
for code identity and use the raw address of `PyCodeObject` to do frame
de-duplication.
That is helpful during profiling and de-duplication:
- During profiling, no need to copy strings (convert to std::string),
therefore less overhead
- During frame de-duplication, no need to read long strings and hash them,
therefore much faster
Debug flag during compilation is added to make debugging easier as the code
starts to mix of C++ and Python objects, which means mistakes could be made
more easily.
Test Plan:
With this patch, `hg id` with frame de-duplication takes 1.0 seconds.
There is no visible overhead doing de-duplication. Cheers!
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5112636
Signature: t1:5112636:1495562232:dd0724198ebb7f2eb4e9fd3a83517fc9160fa01e
Summary:
statprof is now part of core hg. And core hg will import its own version
instead of the external one. So it's no longer to have a separate version.
I did a comparison. The version in fb-hgext is falling behind - it lacks of
Chrome support. The upstream version is also cleaner in terms of Python3 code
standard and error handling.
Test Plan: eyes
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5112666
Signature: t1:5112666:1495562274:7e1ca7d964bf617695dbefd59c3843a3ebf47f9e
Summary:
This diff disables or rewrites problematic parts in linelog code on Windows -
`mmap`, `ftruncate`, `sysconf`, `htonl` so the linelog module could be built
without on-disk linelog support. That means `absorb` can work. `fastannotate`
needs extra effort.
Test Plan:
`python setup.py install` and check in Python shell:
```
>>> import linelog
>>> linelog.linelog()
<open linelog (in-memory) at 0x3ab07bb5c8>
```
Also check the test pass:
```
fb-hgext\linelog\pyext>%PYTHONHOME%\python test-random-edits.py
```
Reviewers: #mercurial, ikostia
Reviewed By: ikostia
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5093192
Signature: t1:5093192:1495161776:06e05544cff808b03546f753a553ecc66fafc79f
Summary:
This was reported on IRC. People building absorb got:
lz4.h: No such file or directory
Test Plan:
`make clean`, and try `setup.py build` with:
- no parameters
- `--component absorb`, make sure lz4 is not needed
- `--component remotefilelog`, make sure it builds
Reviewers: #mercurial, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5014663
Signature: t1:5014663:1494023360:31ededc49de2dda45dfea4f62d6759cb19d819ef
Summary:
I noticed that cdatapack.o was not cleaned with `make clean`. And `.c`
generated by Cython should also be cleaned to avoid surprise.
Since the latter cannot be done cleanly using short shell script, write the
clean logic in Python.
Test Plan: `make clean`
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4979676
Signature: t1:4979676:1493717290:a826da9e50bcd67362c790b27ff0d88714d71d49
Summary:
It's reported Cython 0.20.1post0 version shipped by Ubuntu works.
However, I cannot build linelog using Cython at git commit 0e6e38ec8,
which is supposed to be the version used by that package.
That said, let's remove the explicit version check but run a real build to
detect whether Cython works or not.
[1]: https://bitbucket.org/facebook/hg-experimental/pull-requests/21/build-only-requiring-cython-020-not-022
Test Plan: Run `make local`.
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4972024
Signature: t1:4972024:1493407514:c5e0d21f92e866955b40600d1a1144f780e90c6e
Summary:
This diff changes our code to use the new SHA1 library. See the previous diff
for why we do this.
Test Plan:
Run related tests manually:
```
$ make local PYTHON=python2
$ rt test-remotefilelog-*.t
.........................
# Ran 25 tests, 0 skipped, 0 warned, 0 failed.
$ rt test-treemanifest*.t
........
# Ran 8 tests, 0 skipped, 0 warned, 0 failed.
$ rt test-fastmanifest*.t
.........
# Ran 9 tests, 0 skipped, 0 warned, 0 failed.
```
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4945025
Signature: t1:4945025:1493154873:844e55a51ab250354fc08163e0949eed47b0a861
Summary:
I sync'd a copy of this code into the eden repository.
I had to adjust a couple of include paths to get the code to
compile correctly in the hermetic build environment that is
in use there.
In addition, our linter suite over there found a couple of C++ nits
to be fixed up.
Test Plan: make local
Reviewers: simpkins, ikostia, simonfar, durham
Reviewed By: durham
Subscribers: net-systems-diffs@fb.com, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4879285
Signature: t1:4879285:1492039044:8cb1e033e35ee568806de94dda3d2f6f8e78f5cb
Summary:
The portability/ subdirectory always needs to be listed in the compiler include
paths in order to successfully build. This fixes setup.py to always add it to
the list of include directories, rather than only adding it when INCLUDE_DIRS
was not specified through the environment.
Test Plan:
Successfully built the fb-hgext repository when specifying alternate system
header paths through the INCLUDE_DIRS environment variable.
Reviewers: tja, wez, ikostia
Reviewed By: ikostia
Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4865296
Signature: t1:4865296:1491903269:fc07edf01a89117e25805e9ee936b2e2eee22410
Summary:
Proposed header (or its dir) is a single place to put MSVC/GCC hacks. So
far it only includes the COMPOUND_LITERAL macro which behaves differently
depending on MSVC mode.
When MSVC2015 is used in C++ mode, it does not support things like:
`(my_type) {initializers}`, but in C mode it does.
To clarify: I am not even sure whether we need to have the ability to compile in a purely C mode, but I did not want to figure out.
Test Plan: - on Linux, run `python setup.py build`, run all the tests, see them passing
Reviewers: #sourcecontrol, tja
Reviewed By: tja
Subscribers: tja, jsgf, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4843230
Signature: t1:4843230:1491496062:3fa10ae5a5aac850689991de1ca6ee1ac86d9dce
Summary:
While this does not change any real behavior, it makes `python setup.py`
output a bit cleaner as it does not pass irrelevant options to `cl.exe`
and command lines are shorter.
Test Plan: - build on Linux, make sure it still builds
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4834980
Signature: t1:4834980:1491427834:3f5be0644b02ae488946366ff733d0bf1ffdce71
78d6c7763d48 changed how component is tested, but forgot to change the
default component. This patch fixes it so setup.py running without
"--component" will work as expected.
Summary:
The ideal interactive mode couldn't be implemented trivially. Move `absorb`
to a directory so we can add related, but decoupled components as separate
files.
Test Plan: `make local`
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4813909
Signature: t1:4813909:1491211561:e9c40b1242c9b74230c0b8937723a2d4548e22c3
Summary:
I accidently disabled fastmanifest and a bunch of other packages on non-win
platforms, instead of Windows itself. This should fix it.
Test Plan: - makes sense visually
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4761873
Signature: t1:4761873:1490267573:74e81c97652f15b06b7b7a56f96bf3d35385f54e
Summary:
We have encountered a kernel issue where `rmdir` a non-empty directory may race
with other things and hang in kernel for a long time.
This patch changes `os.rmdir` to avoid `rmdir` non-empty directories. It is
written in Cython calling the low-level `readdir` libc friends to make overhead
minimal.
Test Plan: Added a new test
Reviewers: #sourcecontrol, clm, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, simpkins, osandov, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4716711
Tasks: 16647532
Signature: t1:4716711:1489627923:7c7432748c1fd8c070ce257bd172feebd3807f65
Summary:
Now that ctreemanifest no longer depends on python.h, let's move pythonutil over
to cstore where all the python code is.
Test Plan: Ran the build and the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663988
Signature: t1:4663988:1488895919:652b3fc35a2dd12c51a9f70e32997c7b4d037c95
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
Linking native python extensions together is a huge pain and I'm encountering
bugs in the linker in centos6. To simplify everything, I'm going to merge all
the cstore related libraries into a single library.
Step one is to move the C libdatapack code to be a statically linked library.
distutils doesn't actually support using -l or extra args in it's static library
generator, so we need to monkey patch a fix for that here.
Test Plan:
Built using 'python setup.py build --component cdatapack'. Verified the static library was built:
```
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Iclib -I/usr/local/include -I/opt/local/include -I/opt/homebrew/include/ -c cdatapack/cdatapack.c -o build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o -std=c99 -Wall -Werror -Werror=strict-prototypes
ar rc build/temp.linux-x86_64-2.7/libdatapack.a build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o
```
And the resulting cdatapack.o library did not reference a libdatapack shared library any more by running 'ldd cdatapack.so'. On OSX verified the same thing using "otool -L cdatapack.so"
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581274
Signature: t1:4581274:1487842180:d2125ebec7bbb9533e02b0589f188251881ed569
Summary:
As part of a refactoring to move all the store code together, we need to break
the cdatapack extension into two parts, the actual c structure and the python
extension. In a future patch the python extension part will be moved into the
cstore with everything else, but the c structure needs to remain separate since
it's C code and not C++.
Test Plan: Ran the tests
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581268
Signature: t1:4581268:1487669407:9d97fcdfd04689c95494a76ead515043a71a7f38
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary: The situation here is that our FB C extensions just won't compile on MSVC at the moment. This simply strips them out of consideration for building on Windows until they can be compiled on Windows later.
Test Plan: Built fb.hg, it didn't include these c extensions
Reviewers: #idi, durham, #sourcecontrol, rmcelroy, davidsp
Reviewed By: rmcelroy, davidsp
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4270189
Signature: t1:4270189:1481612143:9c035b4c6eb4af6b542795a64b24dc274454843a
Summary:
`hg backup --background` will be used as a `txnclose` hook to backup all of the
local commits to infinitepush.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4175953
Tasks: 12479677
Signature: t1:4175953:1479145307:e698903b519361b376f6e182db7c49869c992617
Summary:
I'm going to use runshellfast in infinitepush.
To avoid copy-paste let's move it to the separate package.
Note: fastmanifest also has runshellfast but it's implementation
is a bit different and seems that it doesn't work with remotefilelog.
So I'll leave it alone for now.
Test Plan: python run-tests.py -j20
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4175836
Summary:
Some users only want to build certain extensions from this repository, so let's
add some options to setup.py to let them pick exactly what parts they want.
This also has the affect of removing the build dependency on Cython unless the
user wants to build linelog.
Test Plan:
hg purge --all
python setup.py build
hg purge --all
python setup.py build --component remotefilelog
hg purge --all
python setup.py build --component remotefilelog --component linelog
Reviewers: #sourcecontrol, simonfar, quark
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3992662
Signature: t1:3992662:1475940088:28ed13ddee5215f7ac0fcb68c0f49294b6ad79e2
Summary:
Update the code to read additional include and library directories from the
INCLUDE_DIRS and LIBRARY_DIRS environment variables, if they are set.
This allows us to build the extensions using custom include/library paths.
While these could have been set through the CFLAGS and LDFLAGS variables,
distutils already includes many default settings in CFLAGS and LDFLAGS by
default (taken from the settings used to build python), and setting CFLAGS and
LDFLAGS ourself would override these defaults. We do still want all of the
flags in CFLAGS and LDFLAGS by default, so it is easier to add support for
separate INCLUDE_DIRS and LIBRARY_DIRS variables.
Test Plan: Tested building fb-mercurial-omnibus on CentOS.
Reviewers: ikostia, simonfar, ttung, durham, mbolin, wez, quark
Reviewed By: quark
Subscribers: ttung, net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3916198
Signature: t1:3916198:1475091590:97904ef87a8a2b78e178c87d9d49bdadcce94e44
Summary:
Explicitly specify the C++ dialect as c++1y when invoking gcc to compile
ctreemanifest's C++ code.
This is needed to build with Facebook's C++ code that uses folly's fbstring for
the std::string implementation.
(Ideally I would actually prefer to use -std=c++1y, but for the moment we still
build with gcc-4.4 on some platforms, and this old compiler does not support
either c++1x or c++1y.)
Test Plan: Tested building ctreemanifest with gcc-4.9.
Reviewers: ikostia, simonfar, quark
Reviewed By: quark
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3916577
Signature: t1:3916577:1474667597:3a915a7bc6d7f070d647ccd0d6b40b3eb051b59d
