Summary:
I sync'd a copy of this code into the eden repository.
I had to adjust a couple of include paths to get the code to
compile correctly in the hermetic build environment that is
in use there.
In addition, our linter suite over there found a couple of C++ nits
to be fixed up.
Test Plan: make local
Reviewers: simpkins, ikostia, simonfar, durham
Reviewed By: durham
Subscribers: net-systems-diffs@fb.com, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4879285
Signature: t1:4879285:1492039044:8cb1e033e35ee568806de94dda3d2f6f8e78f5cb
Summary:
The portability/ subdirectory always needs to be listed in the compiler include
paths in order to successfully build. This fixes setup.py to always add it to
the list of include directories, rather than only adding it when INCLUDE_DIRS
was not specified through the environment.
Test Plan:
Successfully built the fb-hgext repository when specifying alternate system
header paths through the INCLUDE_DIRS environment variable.
Reviewers: tja, wez, ikostia
Reviewed By: ikostia
Subscribers: net-systems-diffs@fb.com, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4865296
Signature: t1:4865296:1491903269:fc07edf01a89117e25805e9ee936b2e2eee22410
Summary:
Proposed header (or its dir) is a single place to put MSVC/GCC hacks. So
far it only includes the COMPOUND_LITERAL macro which behaves differently
depending on MSVC mode.
When MSVC2015 is used in C++ mode, it does not support things like:
`(my_type) {initializers}`, but in C mode it does.
To clarify: I am not even sure whether we need to have the ability to compile in a purely C mode, but I did not want to figure out.
Test Plan: - on Linux, run `python setup.py build`, run all the tests, see them passing
Reviewers: #sourcecontrol, tja
Reviewed By: tja
Subscribers: tja, jsgf, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4843230
Signature: t1:4843230:1491496062:3fa10ae5a5aac850689991de1ca6ee1ac86d9dce
Summary:
While this does not change any real behavior, it makes `python setup.py`
output a bit cleaner as it does not pass irrelevant options to `cl.exe`
and command lines are shorter.
Test Plan: - build on Linux, make sure it still builds
Reviewers: #sourcecontrol, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4834980
Signature: t1:4834980:1491427834:3f5be0644b02ae488946366ff733d0bf1ffdce71
78d6c7763d48 changed how component is tested, but forgot to change the
default component. This patch fixes it so setup.py running without
"--component" will work as expected.
Summary:
The ideal interactive mode couldn't be implemented trivially. Move `absorb`
to a directory so we can add related, but decoupled components as separate
files.
Test Plan: `make local`
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4813909
Signature: t1:4813909:1491211561:e9c40b1242c9b74230c0b8937723a2d4548e22c3
Summary:
I accidently disabled fastmanifest and a bunch of other packages on non-win
platforms, instead of Windows itself. This should fix it.
Test Plan: - makes sense visually
Reviewers: #sourcecontrol, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4761873
Signature: t1:4761873:1490267573:74e81c97652f15b06b7b7a56f96bf3d35385f54e
Summary:
We have encountered a kernel issue where `rmdir` a non-empty directory may race
with other things and hang in kernel for a long time.
This patch changes `os.rmdir` to avoid `rmdir` non-empty directories. It is
written in Cython calling the low-level `readdir` libc friends to make overhead
minimal.
Test Plan: Added a new test
Reviewers: #sourcecontrol, clm, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, simpkins, osandov, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4716711
Tasks: 16647532
Signature: t1:4716711:1489627923:7c7432748c1fd8c070ce257bd172feebd3807f65
Summary:
Now that ctreemanifest no longer depends on python.h, let's move pythonutil over
to cstore where all the python code is.
Test Plan: Ran the build and the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663988
Signature: t1:4663988:1488895919:652b3fc35a2dd12c51a9f70e32997c7b4d037c95
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
Linking native python extensions together is a huge pain and I'm encountering
bugs in the linker in centos6. To simplify everything, I'm going to merge all
the cstore related libraries into a single library.
Step one is to move the C libdatapack code to be a statically linked library.
distutils doesn't actually support using -l or extra args in it's static library
generator, so we need to monkey patch a fix for that here.
Test Plan:
Built using 'python setup.py build --component cdatapack'. Verified the static library was built:
```
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Iclib -I/usr/local/include -I/opt/local/include -I/opt/homebrew/include/ -c cdatapack/cdatapack.c -o build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o -std=c99 -Wall -Werror -Werror=strict-prototypes
ar rc build/temp.linux-x86_64-2.7/libdatapack.a build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o
```
And the resulting cdatapack.o library did not reference a libdatapack shared library any more by running 'ldd cdatapack.so'. On OSX verified the same thing using "otool -L cdatapack.so"
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581274
Signature: t1:4581274:1487842180:d2125ebec7bbb9533e02b0589f188251881ed569
Summary:
As part of a refactoring to move all the store code together, we need to break
the cdatapack extension into two parts, the actual c structure and the python
extension. In a future patch the python extension part will be moved into the
cstore with everything else, but the c structure needs to remain separate since
it's C code and not C++.
Test Plan: Ran the tests
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581268
Signature: t1:4581268:1487669407:9d97fcdfd04689c95494a76ead515043a71a7f38
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary: The situation here is that our FB C extensions just won't compile on MSVC at the moment. This simply strips them out of consideration for building on Windows until they can be compiled on Windows later.
Test Plan: Built fb.hg, it didn't include these c extensions
Reviewers: #idi, durham, #sourcecontrol, rmcelroy, davidsp
Reviewed By: rmcelroy, davidsp
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4270189
Signature: t1:4270189:1481612143:9c035b4c6eb4af6b542795a64b24dc274454843a
Summary:
`hg backup --background` will be used as a `txnclose` hook to backup all of the
local commits to infinitepush.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4175953
Tasks: 12479677
Signature: t1:4175953:1479145307:e698903b519361b376f6e182db7c49869c992617
Summary:
I'm going to use runshellfast in infinitepush.
To avoid copy-paste let's move it to the separate package.
Note: fastmanifest also has runshellfast but it's implementation
is a bit different and seems that it doesn't work with remotefilelog.
So I'll leave it alone for now.
Test Plan: python run-tests.py -j20
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4175836
Summary:
Some users only want to build certain extensions from this repository, so let's
add some options to setup.py to let them pick exactly what parts they want.
This also has the affect of removing the build dependency on Cython unless the
user wants to build linelog.
Test Plan:
hg purge --all
python setup.py build
hg purge --all
python setup.py build --component remotefilelog
hg purge --all
python setup.py build --component remotefilelog --component linelog
Reviewers: #sourcecontrol, simonfar, quark
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3992662
Signature: t1:3992662:1475940088:28ed13ddee5215f7ac0fcb68c0f49294b6ad79e2
Summary:
Update the code to read additional include and library directories from the
INCLUDE_DIRS and LIBRARY_DIRS environment variables, if they are set.
This allows us to build the extensions using custom include/library paths.
While these could have been set through the CFLAGS and LDFLAGS variables,
distutils already includes many default settings in CFLAGS and LDFLAGS by
default (taken from the settings used to build python), and setting CFLAGS and
LDFLAGS ourself would override these defaults. We do still want all of the
flags in CFLAGS and LDFLAGS by default, so it is easier to add support for
separate INCLUDE_DIRS and LIBRARY_DIRS variables.
Test Plan: Tested building fb-mercurial-omnibus on CentOS.
Reviewers: ikostia, simonfar, ttung, durham, mbolin, wez, quark
Reviewed By: quark
Subscribers: ttung, net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3916198
Signature: t1:3916198:1475091590:97904ef87a8a2b78e178c87d9d49bdadcce94e44
Summary:
Explicitly specify the C++ dialect as c++1y when invoking gcc to compile
ctreemanifest's C++ code.
This is needed to build with Facebook's C++ code that uses folly's fbstring for
the std::string implementation.
(Ideally I would actually prefer to use -std=c++1y, but for the moment we still
build with gcc-4.4 on some platforms, and this old compiler does not support
either c++1x or c++1y.)
Test Plan: Tested building ctreemanifest with gcc-4.9.
Reviewers: ikostia, simonfar, quark
Reviewed By: quark
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3916577
Signature: t1:3916577:1474667597:3a915a7bc6d7f070d647ccd0d6b40b3eb051b59d
Summary:
-Werror=strict-prototypes is only valid when compiling C code, not C++.
(For C++ function prototypes are always required, and you'll always get a
compile error if you don't define a function before using it.)
Passing in this flag when compiling a C++ file simply causes gcc to complain,
so drop the flag.
Test Plan:
Built the extensions, and confirmed gcc no longer complains about using this
flag for C++ files.
Reviewers: durham, mitrandir, ttung
Reviewed By: ttung
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3911350
Signature: t1:3911350:1474589572:d6efc4a213e23d3c8c53a36b0f48af23e7eee604
Summary:
This adds a function for computing the final node of a manifest, given its two
parent nodes.
Test Plan:
Used this as part of a future diff that serialize manifests into a
pack file.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3837737
Summary: I'm tired of mucking around in setup.py every time I need a debug build, and then accidentally checking those changes in.
Test Plan: `FB_HGEXT_CDEBUG= make local`
Reviewers: #fastmanifest, quark, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3869286
Signature: t1:3869286:1473922596:af05c5bf0594694c28bca577133f576b8b7d533f
Summary:
buffer.h gained the ability to deal with non-char-sized buffers when I built cdatapack. We need to update the callers in ctreemanifest to be aware of this. Most of this is done with macro magic.
Some functionality was dropped from cdatapack's buffer.h (macro definitions to deal with paths). Those are moved to path_buffer.h
Test Plan:
make local && clion build.
pass cfastmanifest unit tests.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3780767
Signature: t1:3780767:1472255278:40a19edfd171df5804e9cdfa4444d5c6386f00e8
Test Plan: make local, see same errors that I saw in remotefilelog repo
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3777382
Tasks: 12855049
Signature: t1:3777382:1472216857:efc09d134edb63f36c47187f7d443319d4609d62
Summary:
The minimal version of Cython to build linelog is 0.22. Add a check so people
know it's not a source code issue, but Cython needs upgrade.
Test Plan:
Run with different Cython releases and make sure it complains when Cython
version < 0.22. Also make sure it works with strange versions like "0.25a1".
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3706559
Signature: t1:3706559:1471463690:92b981e2df63ad63ab55edb5e0b937455254351d
Summary:
This diff builds linelog. It brings Cython as a build dependency as linelog
CPython wrapper is written in Cython. Alternatively we can check in the
generated, non-human-readable `.c` code if we get feedback that the Cython
dependency is making things hard.
Note that `setuptools` (Python 2.6 version) has issues with Cython therefore
removed.
Test Plan: `make local`, then check `linelog.so` is built and `import linelog` works.
Reviewers: #mercurial, rmcelroy, ttung
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3669753
Signature: t1:3669753:1470403459:5bf81861b31deed5076c2322c4015d5dd4b4673e
Summary:
D3699216 made linelog compatible with C99. Therefore no longer to check if the
compiler supports `-std=c11` or not.
Test Plan: `make local`
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3699242
Signature: t1:3699242:1470863312:934e1105f2c7959b6c78724bae93da879a897dd1
Summary:
We are going to use some `C11` features, namely unnamed union fields.
Sadly we still have to support gcc4.4, which does not support `-std=c11` while
its `-std=gnu99` compiles the code but with the warning
`ISO C doesn't support unnamed structs/unions`.
This patch adds a feature test function to help detect compiler differences.
Test Plan:
```
make local
```
Reviewers: #mercurial, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3649021
Signature: t1:3649021:1470082430:bced7bd155eb60cc8c3e1c2d950092e2abd39a00
Summary:
Since statprof is not an extension, it needs to be treated like a normal python
module.
Test Plan:
```
~/local/fb-hgext> python setup.py build
~/local/fb-hgext> ls build/lib.linux-x86_64-2.6/
cfastmanifest.so hgext3rd/ sqldirstate/
fastmanifest/ phabricator/ statprof.py
```
Reviewers: ttung, #sourcecontrol, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3607202
Signature: t1:3607202:1469216683:9aa08aee567425c239ddc5d1bb68fd892a8cf6a9
Summary:
Be a better citizen under system python path.
Fix all tests issues and change setup.py to use glob pattern to include
all extensions.
Test Plan:
Run tests and `make local`.
Also build and install the package and run `hg sl` in major repos.
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3534311
Signature: t1:3534311:1468275426:fe122646c8bd6c541e1889e73e9df28f86747ff2
