Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
Linking native python extensions together is a huge pain and I'm encountering
bugs in the linker in centos6. To simplify everything, I'm going to merge all
the cstore related libraries into a single library.
Step one is to move the C libdatapack code to be a statically linked library.
distutils doesn't actually support using -l or extra args in it's static library
generator, so we need to monkey patch a fix for that here.
Test Plan:
Built using 'python setup.py build --component cdatapack'. Verified the static library was built:
```
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Iclib -I/usr/local/include -I/opt/local/include -I/opt/homebrew/include/ -c cdatapack/cdatapack.c -o build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o -std=c99 -Wall -Werror -Werror=strict-prototypes
ar rc build/temp.linux-x86_64-2.7/libdatapack.a build/temp.linux-x86_64-2.7/cdatapack/cdatapack.o
```
And the resulting cdatapack.o library did not reference a libdatapack shared library any more by running 'ldd cdatapack.so'. On OSX verified the same thing using "otool -L cdatapack.so"
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581274
Signature: t1:4581274:1487842180:d2125ebec7bbb9533e02b0589f188251881ed569
Summary:
As part of a refactoring to move all the store code together, we need to break
the cdatapack extension into two parts, the actual c structure and the python
extension. In a future patch the python extension part will be moved into the
cstore with everything else, but the c structure needs to remain separate since
it's C code and not C++.
Test Plan: Ran the tests
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581268
Signature: t1:4581268:1487669407:9d97fcdfd04689c95494a76ead515043a71a7f38
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
Building in debug mode was failing because python emits some warning about some
define needing to only be used in opt builds. Since we have -Werror, this gets
treated like an error. Let's not use -Werror in debug builds.
Test Plan: hg purge --all && FB_HGEXT_CDEBUG=1 python setup.py build --component cstore
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552872
Signature: t1:4552872:1487075732:35b639bc0abd0b2d70b8e0dedd83f35c26396b10
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary: The situation here is that our FB C extensions just won't compile on MSVC at the moment. This simply strips them out of consideration for building on Windows until they can be compiled on Windows later.
Test Plan: Built fb.hg, it didn't include these c extensions
Reviewers: #idi, durham, #sourcecontrol, rmcelroy, davidsp
Reviewed By: rmcelroy, davidsp
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4270189
Signature: t1:4270189:1481612143:9c035b4c6eb4af6b542795a64b24dc274454843a
Summary:
`hg backup --background` will be used as a `txnclose` hook to backup all of the
local commits to infinitepush.
Test Plan: Run `test-infinitepush-*`
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D4175953
Tasks: 12479677
Signature: t1:4175953:1479145307:e698903b519361b376f6e182db7c49869c992617
Summary:
I'm going to use runshellfast in infinitepush.
To avoid copy-paste let's move it to the separate package.
Note: fastmanifest also has runshellfast but it's implementation
is a bit different and seems that it doesn't work with remotefilelog.
So I'll leave it alone for now.
Test Plan: python run-tests.py -j20
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4175836
Summary:
Some users only want to build certain extensions from this repository, so let's
add some options to setup.py to let them pick exactly what parts they want.
This also has the affect of removing the build dependency on Cython unless the
user wants to build linelog.
Test Plan:
hg purge --all
python setup.py build
hg purge --all
python setup.py build --component remotefilelog
hg purge --all
python setup.py build --component remotefilelog --component linelog
Reviewers: #sourcecontrol, simonfar, quark
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3992662
Signature: t1:3992662:1475940088:28ed13ddee5215f7ac0fcb68c0f49294b6ad79e2
Summary:
Update the code to read additional include and library directories from the
INCLUDE_DIRS and LIBRARY_DIRS environment variables, if they are set.
This allows us to build the extensions using custom include/library paths.
While these could have been set through the CFLAGS and LDFLAGS variables,
distutils already includes many default settings in CFLAGS and LDFLAGS by
default (taken from the settings used to build python), and setting CFLAGS and
LDFLAGS ourself would override these defaults. We do still want all of the
flags in CFLAGS and LDFLAGS by default, so it is easier to add support for
separate INCLUDE_DIRS and LIBRARY_DIRS variables.
Test Plan: Tested building fb-mercurial-omnibus on CentOS.
Reviewers: ikostia, simonfar, ttung, durham, mbolin, wez, quark
Reviewed By: quark
Subscribers: ttung, net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3916198
Signature: t1:3916198:1475091590:97904ef87a8a2b78e178c87d9d49bdadcce94e44
Summary:
Explicitly specify the C++ dialect as c++1y when invoking gcc to compile
ctreemanifest's C++ code.
This is needed to build with Facebook's C++ code that uses folly's fbstring for
the std::string implementation.
(Ideally I would actually prefer to use -std=c++1y, but for the moment we still
build with gcc-4.4 on some platforms, and this old compiler does not support
either c++1x or c++1y.)
Test Plan: Tested building ctreemanifest with gcc-4.9.
Reviewers: ikostia, simonfar, quark
Reviewed By: quark
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3916577
Signature: t1:3916577:1474667597:3a915a7bc6d7f070d647ccd0d6b40b3eb051b59d
Summary:
-Werror=strict-prototypes is only valid when compiling C code, not C++.
(For C++ function prototypes are always required, and you'll always get a
compile error if you don't define a function before using it.)
Passing in this flag when compiling a C++ file simply causes gcc to complain,
so drop the flag.
Test Plan:
Built the extensions, and confirmed gcc no longer complains about using this
flag for C++ files.
Reviewers: durham, mitrandir, ttung
Reviewed By: ttung
Subscribers: net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3911350
Signature: t1:3911350:1474589572:d6efc4a213e23d3c8c53a36b0f48af23e7eee604
Summary:
This adds a function for computing the final node of a manifest, given its two
parent nodes.
Test Plan:
Used this as part of a future diff that serialize manifests into a
pack file.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3837737
Summary: I'm tired of mucking around in setup.py every time I need a debug build, and then accidentally checking those changes in.
Test Plan: `FB_HGEXT_CDEBUG= make local`
Reviewers: #fastmanifest, quark, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3869286
Signature: t1:3869286:1473922596:af05c5bf0594694c28bca577133f576b8b7d533f
Summary:
buffer.h gained the ability to deal with non-char-sized buffers when I built cdatapack. We need to update the callers in ctreemanifest to be aware of this. Most of this is done with macro magic.
Some functionality was dropped from cdatapack's buffer.h (macro definitions to deal with paths). Those are moved to path_buffer.h
Test Plan:
make local && clion build.
pass cfastmanifest unit tests.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3780767
Signature: t1:3780767:1472255278:40a19edfd171df5804e9cdfa4444d5c6386f00e8
Test Plan: make local, see same errors that I saw in remotefilelog repo
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3777382
Tasks: 12855049
Signature: t1:3777382:1472216857:efc09d134edb63f36c47187f7d443319d4609d62
Summary:
The minimal version of Cython to build linelog is 0.22. Add a check so people
know it's not a source code issue, but Cython needs upgrade.
Test Plan:
Run with different Cython releases and make sure it complains when Cython
version < 0.22. Also make sure it works with strange versions like "0.25a1".
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3706559
Signature: t1:3706559:1471463690:92b981e2df63ad63ab55edb5e0b937455254351d
Summary:
This diff builds linelog. It brings Cython as a build dependency as linelog
CPython wrapper is written in Cython. Alternatively we can check in the
generated, non-human-readable `.c` code if we get feedback that the Cython
dependency is making things hard.
Note that `setuptools` (Python 2.6 version) has issues with Cython therefore
removed.
Test Plan: `make local`, then check `linelog.so` is built and `import linelog` works.
Reviewers: #mercurial, rmcelroy, ttung
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3669753
Signature: t1:3669753:1470403459:5bf81861b31deed5076c2322c4015d5dd4b4673e
Summary:
D3699216 made linelog compatible with C99. Therefore no longer to check if the
compiler supports `-std=c11` or not.
Test Plan: `make local`
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3699242
Signature: t1:3699242:1470863312:934e1105f2c7959b6c78724bae93da879a897dd1
Summary:
We are going to use some `C11` features, namely unnamed union fields.
Sadly we still have to support gcc4.4, which does not support `-std=c11` while
its `-std=gnu99` compiles the code but with the warning
`ISO C doesn't support unnamed structs/unions`.
This patch adds a feature test function to help detect compiler differences.
Test Plan:
```
make local
```
Reviewers: #mercurial, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3649021
Signature: t1:3649021:1470082430:bced7bd155eb60cc8c3e1c2d950092e2abd39a00
Summary:
Since statprof is not an extension, it needs to be treated like a normal python
module.
Test Plan:
```
~/local/fb-hgext> python setup.py build
~/local/fb-hgext> ls build/lib.linux-x86_64-2.6/
cfastmanifest.so hgext3rd/ sqldirstate/
fastmanifest/ phabricator/ statprof.py
```
Reviewers: ttung, #sourcecontrol, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3607202
Signature: t1:3607202:1469216683:9aa08aee567425c239ddc5d1bb68fd892a8cf6a9
Summary:
Be a better citizen under system python path.
Fix all tests issues and change setup.py to use glob pattern to include
all extensions.
Test Plan:
Run tests and `make local`.
Also build and install the package and run `hg sl` in major repos.
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3534311
Signature: t1:3534311:1468275426:fe122646c8bd6c541e1889e73e9df28f86747ff2
Summary:
reflog is dead, long live journal! journal is at feature parity and reflog can
go. I've marked this as a move to help find what happened in history.
Test Plan:
Run all the tests to see if anything broke:
$HGSRC/tests/run-tests.py -l
Reviewers: #sourcecontrol, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3498613
Tasks: 10804988
Signature: t1:3498613:1467232574:75c80b0223c101236b4d4e0f27f46f72797ebe90
Summary:
This is slated to be the upstream version of reflog. It is being hashed out
here to accellerate moving from reflog to journal.
This initial revision only supports bookmarks; later diffs add remote bookmarks
and working copy changes. As such the storage format is built to support those
usecases but the commandline usage is still geared towards only needing to list
bookmarks for now.
We'll replace reflog.py with journal.py altogether, but due to the fine details
of our RPM / chef config duality reflog.py needs to stick around until we
rolled out the RPM everywhere and updated chef to follow.
Test Plan:
Run the included test-journal.t test
$HGSRC/tests/run-tests.py -l test-journal.t
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3458077
Tasks: 10804988
Signature: t1:3458077:1467215042:d3fbba283901de6abee5d2b28a43ed9293627795
Summary: I plan to use it to ensure proper loading ordr in case of sqldirstate
Test Plan:
~/clowncopter/tests/run-tests.py -j 30
Reviewers: #mercurial, ttung, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3472358
Signature: t1:3472358:1466637677:3b8b5a89ef7dfbcde4a1e7b8ea58348b07a6a969
Summary:
This extensions prevents mercurial from being ctrl-c'ed by warning the
user on the first SIGINT. The subsequent interrupts are handled normally.
Test Plan:
see tests
(the tests are lacking the scenario of double ctrl-c - any suggestions on how
to do it in nice way inside the test framework welcome)
Reviewers: #sourcecontrol, durham, wez, quark, akushner, rmcelroy
Subscribers: lcharignon, quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D2936488
Summary: We need to add `patchpython` to `setup.py` so it can get built.
Test Plan: `make local`
Reviewers: #sourcecontrol, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3456715
Summary: Newer versions of gcc (5.x) turn on `-Wstrict-prototype` by default. Turn this on in our setup.py file for older compilers, and fix the errors that crop up from this.
Test Plan: make local
Reviewers: tnardone, wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3414620
Signature: t1:3414620:1465514102:d91026062582e2a3ab3fc773bbf17686fa38609f
Summary:
This overrides the normal Mercurial statprof logic to allow us to pass custom
configuration to our customized version of statprof. In particular, it allows us
to choose between the thread and the signal profile methods, and between the
hotpath and json output formats.
Test Plan: hg.real log -r tip --config extensions.statprofext=../fb-hgext/statprofext.py --profile --config statprof.format='json' --config profiling.output=woooo.log
Reviewers: #mercurial, ttung, lcharignon
Reviewed By: lcharignon
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3408386
Signature: t1:3408386:1465485488:400da1f8e1787674c9ea3888166767a485f2fbdf
Summary: I forgot to add it to setup.py previously
Test Plan: How to test that? Is running the build enough?
Reviewers: ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3399075
Summary: This combines D3351047, D3351086, and D3351137 into one diff. This allows mercurial to better remember where things came from since these are all hg cp commands followed by some edits. However, they were separated into multiple diffs for ease of reviews.
Test Plan: run unit tests
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3354405
Signature: t1:3354405:1464291057:da140a02b9c87186b1dd9f03f8236d5996319fff
