Summary:
This is a s//g replacement of all the `return (type_name) {` with
`return COMPOUND_LITERAL(type_name) {`.
This is the command which produced the diff:
`egrep 'return \(\w*\) \{' -Ir . --exclude='*.py*' --exclude-dir=.hg -l | xargs sed 's/return (\(\w*\)) {/return COMPOUND_LITERAL(\1) {/g' -i `
After I've done this, I checked:
`egrep '\(\w+\) \{' -Ir . --exclude='*.py*' --exclude-dir=.hg | egrep -v '(switch|while)' | grep -v 'if (' | grep -v 'COMPOUND_LITERAL' | less`
and it looks like the only things of `(something) {` syntax are function definitions, adding space before `(` in search pattern yields no results.
This is needed to make this compile on Windows under MSVC2015.
Depends on: D4843230
Test Plan:
- run `python setup.py build -f`, see it compile
- run all the tests, see them pass
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4843240
Signature: t1:4843240:1491495690:a097bfab403805052d5ac25d1db7bb32af3bca28
Summary:
Proposed header (or its dir) is a single place to put MSVC/GCC hacks. So
far it only includes the COMPOUND_LITERAL macro which behaves differently
depending on MSVC mode.
When MSVC2015 is used in C++ mode, it does not support things like:
`(my_type) {initializers}`, but in C mode it does.
To clarify: I am not even sure whether we need to have the ability to compile in a purely C mode, but I did not want to figure out.
Test Plan: - on Linux, run `python setup.py build`, run all the tests, see them passing
Reviewers: #sourcecontrol, tja
Reviewed By: tja
Subscribers: tja, jsgf, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4843230
Signature: t1:4843230:1491496062:3fa10ae5a5aac850689991de1ca6ee1ac86d9dce
Summary:
This is an RFC kind of diff, so I am looking for comments. I am not very
familiar with either C or C++.
The goal is to make all this stuff compile on Windows under MSVC2015. MSVC
does not support variable-sized arrays in C++ out-of-the-box (not sure
whether there are tricks to make it do so), so my proposal is to use
explicit `malloc`s in place.
Test Plan:
Only tested on Linux for now. `python setup.py build` works,
tests pass, seems to make sense.
Reviewers: #sourcecontrol, jsgf
Reviewed By: jsgf
Subscribers: jsgf, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4839968
Signature: t1:4839968:1491492552:c80fac6990aaee78e6bb18522ff13e02eb9521e4
Summary:
Upstream has added a matcher arg to manifest.filesnotin, so let's support it in
our native treemanifest implementation.
Test Plan: Added unit tests for filesnotin with and without matchers.
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677030
Signature: t1:4677030:1489076599:23ab67d85bb4fa689f05349253a4d2c4e7d496cc
Summary:
Upstream has added a matcher argument to the diff API which allows diff to avoid
traversing certain parts of the tree. This adds support for that to our native
treemanifest implementation.
Test Plan: Added tests for diff with matches
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677023
Signature: t1:4677023:1489076627:dbcea209d300a68fa050f68c52b4fd9949b85302
Summary:
Now that ctreemanifest no longer depends on python.h, let's move pythonutil over
to cstore where all the python code is.
Test Plan: Ran the build and the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663988
Signature: t1:4663988:1488895919:652b3fc35a2dd12c51a9f70e32997c7b4d037c95
Summary:
This is the last piece of removing the Python dependency from the core
treemanifest code. This replaces the old PythonObj diff dictionary with a new
DiffResult class that has a PythonDiffResult implementation.
Test Plan:
Ran the test suite, including the unit tests that explicitly cover
treemanifest diff from python.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663984
Signature: t1:4663984:1488889615:62f064924b0d6b45dfa7e490d19418060c374f40
Summary:
As part of breaking the native cstore implementation away from Python, let's
create a Matcher class that can be used to perform path match testing. Initially
the only implementation is the PythonMatcher which just wraps a python match
object.
Test Plan: Covered by existing matcher tests in cstore-treemanifest.py
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663354
Signature: t1:4663354:1488914384:2c33c7e0e7f2eade0786b6ff41503317989fd1e5
Summary:
The upstream matcher has a 'visitdir' function which indicates whether a walk
should traverse into a directory. Let's support this in treemanifest so we can
skip a ton of lookups for parts of the tree that aren't relevant to the match.
Test Plan:
Used gdb to verify commands like 'hg diff -r .^ -r .
--include foo' only traversed into directory foo, and not directory bar. I'm not
sure how to write a test for this since it doesn't actually affect user facing
output.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4648252
Signature: t1:4648252:1488882860:b5e1b738fef2f63336c80cd7c0eb59c2fcf16d0f
Summary:
Previously the treemanifest code itself would create the text deltas when
writing a tree out. This meant we couldn't make the delta decision based on
other data, like if the p1 commit was in the same pack file.
This patch removes treemanifest.write() and moves all calls over to
treemanifest.finalize() which gives the python/pack layer control over delta
choices. A future patch will use this to ensure tree packs always contain
complete delta chains.
Test Plan: All tests pass
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4645942
Signature: t1:4645942:1488880851:d0c8c902e7e849072a53344630a9184b6d8e1e7f
Summary:
Previously, to serialize a tree we would call treemanifest.write() which would
take the stores and call the appropriate add() functions. This meant the
treemanifest code controlled when deltas were used, which makes it hard to
decide deltas based on external factors, like which pack file the parent is in.
As a first step to fixing this, we're adding a new finalize function that
returns a python iterator that allows iterating over the new parts of the tree.
In a future diff we will use this to do serialization and delta decisions at the
python/pack layer instead.
The same future diff will also cover this in tests, as it moves all calls to
treemanifest.write() to use treemanifest.finalize().
Test Plan: The next diff switches all write() calls and tests to finalize()
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4645922
Signature: t1:4645922:1488880722:574a37ba800f4b78faf8bad3979b447f4278cc55
Summary:
Previously the treemanifest auto-tree-creation logic only produced data packs
containing the actual contents of the tree blobs. This lost history information
which is important for our ability to efficiently repack the data files.
This patch creates history packs during pull as well. A future patch will also
create history packs for the local tree blob store.
Test Plan: Updated the tests to cover this
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4638865
Signature: t1:4638865:1488449992:48b60961b50b90b6d0e75a64af1f36fb29944e7a
Summary:
We added a use of std::unordered_set::emplace, which isn't available in
GCC 4.4 (thus CentOS 6). Let's try insert instead, as that's the only other way
to add to an unordered_set.
Test Plan: Build on CentOS 7 to confirm no regression.
Reviewers: #sourcecontrol
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4642804
Summary:
Adds a treemanifest.usecunionstore config flag for enabling and disabling use of
the native code uniondatapackstore.
Since we haven't implemented the repack APIs on the native datapack stores, we
currently have to force repack to use the old python implementations. Instead of
trying to expose just the appropriate APIs through the python interface, I think
we'll rewrite all of repack to be in C++ at a future time, since we can take
advantage of parallelism, etc.
Test Plan:
Updated test-treemanifest.t to use the c datapackstore. Also run all
the tests with --extra-config-opt=treemanifest.usecunionstore=True.
These tests caught a missing null check in the C++ code as well.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609795
Signature: t1:4609795:1488365341:203362db5f470b613c4d6484686cd32c3fa8458f
Summary:
The markForRefresh function allows callers to tell the datapackstore that new
pack files have been written. This is important for being able to support
generation of treemanifests during pull and is a blocker for using the native
datapackstore in production.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4609076
Signature: t1:4609076:1488365022:d5480897f2b7674b4d07243e20a495551f985321
Summary:
Previously treemanifest always called back to Python to fetch the tree content
for a given key. This patch let's us call directly in to the cstore without
going through Python, when possible.
Test Plan:
I manually tested this by modifying the treemanifest extension to
pass a uniondatapackstore to the ctreemanifest constructor, then used the
debugger to ensure this code path was correct. I still need to add a test
though.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4571442
Signature: t1:4571442:1487847314:0d578351354ee4da34b5032c01720bdedf0d185d
Summary:
In a future patch we will want to pass the uniondatapackstore around to other
objects who will contribute to the lifetime. Let's change it to a shared_ptr so
that becomes easy.
Let's also make the destructor virtual, so we can pass different types of stores
around and have them be destructed correctly.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4603893
Signature: t1:4603893:1487847173:2fc3505032ea8c30cf9e0f76ac4e75d64513d87d
Summary:
A future patch will want to import just the cstore py headers into
py-ctreemanifest, so let's move those out to their own header file.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4571320
Signature: t1:4571320:1487774077:64b4c5053b223f075a1c9f73ddcd92bd73b53e23
Summary:
The old code kept a PythonObj around inside the ManifestFetcher for fetching
manifest contents from the store. As part of moving the treemanifest code to use
the new native cstore API let's make the manifest code depend on a Store
abstraction and have one implementation be a PythonStore.
This removes almost all of the python dependencies from the core treemanifest
code, except some logic around running the python matcher during iteration and
writing directly to the python result dict during diff. We'll abstract those
away later.
Test Plan: Built and ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569944
Signature: t1:4569944:1487847102:d005b6484fd7de9335961b0bc4530505b25f961d
Summary:
To begin the process of weaning treemanifest off of Python and on to the new
cstore, let's introduce a base Store class that implements a simple get()
function. UnionDatapackStore already implements this. Future patches will make a
Python version that implements this interface as well, so we can swap between
them easily.
Test Plan: Built and ran the tests.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569870
Signature: t1:4569870:1487846232:aa816311ec37a520a2405b68e3c02ad744613049
Summary:
This implements the get function of UnionDatapackStore which iterates over
deltas and produces a full text. Unfortunately the delta chains currently come
out in reverse order (full text last), so we have to read all of them before we
can produce the delta.
Since mpatch takes a char*, and we want to avoid copying the array into a string
array for every read, we event a ref counted string reference type which let's
us transfer ownership of the char* to the caller.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556547
Signature: t1:4556547:1487200302:c73b8f464f6ce413f1c22c0125a91fbece701f08
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
Implements the getdeltachain function on the new UnionDatapackStore class.
This required some modifications to the DeltaChainIterator. Since the results of
the iterator may cross multiple different chains, we need to keep each chain
alive until the iterator is destructed, so we need to keep a reference to each
chain. We also had to remove the size() property from the iterator since the
fact that the chain spans chains means we don't know the size up front.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556458
Signature: t1:4556458:1487199872:07dffa3121acfbeb6d6993b518e6f4887122d4d5
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
This implements a C++ version of the datapackstore getmissing() function and
adds a test.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552853
Signature: t1:4552853:1487191581:c108be4a85f8eacafddbea8086d5d22fdcabdacf
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
This implements the get function of UnionDatapackStore which iterates over
deltas and produces a full text. Unfortunately the delta chains currently come
out in reverse order (full text last), so we have to read all of them before we
can produce the delta.
Since mpatch takes a char*, and we want to avoid copying the array into a string
array for every read, we event a ref counted string reference type which let's
us transfer ownership of the char* to the caller.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556547
Signature: t1:4556547:1487200302:c73b8f464f6ce413f1c22c0125a91fbece701f08
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
Implements the getdeltachain function on the new UnionDatapackStore class.
This required some modifications to the DeltaChainIterator. Since the results of
the iterator may cross multiple different chains, we need to keep each chain
alive until the iterator is destructed, so we need to keep a reference to each
chain. We also had to remove the size() property from the iterator since the
fact that the chain spans chains means we don't know the size up front.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556458
Signature: t1:4556458:1487199872:07dffa3121acfbeb6d6993b518e6f4887122d4d5
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
This implements a C++ version of the datapackstore getmissing() function and
adds a test.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552853
Signature: t1:4552853:1487191581:c108be4a85f8eacafddbea8086d5d22fdcabdacf
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
