Summary:
The old code kept a PythonObj around inside the ManifestFetcher for fetching
manifest contents from the store. As part of moving the treemanifest code to use
the new native cstore API let's make the manifest code depend on a Store
abstraction and have one implementation be a PythonStore.
This removes almost all of the python dependencies from the core treemanifest
code, except some logic around running the python matcher during iteration and
writing directly to the python result dict during diff. We'll abstract those
away later.
Test Plan: Built and ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569944
Signature: t1:4569944:1487847102:d005b6484fd7de9335961b0bc4530505b25f961d
Summary:
To begin the process of weaning treemanifest off of Python and on to the new
cstore, let's introduce a base Store class that implements a simple get()
function. UnionDatapackStore already implements this. Future patches will make a
Python version that implements this interface as well, so we can swap between
them easily.
Test Plan: Built and ran the tests.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569870
Signature: t1:4569870:1487846232:aa816311ec37a520a2405b68e3c02ad744613049
Summary:
This implements the get function of UnionDatapackStore which iterates over
deltas and produces a full text. Unfortunately the delta chains currently come
out in reverse order (full text last), so we have to read all of them before we
can produce the delta.
Since mpatch takes a char*, and we want to avoid copying the array into a string
array for every read, we event a ref counted string reference type which let's
us transfer ownership of the char* to the caller.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556547
Signature: t1:4556547:1487200302:c73b8f464f6ce413f1c22c0125a91fbece701f08
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
Implements the getdeltachain function on the new UnionDatapackStore class.
This required some modifications to the DeltaChainIterator. Since the results of
the iterator may cross multiple different chains, we need to keep each chain
alive until the iterator is destructed, so we need to keep a reference to each
chain. We also had to remove the size() property from the iterator since the
fact that the chain spans chains means we don't know the size up front.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556458
Signature: t1:4556458:1487199872:07dffa3121acfbeb6d6993b518e6f4887122d4d5
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
This implements a C++ version of the datapackstore getmissing() function and
adds a test.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552853
Signature: t1:4552853:1487191581:c108be4a85f8eacafddbea8086d5d22fdcabdacf
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
This implements the get function of UnionDatapackStore which iterates over
deltas and produces a full text. Unfortunately the delta chains currently come
out in reverse order (full text last), so we have to read all of them before we
can produce the delta.
Since mpatch takes a char*, and we want to avoid copying the array into a string
array for every read, we event a ref counted string reference type which let's
us transfer ownership of the char* to the caller.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: stash, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556547
Signature: t1:4556547:1487200302:c73b8f464f6ce413f1c22c0125a91fbece701f08
Summary:
A future patch will add C++ logic that applies delta's to get full texts, so we
need access to the mpatch code. This is a verbatim copy from core, along with
it's dependencies.
Test Plan: N/A It gets used as part of the next patch
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556537
Signature: t1:4556537:1487076858:528343cb0a74de9262bbb5927ec8d186dafaef45
Summary:
Implements the getdeltachain function on the new UnionDatapackStore class.
This required some modifications to the DeltaChainIterator. Since the results of
the iterator may cross multiple different chains, we need to keep each chain
alive until the iterator is destructed, so we need to keep a reference to each
chain. We also had to remove the size() property from the iterator since the
fact that the chain spans chains means we don't know the size up front.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556458
Signature: t1:4556458:1487199872:07dffa3121acfbeb6d6993b518e6f4887122d4d5
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
This implements a C++ version of the datapackstore getmissing() function and
adds a test.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4552853
Signature: t1:4552853:1487191581:c108be4a85f8eacafddbea8086d5d22fdcabdacf
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
