Summary:
A future patch will want to import just the cstore py headers into
py-ctreemanifest, so let's move those out to their own header file.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4571320
Signature: t1:4571320:1487774077:64b4c5053b223f075a1c9f73ddcd92bd73b53e23
Summary:
To begin the process of weaning treemanifest off of Python and on to the new
cstore, let's introduce a base Store class that implements a simple get()
function. UnionDatapackStore already implements this. Future patches will make a
Python version that implements this interface as well, so we can swap between
them easily.
Test Plan: Built and ran the tests.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569870
Signature: t1:4569870:1487846232:aa816311ec37a520a2405b68e3c02ad744613049
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
All of the test-check*.t tests cd out of the test repository and into the main
normal fb-hgext repository, and attempt to run hg commands there to examine the
repository contents. Unfortunately they were doing so still using the HGRCPATH
settings configured for the test repository. In practice this causes the tests
in many situations, because extensions enabled in the system configuration and
required for the main repository may not have been enabled.
Test Plan:
Ran "arc unit" in a repository with sqldirstate enabled and confirmed the tests
pass now instead of bombing out with screenfuls of error messages.
Reviewers: #sourcecontrol, stash, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4427670
Signature: t1:4427670:1484733701:f6c859c29d34eb33b1280e798ab3398a03a3a461
Summary:
"arc lint" runs the lint script for each file in parallel:
>>> [5, pid=27257] <exec> $ scripts/lint.py 'scripts/lint.py'
>>> [6, pid=27257] <exec> $ scripts/lint.py 'scripts/unit.py'
>>> [7, pid=27257] <exec> $ scripts/lint.py 'tests/test-check-code-hg.t'
>>> [8, pid=27257] <exec> $ scripts/lint.py 'tests/test-check-pyflakes-hg.t'
That looks fine but `lint.py` runs `test-check*.t` which will check all
files. This is a typical `N^2` behavior and needs to be fixed.
This diff changes the linter and test code so the test only checks the
single selected file.
Test Plan:
Run `arc lint --trace` and check `lint.py` time usage. It went down from
10 seconds to 2 seconds for this diff.
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4051080
Signature: t1:4051080:1476985355:45845c6eb0e66cbfa000e61b7b496f2d00aeb042
Summary:
buffer.h gained the ability to deal with non-char-sized buffers when I built cdatapack. We need to update the callers in ctreemanifest to be aware of this. Most of this is done with macro magic.
Some functionality was dropped from cdatapack's buffer.h (macro definitions to deal with paths). Those are moved to path_buffer.h
Test Plan:
make local && clion build.
pass cfastmanifest unit tests.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3780767
Signature: t1:3780767:1472255278:40a19edfd171df5804e9cdfa4444d5c6386f00e8
Test Plan: run tree_dump on a fastmanifest file. did some spot checks against the text manifest with the same sha (`hg debugdata -m`)
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3585843
Tasks: 12119301
Signature: t1:3585843:1469050470:70729d5601f935f2de80f89a7ad83590bcb8dd00
Summary:
It's a common mistake that our tests require foreign extensions (namely evolve
and remotenames) without checking them first.
This diff adds checks to catch these mistakes, adds missing checks, and unifies
our checking logic using `require-ext.sh`, which is aware of `hgext3rd` and
prints skip message.
This affects `arc lint` so hopefully our new testing code would be free of this
kind of mistakes.
Test Plan: `arc lint` would catch errors
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3550977
Signature: t1:3550977:1468455857:e849dfd9e3cbc446cc6e6c662050ee88a3366e6c
Summary:
Be a better citizen under system python path.
Fix all tests issues and change setup.py to use glob pattern to include
all extensions.
Test Plan:
Run tests and `make local`.
Also build and install the package and run `hg sl` in major repos.
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3534311
Signature: t1:3534311:1468275426:fe122646c8bd6c541e1889e73e9df28f86747ff2
Summary: This allows us to use fastmanifest as a directory to drop in the python module.
Test Plan: compiles, passes existing tests.
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3351021
Signature: t1:3351021:1464284417:6cbcde514ab1fd7b5caa6c83cb5577f3502dbc58
Summary:
API to write trees to disk and read trees from disk.
Some interesting things to keep in mind:
* To keep valgrind happy, we have to clear all the memory we allocate but never initialize. That can be padding between fields in structs, fields that don't make sense to initialize, the extra byte in checksums that doesn't get used normally, or portions of bitfields that are never used. All this logic is kept compartmentalized in `initialize_unused_bytes(..)`. Normal runtime does not invoke this code, though I suspect the cost is probably not significant.
* A significant chunk of the write path is made up of the `CHECKED_WRITE_XXX` macros. They're calls to ensure that writes succeed. If not, they jump to a fixed exit point.
* The write path is not very optimized. We can use a bottom-up traversal, similar to tree_convert, to make the write less costly. However, our design doesn't require a fast write, so tentatively, I'm going to reuse `tree_copy(..)` to compact a tree.
Depends on D3255048
Test Plan: pass unit tests!
Reviewers: lcharignon, wez
Reviewed By: wez
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3255656
Tasks: 10589048
Signature: t1:3255656:1462342423:c28d32c610b2351614d7648c03f35f931370a770
Summary: It's too common of a name.
Test Plan: compiles && unit tests pass.
Reviewed By: lcharignon
Differential Revision: https://phabricator.fb.com/D3192722
Summary: This adds the CPythong interface layer for fastmanifest
Test Plan: Compiles
Reviewers: ttung, durham
Differential Revision: https://phabricator.fb.com/D3157095
Summary: Final diff of the series
Test Plan:
All tests pass with --extra-config=extensions.lz4revlog= needed
to run check-code
Reviewers: #sourcecontrol, ttung
Differential Revision: https://phabricator.fb.com/D2812684
