urlgrabber is not available on Windows machines; urllib3 is MIT licensed so can
safely be bundled.
Test Plan:
Run the tests, run hg ssl against a repository with valid arcanist config.
Differential Revision: https://phab.mercurial-scm.org/D1026
Summary:
Now uniondatapackstore can also hold python data stores. PythonDataStore
wrapper simply passes function calls to underlying python objects and marshals
the output.
Test Plan:
* Added test case
* Tested on fbsource with treemanifest.usecunionstore=True
Reviewers: durham, simonfar, ryanmce, #fbhgext
Reviewed By: durham, simonfar, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D631
Summary:
Extended DeltaChain with CDeltaChain and PyDeltaChain which are wrappers around
c and python delta chains respectively. The declaration and implementation
of c and python delta chains as well as DeltaChainLink were put in a different
file.
Test Plan: * Ensure that unit tests pass
Reviewers: ryanmce, durham, simonfar, #fbhgext
Reviewed By: ryanmce, durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D630
Summary:
PythonKeyIterator has been moved to a separate file, because we will need to
include it in several places in future commits.
Test Plan: * Ensure unit tests pass
Reviewers: ryanmce, durham, simonfar, #fbhgext
Reviewed By: ryanmce, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D629
Summary:
Implementation of uniondatapackstore that can hold generic stores.
This diff doesn't support python data stores yet, but provides an abstraction
to the stores that can be used by uniondatapackstore.
DeltaChain and DeltaChainLink wrappers will allow to use C and python chains
and chain links.
Test Plan:
* Ensure that unit tests pass
* Test on fbsource to ensure new code is executed by printfing to stdout
* Ensure the code is built successfully on macOS
Reviewers: durham, ryanmce, simonfar, #fbhgext
Reviewed By: durham, #fbhgext
Differential Revision: https://phab.mercurial-scm.org/D560
Summary:
Update the copyright headers in most of the C/C++ code consistently use the
GPLv2 copyright message. This allows these files to pass Facebook's internal
C/C++ linters.
Some of the files in fbcode/scm/hgext/cstore/ appear to have actually been
copied from the hg-crew repository, and were not originally authored by
Facebook. I have not modified the copyright statements in these files:
- cstore/bitmanipulation.h
- cstore/compat.h
- cstore/mpatch.h
- cstore/mpatch.c
I also have not modified any of the cfastmanifest code.
This corresponds to Facebook diff D5588677.
Test Plan:
Confirmed that Facebook's C++ linters no longer complain about the copyright
messages.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D507
Summary:
Define the ManifestPtr class in its own header file, to eliminate some of the
nasty circular dependencies between manifest.h, manifest_entry.h, and
manifest_fetcher.h
This makes the include process for these files much simpler, and prevents
manifest.h and manifest_entry.h from both trying to include each other at
different locations in their files.
This corresponds to Facebook diff D5588672.
Test Plan: Confirmed "make local" succeeds and the unit tests pass.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D506
Summary:
Add a clib/sha1.h file with SHA-1 utility functions that hide the details of
the underlying SHA-1 implementation being used. This will make it easier in
the future if we want to use the faster SHA-1 implementation from OpenSSL if it
is available, but fall back to the sha1collisiondetection library if it is not
available.
Test Plan: Confirmed the code builds and passes unit tests.
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D282
Summary:
Move the third-party sha1collisiondetection code from clib/sha1 to
third-party/sha1dc. This helps isolate third-party code from our own
internally developed code.
This also updates the code to use the same include paths and library names as
used by the sha1collisiondetection's upstream Makefile, which would be needed
to link against a version of sha1collisiondetection installed locally.
Test Plan:
Confirmed "make local" succeeds.
All of the tests pass, except for test-check-commit-hg, which complains about
the fact that some of this third-party code contains multiple empty lines in a
row. It doesn't seem straightforward to update test-check-commit to ignore
this third-party code, but these test failures shouldn't affect any future
commits.
Reviewers: #fbhgext, ryanmce
Reviewed By: #fbhgext, ryanmce
Differential Revision: https://phab.mercurial-scm.org/D281
Summary:
Enhance check-ext script to be more strict:
- Only one foreign extension is allowed: `remotenames`
- Require explicit path for in-repo extensions to avoid wrong extensions
being tested
This would make the test more predicatable since system extensions
will be less likely to be imported. Explicit path is better than
setting `PYTHONPATH` since `hgext/name.py` could override
`hgext3rd/name.py` regardless of `PYTHONPATH`.
Test Plan: arc unit
Reviewers: phillco, durham, ikostia, #mercurial, stash
Reviewed By: stash
Subscribers: medson, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5271430
Signature: t1:5271430:1497861776:7dd35ec7c522cd9b26aa0871cb4306b4f1b8993a
Summary:
This diff removes "unset HGRCPATH" to try to run tests in a more
controlled environment. And skip the test if system HG looks broken.
Test Plan: arc unit
Reviewers: simpkins, #mercurial, stash
Reviewed By: stash
Subscribers: net-systems-diffs@fb.com, mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D5263484
Signature: t1:5263484:1497604378:28ec94c33ba77680c8d2325a8fa36457e0eb8c91
Summary:
This removes Cython as a build dependency as requested by our users.
Added a test to prevent check-in files with `/home/`.
Modified check-code to skip checking Cython generated files.
Test Plan:
Run `make clean local` with Cython installed and uninstalled.
Also run `make clean local` with `USECYTHON=1` and `0`.
Reviewers: #mercurial, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters, medson
Differential Revision: https://phabricator.intern.facebook.com/D5252935
Signature: t1:5252935:1497525190:402798077d44e39e24fcb037535ec7ffd1af9c4b
Summary:
There are two `convert.h`. This patch unifies them and does cleanups so it's
a valid header file which could be included by multiple .c and .cpp files
and linker won't complain re-definition (by adding `static` to everything).
Besides, reformat the code so it could pass check-code. Also fix a compiler
warning about comparing an unsigned integer with a signed integer.
Test Plan: `make clean local`. It still builds.
Reviewers: durham, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5113550
Signature: t1:5113550:1495565413:e399f898ac513e64af37dab5daf55cedbabfc703
Summary:
We now use setup.py to do the build so this is outdated. It does not seem to be
useful so just remove it.
Test Plan: eyes
Reviewers: durham, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, ttung, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5112853
Signature: t1:5112853:1495561149:a4ee408e1fe06dea32f96ae93abb38d7d21d3619
Summary:
statprof is now part of core hg. And core hg will import its own version
instead of the external one. So it's no longer to have a separate version.
I did a comparison. The version in fb-hgext is falling behind - it lacks of
Chrome support. The upstream version is also cleaner in terms of Python3 code
standard and error handling.
Test Plan: eyes
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5112666
Signature: t1:5112666:1495562274:7e1ca7d964bf617695dbefd59c3843a3ebf47f9e
Summary:
This imports a SHA1 hash function implementation from
https://github.com/cr-marcstevens/sha1collisiondetection, commit hash
0572d8a302b1f62f25c1559b0de74908550ab2c3, using the MIT License.
This allows us to decouple from an external crypto library like openssl. The
new library's "safe hash" feature could also mitigate known SHA1 collisions.
test-check-code is modified accordingly to skip checking the sha1 code
without modifying them.
Test Plan: Run test-check-code.t and make sure it's not complaining.
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: ikostia, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4945012
Signature: t1:4945012:1493154822:be0deb451d50f3a8b6cc42addf091349bf6554f4
Summary:
We now use `check-code.py` directly from core mercurial. It's no longer
necessary to have it in our repo.
Test Plan: `arc unit`
Reviewers: #sourcecontrol, jeroenv, rmcelroy
Reviewed By: jeroenv, rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4945054
Signature: t1:4945054:1493122097:5880dc2c0caac215cb10db40d30691dc1a147315
Summary:
Now that ctreemanifest no longer depends on python.h, let's move pythonutil over
to cstore where all the python code is.
Test Plan: Ran the build and the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663988
Signature: t1:4663988:1488895919:652b3fc35a2dd12c51a9f70e32997c7b4d037c95
Summary:
As part of breaking the native cstore implementation away from Python, let's
create a Matcher class that can be used to perform path match testing. Initially
the only implementation is the PythonMatcher which just wraps a python match
object.
Test Plan: Covered by existing matcher tests in cstore-treemanifest.py
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663354
Signature: t1:4663354:1488914384:2c33c7e0e7f2eade0786b6ff41503317989fd1e5
Summary:
A future patch will want to import just the cstore py headers into
py-ctreemanifest, so let's move those out to their own header file.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4571320
Signature: t1:4571320:1487774077:64b4c5053b223f075a1c9f73ddcd92bd73b53e23
Summary:
To begin the process of weaning treemanifest off of Python and on to the new
cstore, let's introduce a base Store class that implements a simple get()
function. UnionDatapackStore already implements this. Future patches will make a
Python version that implements this interface as well, so we can swap between
them easily.
Test Plan: Built and ran the tests.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569870
Signature: t1:4569870:1487846232:aa816311ec37a520a2405b68e3c02ad744613049
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
Summary:
As part of unifying our storage layer into a single library, let's move
py-cdatapack into the new cstore directory. Future patches will move
ctreemanifest and the upcoming datapackstore into here as well.
py-cdatapack.h required some reordering since it seems forward declarations work
a little differently between C and C++. There were no code changes though,
except one int->size_t fix.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4581320
Signature: t1:4581320:1487788968:e8a34c7a03a16db282214c7dd476b749b92a1bfa
Summary:
This adds a new C++ UnionDatapackStore implementation that only has the
getmissing() function at the moment.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556052
Signature: t1:4556052:1487161607:664752df19c63c06819ee2af5b4c436f1b76609d
Summary:
The remaining python parts of the store are a perf bottleneck when accessing
hundreds of thousands of pack file entries (like in treemanifest). Let's
implement them in C++.
This first patch just add the basic boiler plate, and implements a single
function getdeltachain(), with a test. Future patches will add more
functionality and other parts of the store.
Since cstore depends on cdatapack and ctreemanifest (the pythonutils.h part for
now), we need to tweak our setup.py to enforce a certain build order too.
Test Plan: Added a test, yo
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4547929
Signature: t1:4547929:1487181318:21c146cf370d26cb97efe6a883868b85b4e32f49
Summary:
All of the test-check*.t tests cd out of the test repository and into the main
normal fb-hgext repository, and attempt to run hg commands there to examine the
repository contents. Unfortunately they were doing so still using the HGRCPATH
settings configured for the test repository. In practice this causes the tests
in many situations, because extensions enabled in the system configuration and
required for the main repository may not have been enabled.
Test Plan:
Ran "arc unit" in a repository with sqldirstate enabled and confirmed the tests
pass now instead of bombing out with screenfuls of error messages.
Reviewers: #sourcecontrol, stash, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, net-systems-diffs@, yogeshwer, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4427670
Signature: t1:4427670:1484733701:f6c859c29d34eb33b1280e798ab3398a03a3a461
Summary:
"arc lint" runs the lint script for each file in parallel:
>>> [5, pid=27257] <exec> $ scripts/lint.py 'scripts/lint.py'
>>> [6, pid=27257] <exec> $ scripts/lint.py 'scripts/unit.py'
>>> [7, pid=27257] <exec> $ scripts/lint.py 'tests/test-check-code-hg.t'
>>> [8, pid=27257] <exec> $ scripts/lint.py 'tests/test-check-pyflakes-hg.t'
That looks fine but `lint.py` runs `test-check*.t` which will check all
files. This is a typical `N^2` behavior and needs to be fixed.
This diff changes the linter and test code so the test only checks the
single selected file.
Test Plan:
Run `arc lint --trace` and check `lint.py` time usage. It went down from
10 seconds to 2 seconds for this diff.
Reviewers: #sourcecontrol, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4051080
Signature: t1:4051080:1476985355:45845c6eb0e66cbfa000e61b7b496f2d00aeb042
Summary:
buffer.h gained the ability to deal with non-char-sized buffers when I built cdatapack. We need to update the callers in ctreemanifest to be aware of this. Most of this is done with macro magic.
Some functionality was dropped from cdatapack's buffer.h (macro definitions to deal with paths). Those are moved to path_buffer.h
Test Plan:
make local && clion build.
pass cfastmanifest unit tests.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3780767
Signature: t1:3780767:1472255278:40a19edfd171df5804e9cdfa4444d5c6386f00e8
Test Plan: run tree_dump on a fastmanifest file. did some spot checks against the text manifest with the same sha (`hg debugdata -m`)
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3585843
Tasks: 12119301
Signature: t1:3585843:1469050470:70729d5601f935f2de80f89a7ad83590bcb8dd00
Summary:
It's a common mistake that our tests require foreign extensions (namely evolve
and remotenames) without checking them first.
This diff adds checks to catch these mistakes, adds missing checks, and unifies
our checking logic using `require-ext.sh`, which is aware of `hgext3rd` and
prints skip message.
This affects `arc lint` so hopefully our new testing code would be free of this
kind of mistakes.
Test Plan: `arc lint` would catch errors
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3550977
Signature: t1:3550977:1468455857:e849dfd9e3cbc446cc6e6c662050ee88a3366e6c
Summary:
Be a better citizen under system python path.
Fix all tests issues and change setup.py to use glob pattern to include
all extensions.
Test Plan:
Run tests and `make local`.
Also build and install the package and run `hg sl` in major repos.
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3534311
Signature: t1:3534311:1468275426:fe122646c8bd6c541e1889e73e9df28f86747ff2
Summary: This allows us to use fastmanifest as a directory to drop in the python module.
Test Plan: compiles, passes existing tests.
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3351021
Signature: t1:3351021:1464284417:6cbcde514ab1fd7b5caa6c83cb5577f3502dbc58
Summary:
API to write trees to disk and read trees from disk.
Some interesting things to keep in mind:
* To keep valgrind happy, we have to clear all the memory we allocate but never initialize. That can be padding between fields in structs, fields that don't make sense to initialize, the extra byte in checksums that doesn't get used normally, or portions of bitfields that are never used. All this logic is kept compartmentalized in `initialize_unused_bytes(..)`. Normal runtime does not invoke this code, though I suspect the cost is probably not significant.
* A significant chunk of the write path is made up of the `CHECKED_WRITE_XXX` macros. They're calls to ensure that writes succeed. If not, they jump to a fixed exit point.
* The write path is not very optimized. We can use a bottom-up traversal, similar to tree_convert, to make the write less costly. However, our design doesn't require a fast write, so tentatively, I'm going to reuse `tree_copy(..)` to compact a tree.
Depends on D3255048
Test Plan: pass unit tests!
Reviewers: lcharignon, wez
Reviewed By: wez
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3255656
Tasks: 10589048
Signature: t1:3255656:1462342423:c28d32c610b2351614d7648c03f35f931370a770
