Summary:
Clean up the include guards to be more consistent and unique.
Some files used include guards like "KEY_H" and "STORE_H" which were not very
unique, and are more likely to collide with definitions provided by header
files from other projects. Some of the py-*.h files were missing include
guards altogether.
This corresponds to Facebook diff D5588670.
Test Plan: Confirmed the code still builds and passes tests.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D508
Summary:
Update the copyright headers in most of the C/C++ code consistently use the
GPLv2 copyright message. This allows these files to pass Facebook's internal
C/C++ linters.
Some of the files in fbcode/scm/hgext/cstore/ appear to have actually been
copied from the hg-crew repository, and were not originally authored by
Facebook. I have not modified the copyright statements in these files:
- cstore/bitmanipulation.h
- cstore/compat.h
- cstore/mpatch.h
- cstore/mpatch.c
I also have not modified any of the cfastmanifest code.
This corresponds to Facebook diff D5588677.
Test Plan:
Confirmed that Facebook's C++ linters no longer complain about the copyright
messages.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D507
Summary:
Convert the C and C++ files in cdatapack, clib, cstore, and ctreemanifest
to always include files from the root of fb-hgext. This simplifies the build
process by no long requiring a variety of separate include directories to be
specified on the compiler command line.
This will also make it easier to re-use these extensions in other projects
with different build systems.
This corresponds to the Facebook diff D5588676.
Test Plan: Confirmed that "make local" succeeds from a clean build.
Reviewers: #fbhgext, quark
Reviewed By: #fbhgext, quark
Differential Revision: https://phab.mercurial-scm.org/D505
Summary:
This patch removes all `#include "../` lines and use the shortest possible
include path.
Test Plan: `make clean build`
Reviewers: durham, #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D5113672
Signature: t1:5113672:1495565454:961fb6f2f57a81a95013e0b8f67b2917c2e4523e
Summary:
this simplifies the code and resolves a `new[]/delete`
allocator/deallocator mismatch that I tripped over in D4929693
I think this is pretty straight forward. The diff is made a little
larger than I would have liked because there were some missing `const`
qualifiers that had to be propagated out to various callers.
Test Plan:
ran the tests:
```
11:23 $ rt
.................................s.........s.s.s......s..sss............s.....s...................................................................................................................
Skipped test-p4fastimport-import-incremental.t: missing feature: Perforce server and client
Skipped test-p4fastimport-criss-cross.t: missing feature: Perforce server and client
Skipped test-p4fastimport-case-insensitive-rename.t: missing feature: Perforce server and client
Skipped test-p4fastimport-import-modes.t: missing feature: Perforce server and client
Skipped test-p4fastimport-import.t: missing feature: Perforce server and client
Skipped test-p4fastimport-import-deletes.t: missing feature: Perforce server and client
Skipped test-p4fastimport-case-insensitivity.t: missing feature: Perforce server and client
Skipped test-p4fastimport-import-parallel.t: missing feature: Perforce server and client
Skipped test-infinitepush-backup-sql.t: missing getdb.sh
Skipped test-infinitepush-sql.t: missing getdb.sh
# Ran 184 tests, 10 skipped, 0 warned, 0 failed.
```
Reviewers: simpkins, durham
Reviewed By: durham
Subscribers: net-systems-diffs@fb.com, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D4930821
Signature: t1:4930821:1493160075:5ee1d452b394c4d6f347029b0ff3fd89377624c5
Summary:
Adds a walksubtrees api that returns a set of subtrees from the given
treemanifest. This will be used in a later patch to return just the subtrees
that are new in one treemanifest. For now it's only used by the unit tests
Test Plan: Added unit test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4901241
Signature: t1:4901241:1492645063:5b3617cd36f3205bef8e09c4d37dd132e0bece4e
Summary:
A future patch will want to use SubtreeIterator for comparing two arbitrary
trees. To do this, we need to refactor out all the logic that actually computes
permanent hashes based on the other tree (since if I'm comparing against an
arbitrary tree, I don't want to use its hashes as my p1/p2 like I would if I'm
serializing the new parts of a tree).
This drastically simplifies the code as well, making it much easier to maintain.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4895795
Signature: t1:4895795:1492629678:18617385720b0e1ef5d1513085910a0d3968e29b
Summary:
A future patch will make SubtreeIterator more generic for diffing trees. To do
that we'll need to move the finalization logic out of SubtreeIterator. This
patch adds a new FinalizerIterator class that simply wraps SubtreeIterator. In
the next patch logic will move between the two classes.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4895794
Signature: t1:4895794:1492629514:9ee6e60c32e99041ce16f92c2923de6b4ad0b104
Summary:
A future patch will call ManifestEntry.update in more places, and it is
convienent to have the bin-to-hex conversion unified into one place.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4895788
Signature: t1:4895788:1492629387:7d64db0169bc47e5996666c4fb8fde639e83ba8b
Summary:
Previously the manifest node was only contained by the pointer to the manifest
(usually the container manifest). This made accessing the node awkward in a
bunch of places. Let's refactor it so the node is kept on the Manifest itself as
well, and it handles invalidation and computation of that node as it is editted.
Test Plan: Ran the tests
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4880076
Signature: t1:4880076:1492591856:f8cd065cac8d3f7cac3069ee9cb61881a24ee426
Summary:
While debugging I noticed newtreeiter_iternext doesn't handle c++ exceptions.
Let's handle it so the process doesn't crash entirely.
Test Plan:
Ran the tests. This kind of exception shouldn't be possible in
normal execution, so there's not really a test to write.
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4880068
Signature: t1:4880068:1492590284:87f986dc9fa3650524d8c71f69c28971e2552640
Summary:
The NewTreeIterator class was used for iterating over a new tree, when given
it's p1. Mainly for performing the write of a new tree. We want to make this a
more generic mechanism so we can do the diff of one tree with an arbitrary
nother (like when deciding what trees to send a client if they have a given base
node). Let's start by renaming the class.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4880048
Signature: t1:4880048:1492110564:bd51d35e4cfcca2d6b1d4cf81d78fdacf2b52f3d
Summary:
Upstream has added a matcher arg to manifest.filesnotin, so let's support it in
our native treemanifest implementation.
Test Plan: Added unit tests for filesnotin with and without matchers.
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677030
Signature: t1:4677030:1489076599:23ab67d85bb4fa689f05349253a4d2c4e7d496cc
Summary:
Upstream has added a matcher argument to the diff API which allows diff to avoid
traversing certain parts of the tree. This adds support for that to our native
treemanifest implementation.
Test Plan: Added tests for diff with matches
Reviewers: #mercurial, stash
Reviewed By: stash
Subscribers: stash, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4677023
Signature: t1:4677023:1489076627:dbcea209d300a68fa050f68c52b4fd9949b85302
Summary:
This is the last piece of removing the Python dependency from the core
treemanifest code. This replaces the old PythonObj diff dictionary with a new
DiffResult class that has a PythonDiffResult implementation.
Test Plan:
Ran the test suite, including the unit tests that explicitly cover
treemanifest diff from python.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663984
Signature: t1:4663984:1488889615:62f064924b0d6b45dfa7e490d19418060c374f40
Summary:
As part of breaking the native cstore implementation away from Python, let's
create a Matcher class that can be used to perform path match testing. Initially
the only implementation is the PythonMatcher which just wraps a python match
object.
Test Plan: Covered by existing matcher tests in cstore-treemanifest.py
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4663354
Signature: t1:4663354:1488914384:2c33c7e0e7f2eade0786b6ff41503317989fd1e5
Summary:
The upstream matcher has a 'visitdir' function which indicates whether a walk
should traverse into a directory. Let's support this in treemanifest so we can
skip a ton of lookups for parts of the tree that aren't relevant to the match.
Test Plan:
Used gdb to verify commands like 'hg diff -r .^ -r .
--include foo' only traversed into directory foo, and not directory bar. I'm not
sure how to write a test for this since it doesn't actually affect user facing
output.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4648252
Signature: t1:4648252:1488882860:b5e1b738fef2f63336c80cd7c0eb59c2fcf16d0f
Summary:
Previously the treemanifest code itself would create the text deltas when
writing a tree out. This meant we couldn't make the delta decision based on
other data, like if the p1 commit was in the same pack file.
This patch removes treemanifest.write() and moves all calls over to
treemanifest.finalize() which gives the python/pack layer control over delta
choices. A future patch will use this to ensure tree packs always contain
complete delta chains.
Test Plan: All tests pass
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4645942
Signature: t1:4645942:1488880851:d0c8c902e7e849072a53344630a9184b6d8e1e7f
Summary:
Previously, to serialize a tree we would call treemanifest.write() which would
take the stores and call the appropriate add() functions. This meant the
treemanifest code controlled when deltas were used, which makes it hard to
decide deltas based on external factors, like which pack file the parent is in.
As a first step to fixing this, we're adding a new finalize function that
returns a python iterator that allows iterating over the new parts of the tree.
In a future diff we will use this to do serialization and delta decisions at the
python/pack layer instead.
The same future diff will also cover this in tests, as it moves all calls to
treemanifest.write() to use treemanifest.finalize().
Test Plan: The next diff switches all write() calls and tests to finalize()
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4645922
Signature: t1:4645922:1488880722:574a37ba800f4b78faf8bad3979b447f4278cc55
Summary:
Previously the treemanifest auto-tree-creation logic only produced data packs
containing the actual contents of the tree blobs. This lost history information
which is important for our ability to efficiently repack the data files.
This patch creates history packs during pull as well. A future patch will also
create history packs for the local tree blob store.
Test Plan: Updated the tests to cover this
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4638865
Signature: t1:4638865:1488449992:48b60961b50b90b6d0e75a64af1f36fb29944e7a
Summary:
Previously treemanifest always called back to Python to fetch the tree content
for a given key. This patch let's us call directly in to the cstore without
going through Python, when possible.
Test Plan:
I manually tested this by modifying the treemanifest extension to
pass a uniondatapackstore to the ctreemanifest constructor, then used the
debugger to ensure this code path was correct. I still need to add a test
though.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4571442
Signature: t1:4571442:1487847314:0d578351354ee4da34b5032c01720bdedf0d185d
Summary:
In a future patch we will want to pass the uniondatapackstore around to other
objects who will contribute to the lifetime. Let's change it to a shared_ptr so
that becomes easy.
Let's also make the destructor virtual, so we can pass different types of stores
around and have them be destructed correctly.
Test Plan: Ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4603893
Signature: t1:4603893:1487847173:2fc3505032ea8c30cf9e0f76ac4e75d64513d87d
Summary:
The old code kept a PythonObj around inside the ManifestFetcher for fetching
manifest contents from the store. As part of moving the treemanifest code to use
the new native cstore API let's make the manifest code depend on a Store
abstraction and have one implementation be a PythonStore.
This removes almost all of the python dependencies from the core treemanifest
code, except some logic around running the python matcher during iteration and
writing directly to the python result dict during diff. We'll abstract those
away later.
Test Plan: Built and ran the tests
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4569944
Signature: t1:4569944:1487847102:d005b6484fd7de9335961b0bc4530505b25f961d
Summary:
Implements the getdeltachain function on the new UnionDatapackStore class.
This required some modifications to the DeltaChainIterator. Since the results of
the iterator may cross multiple different chains, we need to keep each chain
alive until the iterator is destructed, so we need to keep a reference to each
chain. We also had to remove the size() property from the iterator since the
fact that the chain spans chains means we don't know the size up front.
Test Plan: Adds a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4556458
Signature: t1:4556458:1487199872:07dffa3121acfbeb6d6993b518e6f4887122d4d5
Summary:
As part of unifying our native store data structures into a single library,
let's move the treemanifest (including the python extension) into py-cstore.
Test Plan:
Built and ran the tests. Verified there was no ctreemanifest.so
dependency in the built cstore.so by using 'ldd cstore.so' on Linux and 'otools
-L cstore.so' on OSX.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4602484
Signature: t1:4602484:1487842683:964cbb43b7cb20d0db699ef691fe7fcf6bccf2e8
