Summary: `initialize()` does it already.
Test Plan: run unit tests.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3869524
Signature: t1:3869524:1474225262:57ccfd92a4d02bc92674007bcafaeafedef413a0
Summary: D3842603 broke reads from datapacks because `.initialize()` creates a blank manifest and attaches it to the manifest entry. When we subsequently try to read the root, we don't properly actually resolve it to the manifest in the datapack. Instead, we use that blank manifest which is wrong.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py --config remotefilelog.fastdatapack=False testtree --kind flat,ctree,fast --test fulliter --build "master~50000::master" --revs 'master + master~5000'
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3862319
Signature: t1:3862319:1474225268:f01ef876d569bc09c1e0ace71492bbaae017404e
Summary: Any modifying operation sholud set the `invalidate_checksum` flag, and as we unwind the stack, we clear the nodes if the flag is set.
Test Plan: not yet tested
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3842843
Tasks: 13268688
Signature: t1:3842843:1473709799:89a82ffade1e56fe64ccae50abecd3de4b46f6bd
Summary:
This adds a Manifest function that can produce a serialized version of the
manifest that is appropriate for writing into a store.
Test Plan:
Used it as part of a future diff that serialized manifests to a pack
file.
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Subscribers: ttung, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3837714
Signature: t1:3837714:1473454260:1c5864730744a560cbad4fc4ab5263ef4447f7dc
Summary:
This implements the copy function, which will be used to create a copy of a
parent manifest when beginning to create a new manifest.
Test Plan:
Used in conjunction with future diffs to serialize a new treemanifest
into a pack file.
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Subscribers: ttung, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3837415
Signature: t1:3837415:1473453896:494fc856256764f14741c65258f354e08383f1f7
Summary:
Removes are treated as set to None.
The API is not exactly correct, as manifestdict needs to implement size/getitem/setitem, but we can refactor in a subsequent diff.
Test Plan: pass unit test
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3836197
Signature: t1:3836197:1473363870:21a447e4bc17148193fde7ed4887927089e54930
Summary: Uses `walk()` to locate the element. Callback is super simple again.
Test Plan: used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3836142
Signature: t1:3836142:1473363781:0384d2c0b7154d3acbbab436c12bbdb55a5a63a7
Summary:
Python binding is pretty straightforward. The test exercises a few simple paths: set, update, and conflict detection.
Not yet done: tests to set a deeply nested directory structure.
Test Plan: PASSED THE SIMPLE TESTS. YES!
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832682
Signature: t1:3832682:1473363415:60240c6bbbcec82184b4588b83bbda750e8b77f1
Summary: This uses the `treemanifest.find(..)` interface to locate the desired tree entry and add/update a leaf node (a.k.a. file). The `set()` method is mostly setup and teardown logic for find, and the little unique code that needs to exist is in `set_callback()`
Test Plan: used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: trunkagent, durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832664
Signature: t1:3832664:1473459754:a258f42791b4c4d561abad043d8ad21ddb48fbed
Summary: This is needed when we're updating a node that already exists.
Test Plan: used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832631
Signature: t1:3832631:1473362920:8a39f02ced91cc9328c57e416d7dfc9a704b0cc6
Summary: Internally, mercurial transacts in bin nodes.
Test Plan: used in later diff
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832614
Signature: t1:3832614:1473359939:033cb1d944680287fb6563523c7782a80118b9e2
Summary:
flags should be == NULL if it's not present, not that it points to NULL.
Since the flags pointer isn't always set, we have to use the filename pointer to set the separator field ('\n').
Test Plan: used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832555
Signature: t1:3832555:1473359723:9f05dd353f0c1866232214cd0d489d1fb4573f51
Summary:
This implements the walk function, which is almost identical to iterkeys, except
it filters the result using a provided matcher.
Test Plan:
As before, not sure how to execute this from in mercurial, but it's
pretty small so we'll test it with the full mercurial test suite once we can do
edits.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3775631
Summary:
This implements the ability to iterate of the entries of the manifest. We reuse
the key iter, but add a boolean option for returning the full (path, node, flag)
tuple instead of just the path.
Test Plan:
I'm not sure of how to execute this via an hg command, so for now I'm
just going to
doitlive
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3775619
Summary: This is needed when we execute a `set`, which comes in with a binary node, but is stored as a hex string.
Test Plan: make local; used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3831923
Signature: t1:3831923:1473292766:bd4c1fdde356a8de5b4b74328e3805b7e1eae752
Summary:
1. remove unused `fetcher` variables.
2. std::string's second arg is a size_t, not a Py_ssize_t.
Test Plan: make local
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3830569
Signature: t1:3830569:1473292747:823b5c06490c45c0bfcc59373e7935af0c1b3630
Summary: Since a lot of accesses are coming from within the treemanifest class, I renamed the cached variable as rootManifest_DO_NOT_ACCESS_DIRECTLY to make people think twice before accessing it.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3823297
Signature: t1:3823297:1473213565:54657eeecff35549c71dc0af91c71cca9d426ac1
Summary:
The root node is now optional. If it's not specified, an empty tree is constructed.
Added an unit test to cover this code path.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py --config remotefilelog.fastdatapack=True testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"` still runs.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803569
Signature: t1:3803569:1472759620:02c1f723fe3b7f4cdca9da6399fc2296c6d5a022
Summary: This allows us to set the node hash when we add the child.
Test Plan: make local
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803564
Signature: t1:3803564:1472759281:a5e413ba7cf12319c4460da213ad8be7a0eed673
Summary: This removes a bunch of extra arguments we were passing in because the treemanifest knows about it already. Also inlined the entire resolve-the-root-manifest operation into a separate method.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py --config remotefilelog.fastdatapack=True testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3800793
Signature: t1:3800793:1472758998:c4617894c947f84268a3fa1d0cdde03793889640
Summary: We don't actually care about the store. We want to invoke the fetcher. At some future point, we could replace the fetcher with something that goes to the network or whatever, or unit tests can point to something special.
Test Plan: make local && `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py --config remotefilelog.fastdatapack=True testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3800789
Signature: t1:3800789:1472758892:67cee86424da4b7b903f2495e31bdd03d793b21a
Summary: treemanifest_find is now a generic method to traverse a tree (like tree_path in cfastmanifest). It manages the tree by adding and removing intermediate nodes (if requested). Unlike _treemanifest_find, it uses lower-level APIs such as findChild and addChild, which uses std::list::iterator to avoid duplication of work. Previously, we would need to traverse the entire list of children to find a child, and then if it's not found, traverse the entire list again to properly locate it. Now we just find the exact location where the child is or should be, and then save that.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/remotefilelog:~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/remotefilelog/tests/perftest.py testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mjpieters, durham, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3772947
Signature: t1:3772947:1472758787:8e8597c30f2cf1512ac7195bfa44ed62c201deb2
There was a dumb bug where we attempted to use the matcher if it was NULL,
instead of only using the matcher when it was not NULL. We just need to remove
the bad !.
Summary:
It turns out PyArg_ParseTuple doesn't create new refs, so if we put any
arguments in PythonObj wrappers, they will be incorrectly decremented during
destruction. Let's increment the ref before putting any args in PythonObj.
Test Plan:
N/A, I just eyeballed this. I did look at all other uses of
PyArg_ParseTuple to verify no other cases had this issue.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3766116
Summary:
With this implemented, it's now possible to run 'hg diff' and have it use
exclusively our treemanifest implementation.
Test Plan:
Ran 'hg diff -r master^ -r master' with some future patches and
verified the result was correct (it originally failed entirely because
__contains__ was not implemented).
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3766114
Summary: The filesnotin function is necessary for hg diff, so let's implement it.
Test Plan:
Ran 'hg diff -r master^ -r master' with some future patches and
verified the result was correct (it originally failed entirely because
filesnotin was not implemented).
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3766108
Summary:
Turns out that manifest.diff accepts kwargs, so we need to support that
signature. We'll come back and actually implement clean=True support later.
Test Plan:
Ran 'hg diff -r master^ -r master' with some future patches and
verified the result was correct (it originally failed entirely because diff had
the wrong signature).
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Subscribers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3766104
Signature: t1:3766104:1472156875:0ce2a16286b1e4fd6b0f2260249e3318270d8814
Summary:
This implements the matches function of treemanifest. These are necessary for hg
diff to work.
Test Plan:
Ran 'hg diff -r master^ -r master' with some future patches and
verified the result was correct (it originally failed entirely because matches
was not implemented).
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3766101
Summary:
File iteration is becoming a common pattern, so let's move py_fileiter creation to a
common area. This also move the fileiter initialization logic into the
constructor so it's not duplicated anymore either.
Test Plan:
Ran 'hg files' and 'hg manifest' with future patches that allowed hg
to use the treemanifest and verified they still worked.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3775597
Summary:
This moves the file iterator to a separate function so that it may be reused in
other areas. It also further separates the iterator from the python specific
logic.
Two notable changes to the logic:
1. It now returns the node and flag as well as the path.
2. Instead of copying iter->path straight into a python string for returning,
the caller provides a path, node, and flag character array as arguments and the
result is copied into that array. This results in a little more string copying
during iteration, but since the character array is stack allocated it the actual
performance impact is neglible.
Test Plan:
Ran the perf test, verified fulliter perf didn't change (2.2s before
and after). Also used pdb to verify the results were valid and expected paths.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3760071
Summary:
This implements the flags api. After this, hg manifest --debug can work with
treemanifest.
Test Plan:
In a future patch, I was able to run 'hg manifests' on a commit and
have it return the manifest contents by reading the treemanifest.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3755343
Summary: This implements treemanifest.__getitem__ in the C api.
Test Plan:
In a future patch, I was able to run 'hg manifests' on a commit and
have it return the manifest contents by reading the treemanifest.
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3755338
Signature: t1:3755338:1471930178:933e1d56f98c6e615624e2501a01a6ae925c2f8d
Summary: After a recent refactoring these are now unused.
Test Plan: make local
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3756194
Signature: t1:3756194:1471929972:bdace32b744bfaaf03228f13de0fb6ffec240dab
Summary:
If we don't include Python.hg before some of the standard library (like list),
it causes a build error on some machines ('error: "_POSIX_C_SOURCE" redefined').
So let's move all Python.h to the front.
Also add a missing stdexcept include that's needed for the use of
std::logic_error
Test Plan: make local on centos6
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3756191
Signature: t1:3756191:1471929906:8afc09ee74b5ab5512fc87fadbed311cf15cb768
Summary: When a ManifestEntry is destroyed, we reap the heap-allocated resources it holds. However, this (erroneously) happens when we create a ManifestEntry and add it to the list of children. What we really need is a move constructor from C++11. The workaround is to initialize a blank ManifestEntry, add it to the proper location, and then populate it afterwards.
Test Plan: addChild (in a later diff) no longer tramples over memory it no longer owns.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3772915
Signature: t1:3772915:1472174144:a4baac5cb5f6e01a38042c5c6cd92570c8f8e100
Summary: This gives us a bit of flexibility when we call this method, as references cannot be reassigned.
Test Plan: compiles
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3772844
Signature: t1:3772844:1472174021:565d7f25a27c2fd78ce14b86b5bc499c74d47976
Test Plan: haven't used this yet, but it makes a whole lot of sense.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3770366
Signature: t1:3770366:1472173969:b83e10e27a63b863a3566d2df7cc861417b8208e
Summary:
1. `children()`, which returns the number of children the manifest has.
2. `removeChild()`, which removes the entry that the iterator references.
Test Plan: used in later diffs
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3770363
Signature: t1:3770363:1472173823:737043b4613a669b402dcfd7921ac55278c3ece9
Summary: The strings are not necessarily null-terminated and length needs to be considered.
Test Plan: used in later diff to find a path.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3770358
Signature: t1:3770358:1472173786:9a2f681c90476aafd481c301ff65ac8b199214ec
