Summary: On some platforms, `ssize_t` is defined in `sys/types.h`.
Test Plan:
make local
Reviewers: #mercurial, simonfar, ttung
Reviewed By: ttung
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3334837
Signature: t1:3334837:1464032277:e9c37f1ea280621ab383e7c5aa2a2e67fea3393e
Summary:
This patch reorganizes the import statements in fastmanifest.py and
fixes 3 issues flagged by pyflakes
Test Plan: Test pass
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3335411
Summary:
This adds a new strategy for the cache limit that depends on the free
space availabe in the system. For system with more than 100GB of free space
we allocate 5GB for the cache. With more than 20 GB of free space we allocate
2GB of free space and otherwise 10% of the free space at most.
Test Plan: Add a new test
Reviewers: ttung, durham, rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3330894
Summary:
This adds the first, basic implementation of size limitation for
the fastmanifest cache
Test Plan: Add a few tests that cover edge cases
Reviewers: ttung, simonfar, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3326149
Summary:
This patch makes it possible to cache fastmanifest asynchronously. I added
a test an also tested with fbsource:
hg debugcachemanifest --debug --config extensions.fastmanifest=/data/users/lcharignon/facebook-hg-rpms/fb-hgext/fastmanifest.py -R ~/fbsource --all
hg debugcachemanifest --debug --config extensions.fastmanifest=/data/users/lcharignon/facebook-hg-rpms/fb-hgext/fastmanifest.py -R ~/fbsource --list
This is a tricky code, and I think @quark would be best to review it.
Test Plan: Added a new test
Reviewers: quark, durham, ttung
Subscribers: quark
Differential Revision: https://phabricator.intern.facebook.com/D3323540
Summary: This is a refactoring of the _cachemanifest function
Test Plan: test output does not change
Reviewers: ttung, simonfar
Differential Revision: https://phabricator.intern.facebook.com/D3330501
Summary: When listclean == True and the two nodes are the same, then we are supposed to return None. We did not implement this behavior correctly.
Test Plan: passed test-status.t test-status-rev.t test-fileset.t test-add.t
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3327261
Tasks: 11144982
Signature: t1:3327261:1463844770:7714213b927cc4f55466491b8b317dc0460136af
Summary:
We use the manifestnode from the changelog and avoid having to reach the manifest
completely to check if it is already in cache.
This goes with a refactoring of the contains method. Before this patch, the
contains method was misleading because an entry could be contained in the cache
but not show up when you iterate over the cache. This happened because contains
was operating on nodes and iterating showed filesnames (so keys + prefix). To
make it clear that contains operates on nodes, we change its name to containsnode
We also rename key to hexnode to make it clearer that the keys are hexnodes.
Test Plan: Existing tests pass. We also add a new test to show that the fast
pass is hit.
Reviewers: durham, ttung
Differential Revision: https://phabricator.intern.facebook.com/D3325335
Summary: This will be useful for implementing garbage collection later.
Test Plan: Add a new test that exercises the new function
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3322426
Summary:
This patch changes fastmanifest to add automatic caching of relevant
manifest when bookmarks change or if the parent of the working copy changes.
Test Plan: Added a test
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3318775
Summary:
Touching on access allows us to order the entries by access time and
implement a LRU caching strategy, see D3326149.
Test Plan: Tests pass
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3328205
Summary:
Before this patch, if a second process was to remove a cached entries during
a pruneall operation from another process, we could crash.
Test Plan: Tests pass
Reviewers: ttung, rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3326100
Summary:
This will makes implementation of other things easier like garbage
collection.
Test Plan:
Existing test pass and the existing code exercise the new code path
so we should be good.
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3322423
Summary:
This is typically what we do with other extensions, either the top or the
bottom of the file to increase legibility.
Test Plan: Tests pass
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3318766
Summary:
We move wrapfilecache from reflog to extutil to be able to reuse it
in fastmanifest
Test Plan: Tests pass
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3318762
Summary:
Adding a pruneall method to prune all cache manifest. This will be
useful to test the caching logic without having to run shell commands to remove
manifests.
Test Plan: Add a new test
Reviewers: ttung, durham
Differential Revision: https://phabricator.intern.facebook.com/D3314158
Summary: Sometimes the dirstate is written without calling dirstate.write.
Test Plan:
$ ../../hg-crew/tests/run-tests.py test-reflog.t
.
# Ran 1 tests, 0 skipped, 0 warned, 0 failed.
Reviewers: #mercurial, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3324632
Summary: If it's malformed, set self->tree to NULL.
Test Plan: half of the fix for test-verify.t
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3321562
Tasks: 11145104
Signature: t1:3321562:1463668231:e674aa9f343ece60cdf8b579a1a9d32c228b74a3
Summary:
We just materialize an empty fastmanifest and return it.
Also, not emptymanifest => True. Did I mention how much I hate implicit casts?
Test Plan: pass test-subrepo.t
Reviewers: durham, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3320669
Tasks: 11320498
Signature: t1:3320669:1463668266:0e7fd7d7123d37c7e71215f0632314ea804f1d43
Summary: This is the behavior of lazymanifest (c implementation) and _lazymanifest (python implementation).
Test Plan: pass all but one of the subrepo tests
Reviewers: durham, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3320470
Tasks: 11319460
Signature: t1:3320470:1463668278:01968bf3a95070a695fbac8ffec675c2ff616434
Summary: We want to always present a fastmanifest, so we always return true for `_incache()`. `_cachedmanifest()` is modified to extract the flat manifest when debugfastmanifest is set, and convert the extracted manifest.
Test Plan: down to 45 failures with this and the `filtercopy()` fixes. #10644559
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3320099
Tasks: 11278537
Signature: t1:3320099:1463668296:3f0e6b37524d8a74f531bc418a5594e13994d2b2
Summary:
Implement a copy mechanism that filters along the way. For leaf nodes, it's pretty straightforward.
For implicit nodes, instead of allocating full-sized nodes on the arena, we allocate a temporary node, write the children there. Once we finish, we determine whether or not we still need the implicit node (because we may end up with 0 children). If it is, we create the permanent node and copy the entries over. If a subtree is not fully copied over, we invalidate the checksum on the implicit node.
Test Plan: wrote a unit test that prunes the entire tree. wrote a unit test that prunes a subtree. verifies that the nodes and their checksums.
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3318895
Tasks: 10644559
Signature: t1:3318895:1463668333:e2bfee36ab4b324dc3a93dedbe9bb36459f2194b
Summary:
When people combine --root-relative with a file pattern, they would probably
expect the output to be relative to the root -- but it's not! In this case, it
would fall back to the default file pattern behavior, which is cwd-relative.
Instead of confusing the user with apprarently errorneous output, let's abort
explicitly and provide a helptful hint.
Test Plan: added a new test, existing tests still pass
Reviewers: #mercurial, simpkins, ttung, quark
Reviewed By: quark
Subscribers: quark, net-systems-diffs@, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3319366
Signature: t1:3319366:1463610688:3f129c97f68f43ac85d2b31b55fac5c859e85c04
Summary: I want to reuse these in other tests, so I don't want to imply they are only good for this test.
Test Plan: run unit tests
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3318288
Signature: t1:3318288:1463603748:3dc18243f2150a71485a4be7c452542db1790c48
Summary: It was previously being masked by other includes prior to buffer.h
Test Plan: later diff compiles.
Reviewers: wez, lcharignon, mitrandir
Reviewed By: mitrandir
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3317224
Signature: t1:3317224:1463601688:7b336b4ba5abb4ab699fcaa604e178861b2f2959
Summary:
Rather than call `PATH_EXPAND_TO_FIT` and then appending the path component, just do it in one step.
Depends on D3313016.
Test Plan: pass unit tests
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3313078
Signature: t1:3313078:1463603770:a62a5b0a1bc01ee530765176b513a46c7a1de386
Summary:
# Remove `CONVERT_BUFFER_APPEND`, which is never used.
# Switch path buffer expansion to use the ones defined in buffer.h
Test Plan: pass unit tests
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3313016
Signature: t1:3313016:1463603786:b53db5476fd7003418a3b74cbb9ea206647694e7
Summary: Constructing paths is a common idiom. Let's not have to redefine these macros everywhere.
Test Plan: run unit tests.
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3313010
Signature: t1:3313010:1463603801:e248cd682c0af606a81efd8ea3aba8f607bb9fa2
Summary: It should just return False if the key is not a string, like lazymanifest.
Test Plan: pass test-revlog-ancestry.py, where it previously failed.
Reviewers: durham, wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3308314
Tasks: 11233878
Signature: t1:3308314:1463507845:a2cc81833022770f0288dd3c0d830b6aff3b9ff9
Summary:
* Handle the scenario where ui is not set (like in test-fastmanifest.py).
* Add in the same logic for `fastmanifestcache` as `hybridmanifest`
Test Plan: run test-fastmanifest.py test-fastmanifest.t
Reviewers: lcharignon, durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3311487
Summary:
This solves two problems we're seeing with our current approach.
1) If we have a file `f` in directory `t` (thus `t/f`) and a file called `t.txt`, we'll order the nodes as `t/f`, then `t.txt`. This is not the order lazymanifest expects, as `t.txt` *precedes* `t/f` in lexicographical order. This is because we order `t` before `t.txt`. However, if we have `t/` vs `t.txt`, `t.txt` will come first.
2) We eliminate the issue where mercurial sometimes adds a file with the same name as a directory. This will no longer happen because the directory will have the trailing slash.
Test Plan: pass all existing fastmanifest unit tests
Reviewers: lcharignon, wez
Reviewed By: wez
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3297330
Tasks: 11234430
Signature: t1:3297330:1463163665:41ba1b3389d35fd9417f378940e59e38e2edbc85
Summary:
When fastmanifest.debugfastmanifest is set to True, we always return a fastmanifest. If the cache is empty, we will convert a flat manifest into a fastmanifest and use that for all operations.
Same goal as D3247484.
Test Plan: pass some unit tests in the mercurial suite. need to investigate the remaining issues.
Reviewers: #mercurial, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3295434
Signature: t1:3295434:1463439017:0a6d20fa23fe409094e35a46eb195495455b8602
Summary: Unlike manifestdict, we don't accept a 0-argument constructor. Instead, we build up a cfastmanifest and then wrap a fastmanifestdict around it.
Test Plan: pass more unit tests
Reviewers: #mercurial, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3295344
Signature: t1:3295344:1463438969:8e66c79aac0d5914549852b233d07b1ab3bf11ff
Summary: Since there's no flatmanifestcache, we can collapse the two classes.
Test Plan: `python ~/work/mercurial/facebook-hg-rpms/hg-crew/tests/run-tests.py test-fastmanifest.t test-fastmanifest.py`
Reviewers: #mercurial, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3294646
Signature: t1:3294646:1463438882:6d9c3eaa7ac739f27672d4557f6031abbd1033a7
Summary: If it is 0-argument, then just attach an empty tree.
Test Plan:
```
[andromeda]:~/work/mercurial/facebook-hg-rpms/fb-hgext:24160e2> ipython-2.7
Python 2.7.11 (default, Mar 1 2016, 18:40:10)
Type "copyright", "credits" or "license" for more information.
IPython 2.2.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import fastmanifest_wrapper
In [2]: a = fastmanifest_wrapper.fastManifest()
In [3]: a['foo'] = ('c'*20, "")
In [4]: a.text()
Out[4]: 'foo\x006363636363636363636363636363636363636363\n'
In [5]: del a['foo']
In [6]: a.text()
Out[6]: ''
```
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3289430
Tasks: 11145013
Signature: t1:3289430:1463063731:f99f4690c9c8e57b2845010a62ceaea16d3643cc
Summary: When converting from a flat manifest, we inappropriately set the type of the root to IMPLICIT. This meant that when we removed all the nodes in the tree, we would end up removing the root node, as it looked like just any other directory.
Test Plan: wrote a unit test to hit this scenario. it failed without the fix. passed with the fix.
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3289249
Tasks: 11145050
Signature: t1:3289249:1463063771:0e5d4f1bca296b1009c9ec6461f7aab85546fc8f
Summary: The assertion is incorrect, as we start with the root node. Therefore, it will fire during unit testing.
Test Plan: wrote a unit test that triggers it
Reviewers: wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3289178
Tasks: 11145050
Signature: t1:3289178:1463063785:e4cc9af68154baa8004fd384c1153bcc44f814fd
Summary:
* `_manifest()` should attempt to retrieve a fastmanifest if possible.
* `self.incache` represents a tristate indicating True if the fastmanifest is available, False if the fastmanifest is not, and None if we haven't tried to determine its availablility.
* clarified the debug messages a bit
* re-introduce the test removed in D3277498 (since it actually can work now).
Test Plan: pass unit tests!
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3288967
Signature: t1:3288967:1463416544:416f28ecdc5e6a26545f3d4215fc8baa9222af7c
Summary: `fastmanifestdict`'s constructor takes a fastmanifest. there is no default constructor.
Test Plan: used in later diffs
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D3288939
Signature: t1:3288939:1463416036:a6475f5693d48175c81262b412ebdf0525b60fb2
Summary: `manifest.node` is a bin node value. To get the hex version, we need to call `revlog.hex(..)` on it.
Test Plan: used in later diff.
Reviewers: durham, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D3288910
Signature: t1:3288910:1463416018:2c6e09a6c80b2e0a04ddd712d7b7b506c334afb1
Summary:
# `decLeftReference` and `decRightReference` in `fastmanifest_diff_callback` not always initialized.
# in the success condition, `py_left` and `py_right` could be decref'ed twice in `fastmanifest_diff_callback`
# called `PY_CLEAR(es)` twice (although this is harmless).
Test Plan: ran hg diff without crash
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3285513
Signature: t1:3285513:1463416642:eca5227ab56edb666ff4cf615f60730429f34924
Summary: Since self.node is passed to `copy()`, we should pass the unadulterated node. Otherwise, we end up calling `revlog.hex(..)` on a hex hash.
Test Plan: pass tests/test-fastmanifest.py
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3284150
Signature: t1:3284150:1463067635:938e45fcd706844f86ed5a84b6dd0462a80adcbe
Summary: since it's a singleton, we can have one instance of the code for retrieving the instance.
Test Plan: run tests/test-fastmanifest.t
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3284113
Signature: t1:3284113:1463416610:207c7e9954cc0b830e77f2fecc0a65acfcfd1096
Summary:
Right now, neither `copy()` nor `matches(..)` behave as one might expect. If the hybridmanifest is wrapping a flat manifest, then calling `copy()` or `matches(..)` results in an exception as it explicitly calls for a flatmanifest.
These methods should call the method (`copy` or `matches` on whatever manifest the hybridmanifest is wrapping), and wrap the result in a hybridmanifest. To support this change, we modify `hybridmanifest()` to accept three possible data sources:
# A flat manifest
# A fast manifest
# A method to return a flat manifest
Test Plan: tests/test-fastmanifest.t passes
Reviewers: durham, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D3284044
Tasks: 10589051
Signature: t1:3284044:1463063458:c0636e0db1ee726e7db55cc62b7dae419049b584
