Summary:
We are going to log many different pieces of information in this extensions.
Therefore the `reporootlog` is not an appropriate name anymore.
Change it to `logginghelper`.
Test Plan:
Note that repo root is still logged instead of repo name. It will be changed in subsequent diffs
{P56637657}
Reviewers: rmcelroy, quark, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3840985
Signature: t1:3840985:1473410718:62d83f008f2973b7ed5266b0a5ebf67fe54b48c4
Summary: This is needed when we're updating a node that already exists.
Test Plan: used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832631
Signature: t1:3832631:1473362920:8a39f02ced91cc9328c57e416d7dfc9a704b0cc6
Summary: Internally, mercurial transacts in bin nodes.
Test Plan: used in later diff
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832614
Signature: t1:3832614:1473359939:033cb1d944680287fb6563523c7782a80118b9e2
Summary:
flags should be == NULL if it's not present, not that it points to NULL.
Since the flags pointer isn't always set, we have to use the filename pointer to set the separator field ('\n').
Test Plan: used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3832555
Signature: t1:3832555:1473359723:9f05dd353f0c1866232214cd0d489d1fb4573f51
Summary:
This implements the walk function, which is almost identical to iterkeys, except
it filters the result using a provided matcher.
Test Plan:
As before, not sure how to execute this from in mercurial, but it's
pretty small so we'll test it with the full mercurial test suite once we can do
edits.
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3775631
Summary:
This implements the ability to iterate of the entries of the manifest. We reuse
the key iter, but add a boolean option for returning the full (path, node, flag)
tuple instead of just the path.
Test Plan:
I'm not sure of how to execute this via an hg command, so for now I'm
just going to
doitlive
Reviewers: #fastmanifest
Differential Revision: https://phabricator.intern.facebook.com/D3775619
Summary: This is needed when we execute a `set`, which comes in with a binary node, but is stored as a hex string.
Test Plan: make local; used in later diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3831923
Signature: t1:3831923:1473292766:bd4c1fdde356a8de5b4b74328e3805b7e1eae752
Summary:
1. remove unused `fetcher` variables.
2. std::string's second arg is a size_t, not a Py_ssize_t.
Test Plan: make local
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3830569
Signature: t1:3830569:1473292747:823b5c06490c45c0bfcc59373e7935af0c1b3630
Summary: Since a lot of accesses are coming from within the treemanifest class, I renamed the cached variable as rootManifest_DO_NOT_ACCESS_DIRECTLY to make people think twice before accessing it.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3823297
Signature: t1:3823297:1473213565:54657eeecff35549c71dc0af91c71cca9d426ac1
Summary:
Use external bundles to store bundles.
Put and get binaries should be provided to upload/download bundles to/from external store (see `put_binary` and `get_binary` config options). Put binary should output a handle, that can be used with `get_binary` to retrieve bundle. `put_args` and `get_args` config options can be used to pass additional params to the binaries.
Test Plan:
Change `test-infinitepush-bundlestore.t` test file to point to external store and run it
Reviewers: rmcelroy, mitrandir, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803936
Tasks: 12479677
Signature: t1:3803936:1473123987:c34987ee459b2a66ee831574a32cdc8231043cb2
Summary: Add exceptions that will be thrown from bundle stores
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803928
Tasks: 12479677
Signature: t1:3803928:1473073448:32f6191ea0548c3d83cbaad100b6c8c48cbd8131
Summary:
The root node is now optional. If it's not specified, an empty tree is constructed.
Added an unit test to cover this code path.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py --config remotefilelog.fastdatapack=True testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"` still runs.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803569
Signature: t1:3803569:1472759620:02c1f723fe3b7f4cdca9da6399fc2296c6d5a022
Summary: This allows us to set the node hash when we add the child.
Test Plan: make local
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3803564
Signature: t1:3803564:1472759281:a5e413ba7cf12319c4460da213ad8be7a0eed673
Summary: This removes a bunch of extra arguments we were passing in because the treemanifest knows about it already. Also inlined the entire resolve-the-root-manifest operation into a separate method.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py --config remotefilelog.fastdatapack=True testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3800793
Signature: t1:3800793:1472758998:c4617894c947f84268a3fa1d0cdde03793889640
Summary: We don't actually care about the store. We want to invoke the fetcher. At some future point, we could replace the fetcher with something that goes to the network or whatever, or unit tests can point to something special.
Test Plan: make local && `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/fb-hgext/tests/perftest.py --config remotefilelog.fastdatapack=True testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3800789
Signature: t1:3800789:1472758892:67cee86424da4b7b903f2495e31bdd03d793b21a
Summary: treemanifest_find is now a generic method to traverse a tree (like tree_path in cfastmanifest). It manages the tree by adding and removing intermediate nodes (if requested). Unlike _treemanifest_find, it uses lower-level APIs such as findChild and addChild, which uses std::list::iterator to avoid duplication of work. Previously, we would need to traverse the entire list of children to find a child, and then if it's not found, traverse the entire list again to properly locate it. Now we just find the exact location where the child is or should be, and then save that.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/remotefilelog:~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/hg --config extensions.perftest=~/work/mercurial/facebook-hg-rpms/remotefilelog/tests/perftest.py testtree --kind flat,ctree,fast --test fulliter,diff,find --build "master~5::master"`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mjpieters, durham, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3772947
Signature: t1:3772947:1472758787:8e8597c30f2cf1512ac7195bfa44ed62c201deb2
Summary:
In upstream mercurial, c472ca028b32 changed changegroup.getchangegroup() to not
take a 'common' argument anymore.
Test Plan: run tests, no more stack traces
Reviewers: #mercurial, ttung, durham, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3819304
Signature: t1:3819304:1473091041:587ecb88dfcdd806d372825ec4d7512eb7fdb94e
Summary:
In upstream mercurial, 3d5e0fd41e0e changed discovery.outgoing() from taking a
changelog to taking the full repo object.
Test Plan: run tests, fewer stacktraces
Reviewers: #mercurial, ttung, durham, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3819300
Signature: t1:3819300:1473090807:fd0e4c2956fee4ad7d45051b6b7c5be481c766f1
getfile should return the file mode as (islink, isexec),
but it was returned as (islink, isbinary).
As a result, using 'hg absorb' removed executable mode from
the relevant changeset so the mode change showed up in the working directory.
Issue reported with test case by Thomas De Schampheleire.
Summary:
See http://bugs.python.org/issue27666 for details.
`win.box(0, 0)` will crash python if it is built with stack check / protection
(by using hardening-wrapper with `HARDENING_STACK_CHECK=1` and
`HARDENING_STACK_PROTECTOR=2`).
Test Plan: Run `hg chistedit tip` on Arch Linux, and it does not crash instantly.
Reviewers: #sourcecontrol, ttung, jeroenv
Reviewed By: jeroenv
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3806224
Signature: t1:3806224:1472758902:8a73645fd1ef98c9106c3b2766e368d4072a2d46
Summary:
The sshaskpass code is both an hg extension and a standalone python script.
In the latter case, it does not need to import mercurial packages (and
doing that may cause `ImportError` if `mercurial` is not in python path).
Test Plan:
Manually make the change in a Mac Mini device and check `chg pull` works
with ssh password authentication.
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3799653
Signature: t1:3799653:1472675282:9f7e9f4f29281db239f33cf396057ea12ff21773
Summary:
We were seeing performance issues on linux if we performed the madvise every
100MB. Let's do it once every gigabyte instead. The performance cost is pretty
negligble at that level.
Test Plan:
Ran the perf test for fulliter and verified the time went back down
to normal (a factor of 6x).
Reviewers: #fastmanifest, ttung
Reviewed By: ttung
Subscribers: ttung, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3799820
Signature: t1:3799820:1472675258:d279ac773aa4a2e027f4ecbe8a485b299a364efe
Summary: strnlen on some platforms use xmm registers to parallelize the operation. this may cause reads from uninitialized memory, which valgrind generates a false positive error. to avoid that, we preinitialize the memory *beyond* what we normally write so valgrind does not flag that as reading from uninitialized memory.
Test Plan: valgrind only complains about one uninitialized read (fixed in next diff)
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3799193
Signature: t1:3799193:1472672134:c866b6295cbfc9f4dd5e0d84a3af9d41144981c8
Summary:
This is to fix the below compiling error:
```
remotefilelog/cdatapack/cdatapack.c:102:28: error: implicit declaration of function ‘be64toh’ [-Werror=implicit-function-declaration]
packindex->data_offset = ntoh_data_offset(
^^^^^^^^^^^^^^^^
```
Feature test macros must be defined before libc `#include`s.
Test Plan: `make local` and the above error disappers.
Reviewers: #sourcecontrol, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797669
Signature: t1:3797669:1472671003:9f8674a14c39042d24c673e41321683807a60fab
Summary: D3786546 was broken in that it used an uninitialized return value.
Test Plan: `valgrind /Users/tonytung/Library/Caches/CLion2016.2/cmake/generated/fb-hgext-c4fc6e6f/c4fc6e6f/RelWithDebInfo1/cdatapack_get ~/.hgcache/fbsource/packs/e70f2fc9ec08516a196a6f5f3a475b21b6cee3b2 749a00bf1a195a711de2818b23da78e7f1ee034a` is 100% clean.
Reviewers: #fastmanifest, quark
Reviewed By: quark
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3799213
Signature: t1:3799213:1472671038:51522987c9b3bebc281d17f72a422af542d3d815
Test Plan: No need to test - comment change
Reviewers: rmcelroy, durham, mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3791555
Tasks: 12479677
Summary: --create and --to options work with non-scratch bookmarks too
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, durham, rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797183
Tasks: 12479677
Summary: It's safer to don't have bundlestore then to have uninitialized bundlestore
Test Plan: Run test-infinitepush-bundlestore.t
Reviewers: mitrandir, rmcelroy, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3797175
Tasks: 12479677
Summary:
That will allow us to figure out the best one (or even configure it on
per-repo basis if we want)
Test Plan: ran tests
Reviewers: #mercurial, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3723215
Signature: t1:3723215:1471373384:e2e68369b0a1a3a5e7d630ea97b33061e24832bf
There was a dumb bug where we attempted to use the matcher if it was NULL,
instead of only using the matcher when it was not NULL. We just need to remove
the bad !.
Summary:
Things I considered:
1. doing this after a fixed number of delta chain reads (this would likely be just fine, actually). there is the risk of a few very long/large chains blowing out the memory footprint.
2. tracking the actual memory addresses and not madvising away everything away. (unlikely to help, since it's probably the syscall + page table updates that is costly, not the address range)
3. letting FreeBSD / MACOS go nuts with MADV_FREE while limiting only linux (which uses the more costly MADV_DONTNEED)
In the end, I settled on this because we're already using this for the python version and it hasn't killed anyone.
Test Plan:
run:
```
#!/usr/bin/env python2.7
import binascii
import cdatapack
dp = cdatapack.datapack('/Users/tonytung/.hgcache/fbsource/packs/e70f2fc9ec08516a196a6f5f3a475b21b6cee3b2')
for ix, line in enumerate(open('/tmp/hashes', 'r')):
line = line.strip()
try:
dp.getdeltachain(binascii.unhexlify(line))
except:
print "failed when retrieving " + line
raise
```
and observed that the memory footprint stayed quite low.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3794420
Tasks: 12829688
Signature: t1:3794420:1472597468:6642f1e6cd85859dbf7cce3e4ce35d5375601be0
Summary: When lz4 gets a 0-length input to compress, it outputs a standard header (0 byte uncompressed output, followed by 12 mystery bytes.) When we uncompress this, `LZ4_decompress_fast` returns -1 (!). So we treat this as an oddball case and don't declare it corrupt.
Test Plan: dump all the entries in my fbsource datapack without any corruption reports
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3794368
Signature: t1:3794368:1472597620:a1368e55b6f74882c778036a5459e92d9c026b6a
Summary: We're not traversing the delta chain links sequentially, so we don't need this. Just discard and move on.
Test Plan: make local, run test from D3792849
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3794320
Signature: t1:3794320:1472597597:0a3087317c6d34f51888b69590db52d4f51bd27e
Summary: Most of this is pretty self-explanatory. We're creating two different test cases that invokes slightly different APIs to read the datapack (conventional datapack vs fastdatapack). The quirk is that fastdatapack does not expose the fanout prefix, so we suppress the test involving that for fastdatapack.
Test Plan: `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/lz4revlog/ /opt/local/bin/python2.7 ~/work/mercurial/facebook-hg-rpms/hg-crew/tests/run-tests.py test-remotefilelog-datapack.py`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3792849
Tasks: 12932864
Signature: t1:3792849:1472597547:35503ff5ef0c4910f3ae7f49b1268d3305a0b6c9
Summary: On some platforms, it's defined by errno.h. WTF.
Test Plan: make local
Reviewers: #fastmanifest, zamsden, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3789066
Signature: t1:3789066:1472521513:1b53feab3f46b55b2b0f4d20f82b829f304f5daf
