Previously we were kicking off background repacks even for non remotefilelog
repos. Moving the repack to be inside the remotefilelog requirement check will
prevent this.
Summary: This is to match the Python API.
Test Plan: used in later diff to pass test-datapack.t
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3668366
Signature: t1:3668366:1470341366:7c77684e646fe31a742e76158133597b46307797
Summary: The raw index is a byte offset, not an entry number. I hope the compiler is smart enough to optimize out the divide and multiply. :)
Test Plan: cdatapack_get on a delta chain that has a deltabasenode does not crash!
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3667033
Signature: t1:3667033:1470341429:b37da6c9ea6e37fe79b48ec6766c857b5e56c36a
Summary:
Replace some TODOs with actual error handling code.
Also lumped in typo fixes and style changes. Sorry.
Test Plan: Used in a later diffs to pass test-datapack.t
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3666875
Signature: t1:3666875:1470341306:67cd439341ad30fcd690ff2e399e8beacea1c0bb
Summary:
1. When bisecting, we don't want to wrap around. If middle == 0 and we're lesser than that, we should just fail.
2. large fanout should be header->config & LARGE_FANOUT. | means it's always a large fanout.
3. the format of the fanout table on disk makes it impossible to differentiate between an empty fanout entry and the first fanout entry, as they are both '0'. Therefore, any entry before the *second* fanout entry must implicitly search the 0th element.
4. fixed a bug in the calculation of the last index entry.
Test Plan: passed test-datapack.t with other fixes applied.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3666770
Signature: t1:3666770:1470341277:3f4f63a365e8bb0f4da6e574fc7f15228877c682
Summary: We don't ever need to modify the node sha data, so make it const.
Test Plan: compiles
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3666714
Signature: t1:3666714:1470340104:080ffa290a49388e797dcefc66976f6341932b76
Summary: Needed if we want to do a hybrid implementation of cdatapack
Test Plan: used in following diff.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3660087
Signature: t1:3660087:1470339373:4e8b548f1509af7f34d0a4bf8bd85723f38d238d
Summary: `iteritems()` differs from `__iter__()` slightly in that it yields the delta base and delta.
Test Plan:
run this toy program
```
#!/usr/bin/env python
import cdatapack
a = cdatapack.datapack('d864669a5651d04505ec6e5e9dba1319cde71f7b')
for x in a.iterentries():
print x[0], repr(x[1]), repr(x[2]), len(x[3])
for x in a:
print x[0], repr(x[1])
```
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3659133
Signature: t1:3659133:1470339839:dbdce5990a30ffe019ccc44fce97925b64524acd
Summary: cdatapack now has a getiter function, and it returns a cdatapack_iterator.
Test Plan:
using this toy program, dumped a pack file.
```
#!/usr/bin/env python
import cdatapack
a = cdatapack.datapack('d864669a5651d04505ec6e5e9dba1319cde71f7b')
for x in a:
print x
```
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3659005
Signature: t1:3659005:1470339657:aa39cc57a669b9bc4604933ce35ed20b3f81b468
Summary:
1. Get ntohl from arpa/inet.h as per the posix spec
2. Get ntohll from endian.h's be64toh
Test Plan: make local
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3671211
Signature: t1:3671211:1470341382:e6b0fe12094246aeb6be09252122bde9680e4599
Summary: Just a simple module declaration with no logic yet.
Test Plan:
```
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:2445a3a> make local
<output snipped>
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:2445a3a> python
Python 2.7.11 (default, Mar 1 2016, 18:40:10)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cdatapack
>>>
```
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3654786
Signature: t1:3654786:1470175354:c7e8847dcc74c83483d21888ad30cd9242fb461c
Summary:
Given a node sha, find it in the index file and retrieve the deltas. Checksum the data and dump it.
Depends on D3637000, D3636945
Test Plan:
```
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:68cd351> /Users/tonytung/Library/Caches/CLion2016.2/cmake/generated/cdatapack-64b7828e/64b7828e/Debug0/cdatapack_get d864669a5651d04505ec6e5e9dba1319cde71f7b f2e53f83c5dc806aa2eda87bb15fe0367baf3a7e
source/zippydb/tier_spec/tier_settings/zippydb.wildcard.tmpfs.zippydb_settings.cconf
Node Delta Base Delta SHA1 Delta Length
f2e53f83c5dc806aa2eda87bb15fe0367baf3a7e 0000000000000000000000000000000000000000 f32b366a6c44430df6526133f82f9638426ba9c5 37769
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:68cd351> hg debugdatapack d864669a5651d04505ec6e5e9dba1319cde71f7b --node f2e53f83c5dc806aa2eda87bb15fe0367baf3a7e
source/zippydb/tier_spec/tier_settings/zippydb.wildcard.tmpfs.zippydb_settings.cconf
Node Delta Base Delta SHA1 Delta Length
f2e53f83c5dc806aa2eda87bb15fe0367baf3a7e 0000000000000000000000000000000000000000 f32b366a6c44430df6526133f82f9638426ba9c5 37769
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:68cd351>
```
Reviewers: durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3637416
Signature: t1:3637416:1470094723:bce7e903cd0b80c293e16b7532c49e552d3039ef
Summary: Now that we report uncompressed lengths, the test output needs to be updated.
Test Plan: pass `PYTHONPATH=~/work/mercurial/facebook-hg-rpms/fb-hgext/:~/work/mercurial/facebook-hg-rpms/lz4revlog/:~/work/mercurial/facebook-hg-rpms/remotefilelog/ python ~/work/mercurial/facebook-hg-rpms/hg-crew/tests/run-tests.py -j32 test-repack.t`
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3656796
Signature: t1:3656796:1470175464:56bf12516710cef8e8aaa7e7b3e0dbdfa220d797
Summary:
1. offsets are absolute byte offsets. convert them to entry offsets to make the bisect code a lot simpler.
2. when writing entries to pack chain, we need to advance the pointer.
Depends on D3627122
Test Plan: used in later diff.
Reviewers: durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3637000
Signature: t1:3637000:1469741885:c2416a3b30e5bb2b64e6bb7062f4c02098be91eb
Summary:
When retrieving a delta chain, datapack.py uncompresses the delta chain data. However, when iterating over the datapack, we get the compressed length. THis is not desirable as the output is no longer consistent. This diff peeks into the lz4 header to get the uncompressed length when iterating.
Depends on D3627119
Test Plan:
```
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:e2ef218> hg debugdatapack d864669a5651d04505ec6e5e9dba1319cde71f7b --node f2e53f83c5dc806aa2eda87bb15fe0367baf3a7e
source/zippydb/tier_spec/tier_settings/zippydb.wildcard.tmpfs.zippydb_settings.cconf
Node Delta Base Delta SHA1 Delta Length
f2e53f83c5dc806aa2eda87bb15fe0367baf3a7e 0000000000000000000000000000000000000000 f32b366a6c44430df6526133f82f9638426ba9c5 37769
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:e2ef218> hg debugdatapack d864669a5651d04505ec6e5e9dba1319cde71f7b | tail -n 4
source/zippydb/tier_spec/tier_settings/zippydb.wildcard.tmpfs.zippydb_settings.cconf
Node Delta Base Delta Length
f2e53f83c5dc 000000000000 37769
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:e2ef218>
```
Reviewers: durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3636945
Signature: t1:3636945:1469811243:b21d90d9599244ed4600c5336818b9a18eacf3ff
Summary:
`->index_table` is not heap-alloacted. however, `->fanout_table` is and should be released.
Also added call to `close_datapack()` at the end of `cdatapack_dump.c`.
Depends on D3627122
Test Plan: valgrind is much happier now.
Reviewers: durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3631368
Signature: t1:3631368:1469741779:e0c4e5d59c7e73c8aa3507901df3005383f0d3f5
Summary: This is not yet complete, but seems to be able to parse a data file.
Test Plan:
`/Users/tonytung/Library/Caches/CLion2016.2/cmake/generated/cdatapack-64b7828e/64b7828e/Debug/cdatapack_dump d864669a5651d04505ec6e5e9dba1319cde71f7b > /tmp/2`
compare it with the output of `hg debugdatapack --long d864669a5651d04505ec6e5e9dba1319cde71f7b > /tmp/1`
and it exactly matches.
Reviewers: durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3627122
Signature: t1:3627122:1470085301:c9b9e8b2fa57bb7a09dd56d3c811ff8eadbb85ba
Summary:
This diff adds a check: after `linelog_updatelines`, `annotateresult *ar`
should be updated as if we have called a separate `linelog_annotate`.
Test Plan:
Run `afl` for a while and make sure it does not find any crashes:
```
mkdir ./testdir
CC=afl-gcc make
./linelogcli ./testdir/readme init annotate 0 replacelines 10 0:0 0:3 replacelines 20 2:2 1:3 replacelines 30 1:3 1000:1000
afl-fuzz -i testdir -o findingdir ./linelogcli @@ dump annotate 10 replacelines 10 0:0 0:3 replacelines 20 2:2 1:3 replacelines 30 1:3 1000:1000
```
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3650197
Signature: t1:3650197:1470336333:2f5e6d2e914917b33bddc595344d2a0b98067606
Summary:
Previously, `replacelines` assume the `annotateresult` user passed has the
property: `linecount < maxlinecount`. But that's not true after
`linelog_annotateresult_clear`, where `linecount = maxlinecount = 0` and
that leads to `SIGSEGV` if both `a1` and `a2` passed to `replacelines` are `0`.
This issue was found by the following `afl` testcase:
./linelogcli ./testdir/readme init annotate 0 replacelines 10 0:0 0:3 replacelines 20 2:2 1:3 replacelines 30 1:3 1000:1000
afl-fuzz -i testdir -o findingdir ./linelogcli @@ dump annotate 10 replacelines 10 0:0 0:3 replacelines 20 2:2 1:3 replacelines 30 1:3 1000:1000
Test Plan:
Run the above `afl` test. Make sure it won't report a crash in a couple
of minutes.
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3676049
Signature: t1:3676049:1470661720:e5c4d6f45ac51d58fae856b08f79982bec44b535
Summary: Most of them are pretty simple and don't need to be standalone functions.
Test Plan:
```
[andromeda]:~/work/mercurial/facebook-hg-rpms/fb-hgext:ae2ccaa> make local
<snipped>
[andromeda]:~/work/mercurial/facebook-hg-rpms/fb-hgext:ae2ccaa> cd tests/
[andromeda]:~/work/mercurial/facebook-hg-rpms/fb-hgext/tests:ae2ccaa> PYTHONPATH=~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/mutable-history/hgext python ~/work/mercurial/facebook-hg-rpms/hg-crew/tests/run-tests.py -j32 test-fastmanifest*.{py,t}
.........
# Ran 9 tests, 0 skipped, 0 warned, 0 failed.
[andromeda]:~/work/mercurial/facebook-hg-rpms/fb-hgext/tests:ae2ccaa>
```
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mjpieters, mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3659242
Signature: t1:3659242:1470340089:e1a569ce05bc3d40af043369b7bc4c76616f5b88
Summary:
1) Add sync status
2) Combine sync status and phab status to use one unified conduit call,
i.e. differential.querydiffhashes
Test Plan:
cd hg-crew
make local
cd ../fb-hgext
make local
cd tests
../../hg-crew/tests/run-tests.py test-phabstatus.t
../../hg-crew/tests/run-tests.py test-syncstatus.t
Reviewers: wqfish, lcharignon, #sourcecontrol, wez, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3651915
Tasks: 10100400
Signature: t1:3651915:1470340328:bf003006f6afe9b86a40f204e150e0d12350c21d
Summary:
Fastmanifest debug logs in blackbox.log are less interesting in a rage report,
but they take a lot of lines. This diff uses `grep` to filter them out from rage report.
Test Plan: `hg rage --preview`
Reviewers: #mercurial, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3676751
Signature: t1:3676751:1470421349:3b5e0cb4118473f5550e83bf9c5e7ac27a59bcb0
Summary:
The thin wrapper defines common APIs like `resize` `flush`, `close`, and
`copyfrom`. Some of them are used in sub-classes. It also handles
`LINELOG_RESULT_ENEEDRESIZE` and raise `LinelogError` for other errors.
Test Plan: `cython2 linelog.pyx`
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3655463
Tasks: 11291522
Signature: t1:3655463:1470402560:75e552ed3d6205ae45812d1cdfe7bc14cfb1eec3
Summary:
`linelog.pyx` is meant to be the CPython wrapper for linelog.
This diff declares types and functions in `linelog.h`.
According to http://docs.cython.org/en/latest/src/userguide/external_C_code.html,
we have to duplicate them as Cython cannot parse `.h` files.
Test Plan: `cython2 linelog.pyx`
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3655437
Tasks: 11291522
Signature: t1:3655437:1470336663:802ede2b82790a6c901ba0cc76c4f39fcac1e6ac
Summary: This is the core API to do writes.
Test Plan: `gcc -Wall -Wextra -Wconversion -c linelog.c`
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3641637
Tasks: 12416202
Signature: t1:3641637:1470335236:758df55835ac6c42212783d4cc12302588ce2216
Summary: Maintains the lookupcache across edit operations and builds the cache fully after 10,000 reads. This helps cap the overhead of doing sqlreads.
Test Plan:
Ran a large hg update followed by hg status and verified that the status call took 10 seconds instead of 60 seconds.
```
~/local/fbsource> time hg.real up master --profile && sleep 60 && sync && time hg.real status --pager=off
62093 files updated, 0 files merged, 6740 files removed, 0 files unresolved
real 2m24.262s
user 0m52.124s
sys 0m13.462s
real 1m2.942s
user 0m48.617s
sys 0m12.631s
~/local/fbsource> time hg.real up master~50000 --profile && sleep 60 && sync && time hg.real status --pager=off --config extensions.sqldirstate=../fb-hg
ext/sqldirstate/
49029 files updated, 0 files merged, 19804 files removed, 0 files unresolved
real 1m29.619s
user 0m47.244s
sys 0m12.264s
real 0m10.269s
user 0m7.964s
sys 0m0.842s
```
Reviewers: mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3659071
Signature: t1:3659071:1470177313:9af93af70fdbd5929b97879b58553a3cea415c19
Summary:
It should include the filelen and the deltalen fields, which are
2 and 8 bytes.
Test Plan: visual.
Reviewers: durham
Reviewed By: durham
Subscribers: mitrandir
Differential Revision: https://phabricator.intern.facebook.com/D3627108
Signature: t1:3627108:1469742083:ffb59768906d9e5463065eec92e1c80cc8482884
Summary:
This is mainly to address the same issue as D3609747: prevent issues caused
by running hg as root. The difference is this extension reads owner and uid
dynamically without config when initializing of a localrepo object.
As a side effect it covers more situations like running hg in others' repos, while
being less restrictive for some other commands like `sudo hg version`.
Test Plan: Added a new test
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: ikostia, rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3619997
Tasks: 11723374
Signature: t1:3619997:1470176448:beaf53c09fca498206767641ffa4315a744ee07e
Summary:
Implement `linelog_annotate`, which is the core algorithm for generating
`linelog_annotateresult`. It is basically walking though and executing
the instructions.
Test Plan: `gcc -Wall -Wextra -Wconversion -c linelog.c`
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3629591
Tasks: 12416202
Signature: t1:3629591:1470157936:764bb7c5d1208e73d93d26063a8ad380dbceb495
Summary:
As mentioned in D3628571, the `.c` code manages the memory of
`linelog_annotateresult.lines`. We need a resize function for it.
Test Plan:
`gcc -Wall -Wextra -Wconversion -c linelog.c` and it only reports
"defined but not used" warnings.
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3629350
Tasks: 12416202
Signature: t1:3629350:1470055356:01993a6df2ccd849ed01961eb996c86bb0a17c78
Summary:
As described in the README (D3628440), the linelog buffer is an array of
instructions. The `.h` file only declares the plain buffer but not the
internal representation of the instruction. They are intended to be
invisible to users.
This diff declares the instruction structure, related types and helpers in
`linelog.c`. `decode` and `encode` translate between the internal (`.c`,
structured `linelog_inst`) and the external (`.h`, plain `linelog_buf`)
representation. Other macros are to help making code shorter and easier to
understand.
Test Plan: `gcc -Wall -Wextra -Wconversion -c linelog.c`
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3629132
Signature: t1:3629132:1470157631:4523c53a3af28061828980845d3acfbd82c58361
Summary:
Declare core APIs for linelog:
- `annotate`, the "read" part
- `replacelines`, the "write" part
As mentioned in D3334518, this is part of a rewrite of D3334518.
These 2 APIs are the only ones involving core logic of the linelog structure.
Test Plan: `gcc -Wall -Wextra -Wconversion linelog.h`
Reviewers: #mercurial, zamsden, ttung, simonfar
Reviewed By: simonfar
Subscribers: akushner, zamsden, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3628848
Tasks: 12416202
Signature: t1:3628848:1470080022:1f434d998c0bf63a494f70d47bc6a6fc05954446
Summary:
Declare some trivial APIs, which either clear the memory or return a value
without much calculation.
Test Plan: `gcc -Wall -Wextra linelog.h`
Reviewers: #mercurial, simonfar, ttung
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3628681
Tasks: 12416202
Signature: t1:3628681:1469647679:437db1610dec8a21a71b2d94c64d6d085321e1cd
Summary:
Add essential data structures for upcoming APIs.
This is part of a rewrite of D3334518. The new API removes everything about
I/O (aka. `stdio.h`) and makes it the caller's responsibility to prepare
(`malloc` or `mmap`) and resize the memory buffer.
This makes it possible to have a much shorter (thus easier to review) code
more dedicated to core algorithm, as the following are no longer necessary:
- I/O related APIs: `open`, `close`, `flush`, `saveas`
- I/O error handling
The error handling part is also changed that we now can use integer error
numbers instead of a boolean value + an error string.
Test Plan: `gcc -Wall -Wextra linelog.h`
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: durham, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3628571
Tasks: 12416202
Signature: t1:3628571:1469643667:4fbf4fd7e624820ee6d8409e2067c8bd0e99a829
