Commit Graph

24 Commits

Author SHA1 Message Date
Phil Cohen
37ed176ad7 debugdatapack: return 1 and warn if there are invalid entries
An invalid entry is any entry with a base not in the pack, or whose deltabases
form a cycle.

If there are any entries like that, the output will look like this:

```
(Root):
Node          Delta Base    Delta Length  Blob Size
665a7e7913af  e66038a2894e  61            2142
52bd634be310  000000000000  2142          2142
8b5847087ce0  000000000000  2142          2142
960f5acb3e99  edf2ffd7daab  162           2142
b7d7e5aa692e  8b5847087ce0  162           2142
cdcc4d74d667  960f5acb3e99  324           2142
Total:                      14652         48920     (70.0% smaller)
Bad entry: 960f5acb3e99 has an unknown deltabase (edf2ffd7daab)
Bad entry: b7d7e5aa692e has an unknown deltabase (edf2ffd7daab)
2 invalid entries
```

Differential Revision: https://phab.mercurial-scm.org/D1271
2017-11-07 11:23:29 -08:00
Phil Cohen
d4642ef2ca debugdatapack: suppport passing multiple files
Also print the name of the packfile being inspected, and \itweak newlines a bit.

This simplies debugging several packfiles in tests.

Differential Revision: https://phab.mercurial-scm.org/D1326
2017-11-06 19:09:15 -08:00
Phil Cohen
ef47b31d2f debugdatapack: print delta and blob size totals and compression %
Also print "(empty name)" if the filename is empty.

Differential Revision: https://phab.mercurial-scm.org/D1259
2017-11-05 21:45:40 -08:00
Phil Cohen
53d2c02095 debugdatapack: add blob size
It's useful to know the size of the resultant object, not just the delta entry.

Differential Revision: https://phab.mercurial-scm.org/D1254
2017-11-05 18:45:21 -08:00
Mihails Smolins
9a121651cc remotefilelog: enable background prefetch for pull
Summary:
Previously prefetch on pull ran in foreground. Now it can run in background
if specified. Optionally background prefetch can be followed by background
repack.

Test Plan: Added a test case

Reviewers: simonfar, durham

Reviewed By: simonfar

Subscribers: medson, mjpieters, #mercurial

Differential Revision: https://phabricator.intern.facebook.com/D5406626

Tasks: 19727343

Signature: t1:5406626:1500059334:207b4100cca536cbe33f6c6dfad596d03a6fa14f
2017-07-17 03:04:22 -07:00
Durham Goode
f02e3e0686 revlog: rename revlog constants
Upstream has renamed some revlog constants, so we need to update our code
accordingly.
2017-05-21 17:09:08 -07:00
Jun Wu
2b3b8e46a5 remotefilelog: add filelog methods
Summary:
Filelog methods like `addrevision`, `revision(raw=True)` are needed for flag
processor (lfs) to work correctly. Add them in remotefilelog so lfs wrapper
code could replace them.

Test Plan:
Run existing tests. Stronger tests and lfs integration test will be added when
this area is more complete.

Reviewers: #mercurial, durham, rmcelroy

Reviewed By: durham

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4903959

Signature: t1:4903959:1493163809:5ebd88fac21d8225a12ce68bfc63a2867ee43769
2017-04-26 19:52:20 -07:00
Kostia Balytskyi
0c2d706810 remotefilelog: wrap lz4 imports for compatibility
Differential Revision: https://phabricator.intern.facebook.com/D4707115
2017-03-17 14:02:26 -07:00
Durham Goode
57af7da9ff remotefilelog: fix debugindex
A while back we added file blobs with '_old' on the end. This actually broke
debugindex. Let's filter them out.
2017-02-26 16:43:19 -08:00
Durham Goode
9514bca8ce packs: make debug*pack commands take more paths
Summary:
Previously 'hg debug[data|hist]pack' required passing the filepath without the
suffix (just up to the hash). This was kind of a pain when using tab complete,
and a pain in tests when trying to use 'find' to run hg debugdatapack on a
number of files. Let's make the debug commands more forgiving.

Test Plan: Manual verification

Reviewers: #mercurial, dsyang, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4261586

Signature: t1:4261586:1480705548:1aa5dbe44ed5e29c28a29a8256de12ffee8d7387
2016-12-02 14:37:53 -08:00
Durham Goode
f43ba75915 remotefilelog: fix pyflakes and module import errors
Summary:
This fixes all the pyflaks and module errors for the main remotefilelog
code base.

Test Plan: ./run-tests.py test-check* test-remotefilelog*

Reviewers: #mercurial, quark

Reviewed By: quark

Subscribers: mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D4055537

Signature: t1:4055537:1477049663:ee904d311d17d3659e055e2c109c68c9023cfd1f
2016-10-21 11:02:09 -07:00
Ryan McElroy
f4dd73e113 remotefilelog: pass modern check-code
Test Plan: run-tests.py test-check-code-hg.t

Reviewers: #mercurial, ttung, simonfar

Reviewed By: simonfar

Subscribers: simonfar, mjpieters

Differential Revision: https://phabricator.intern.facebook.com/D3777581

Tasks: 12855049

Signature: t1:3777581:1472224785:a15040cec1c95ca60d1be837d905b3c3d87be362
2016-08-26 08:48:07 -07:00
Tony Tung
9e557758b0 [datapack] add --node as a parameter to dump extra data about a node
Summary:
It obtains the deltachain and dumps the chain to the console.

Depends on D3627117.

Test Plan:
```
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:3266095> hg debugdatapack d864669a5651d04505ec6e5e9dba1319cde71f7b --node ba5fbf1aba48f25d46228626917b2705adc9e7c8

arcanist/__phutil_library_map__.php
Node                                      Delta Base                                Delta SHA1                                Delta Length
ba5fbf1aba48f25d46228626917b2705adc9e7c8  0000000000000000000000000000000000000000  df442a6f976b946c266f76b0f63a198e8aabf809  3993
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:3266095>
```

Reviewers: durham

Reviewed By: durham

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3627119

Signature: t1:3627119:1469738313:d61726585a020ed4cbabbb1f623eb202ccd51b9f
2016-07-28 17:15:21 -07:00
Tony Tung
7111de0994 [datapack] allow for long hashes to be printed
Summary:
This will help verify the C datapack reader.

Depends on D3627112.

Test Plan:
```
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:4063a65> hg debugdatapack --long  d864669a5651d04505ec6e5e9dba1319cde71f7b  | head -n 15

arcanist/__phutil_library_map__.php
Node                                      Delta Base                                Delta Length
ba5fbf1aba48f25d46228626917b2705adc9e7c8  0000000000000000000000000000000000000000  1265

arcanist/canary/FacebookConfigeratorArcanistCanaryWorkflow.php
Node                                      Delta Base                                Delta Length
142f9991fca1a16c6544cb6e5a0071296e712268  0000000000000000000000000000000000000000  6546

arcanist/lint/FacebookConfigeratorLintEngine.php
Node                                      Delta Base                                Delta Length
c8630501c45f1bc1dc47df2ee2ad354993438cdb  0000000000000000000000000000000000000000  2811

arcanist/lint/linter/FbcodePyFlake8Linter.php
Node                                      Delta Base                                Delta Length
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:4063a65> hg debugdatapack   d864669a5651d04505ec6e5e9dba1319cde71f7b  | head -n 15

arcanist/__phutil_library_map__.php
Node          Delta Base    Delta Length
ba5fbf1aba48  000000000000  1265

arcanist/canary/FacebookConfigeratorArcanistCanaryWorkflow.php
Node          Delta Base    Delta Length
142f9991fca1  000000000000  6546

arcanist/lint/FacebookConfigeratorLintEngine.php
Node          Delta Base    Delta Length
c8630501c45f  000000000000  2811

arcanist/lint/linter/FbcodePyFlake8Linter.php
Node          Delta Base    Delta Length
[andromeda]:~/work/mercurial/facebook-hg-rpms/remotefilelog:4063a65>
```

Reviewers: durham

Reviewed By: durham

Subscribers: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3627117

Signature: t1:3627117:1469735318:103e9a21be082749332572c9c4f9942ea9c1c248
2016-07-28 17:07:04 -07:00
Martin von Zweigbergk
adcdb9289c commands: tell @command decorator about arguments
Before this patch, debugremotefilelog and verifyremotefilelog would
crash if not given a path. Also, many commands would accept arguments
they then ignored.
2016-06-30 10:14:17 -07:00
Jeroen Vaelen
07efaadb9d [remotefilelog] use hashlib to compute sha1 hashes
Summary:
hg-crew's c27dc3c3122 and c27dc3c3122^ were breaking our extensions:

```
$ hg log -r c27dc3c3122^
changeset:   9010734b79911d2d2e7405d91a4df479b35b3841
user:        Augie Fackler <raf@durin42.com>
date:        Thu, 09 Jun 2016 21:12:33 -0700
s.ummary:     cleanup: replace uses of util.(md5|sha1|sha256|sha512) with hashlib.\1
```

```
$ hg log -r c27dc3c3122
changeset:   0d55a7b8d07bf948c935822e6eea85b044383f00
user:        Augie Fackler <raf@durin42.com>
date:        Thu, 09 Jun 2016 21:13:23 -0700
s.ummary:     util: drop local aliases for md5, sha1, sha256, and sha512
```

I did a grep over facebook-hg-rpms to see what was affected:
```
$ grep "util\.\(md5\|sha1\|sha256\|sha512\)" -r ~/facebook-hg-rpms
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/basestore.py:            sha = util.sha1(filename).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/basestore.py:                sha = util.sha1(filename).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/shallowutil.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/shallowutil.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/debugcommands.py:    filekey = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/historypack.py:        namehash = util.sha1(name).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/historypack.py:        node = util.sha1(filename).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/historypack.py:        files = ((util.sha1(filename).digest(), offset, size)
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/fileserverclient.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/fileserverclient.py:    pathhash = util.sha1(file).hexdigest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/remotefilelog/basepack.py:        self.sha = util.sha1()
/home/jeroenv/facebook-hg-rpms/remotefilelog/tests/test-datapack.py:        return util.sha1(content).digest()
/home/jeroenv/facebook-hg-rpms/remotefilelog/tests/test-histpack.py:        return util.sha1(content).digest()
Binary file /home/jeroenv/facebook-hg-rpms/hg-crew/.hg/store/data/mercurial/revlog.py.i matches
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:            return util.sha1(fh.read()).hexdigest()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:        sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:        sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:        sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/fb-hgext/sparse.py:    sha1 = util.sha1()
/home/jeroenv/facebook-hg-rpms/mutable-history/hgext/simple4server.py:        sha = util.sha1()
/home/jeroenv/facebook-hg-rpms/mutable-history/hgext/evolve.py:        sha = util.sha1()
```
This diff is part of the fix.

Test Plan:
Ran the tests.
```
$MERCURIALRUNTEST -S -j 48 --with-hg ~/local/facebook-hg-rpms/hg-crew/hg
```

Reviewers: #sourcecontrol, ttung

Differential Revision: https://phabricator.intern.facebook.com/D3440041

Tasks: 11762191
2016-06-15 15:48:16 -07:00
Durham Goode
93fbca3e39 repack: add automatic incremental background repacking after pull
Summary:
This runs the incremental background repacking logic after hg pull.

As part of adding tests, I also added a 'hg debugwaitonrepack' function that
will wait until any pending repack is done before returning, so the tests can
wait on repacks without so many sleeps.

Test Plan: Adds a test

Reviewers: mitrandir, #mercurial, ttung, rmcelroy

Reviewed By: rmcelroy

Subscribers: rmcelroy

Differential Revision: https://phabricator.intern.facebook.com/D3306526

Signature: t1:3306526:1463696933:9e27daf0c08076468e8f365a3c372fa7d4f56bde
2016-05-20 09:31:28 -07:00
Martin von Zweigbergk
41226c0217 debugindex: use normal handling for --dir
'debugindex --dir' is used for looking up the revlog of a directory
manifest, so it should use the normal (local file system) handling,
not remotefilelog.
2016-01-04 21:32:21 -08:00
Durham Goode
5e4370b46d packs: add debug commands to view pack contents
Summary: Some simple debug commands to print the contents of each pack.

Test Plan: Ran it manually, and added a simple test

Reviewers: #mercurial, ttung, mitrandir

Reviewed By: mitrandir

Differential Revision: https://phabricator.intern.facebook.com/D3277233

Signature: t1:3277233:1463085196:c54fc875d536a96150bb1461b77247a5d7a9402c
2016-05-16 10:59:09 -07:00
Durham Goode
71bd8c2561 checkcode: fix errors in cacheclient and debugcommands
Summary: Fix failures found by check-code.

Test Plan: Ran the tests

Reviewers: #sourcecontrol, ttung

Reviewed By: ttung

Differential Revision: https://phabricator.fb.com/D3221366

Signature: t1:3221366:1461648117:088f3a5837393499e1a383af860bd1a935e0cba7
2016-04-26 13:00:31 -07:00
Augie Fackler
5eecca9702 remotefilelog: handle the death of repo.sopener (hg change 0bbe3294361a)
repo.sopener has been deprecated since hg 2.3, and repo.svfs replaces
it. Since it's been dead for so long, let's just use svfs and call it
good enough.
2015-06-30 10:12:38 -04:00
Durham Goode
688d0f9594 Fix debugremotefilelog command 2013-12-13 11:42:50 -08:00
Durham Goode
393958c76b Allow naming repos
Enables specifying a name for a repo that is used in the cache key.
This allows multiple repos on a machine to share a cache without the
risk of keys overlapping.
2013-08-15 11:00:51 -07:00
Durham Goode
85e48b58fd Move server and debug logic into their own files
__init__.py was getting quite large. This change moves the server and debug
logic into their own files.  Client-side logic remains in __init__.py
2013-11-25 16:36:44 -08:00