Summary:
Before this patch, it wasn't clear from hg debugcachemanifest --list
which manifest were older and newer or what rev were actually cached. This
patch improves the output of hg debugcachemanifest --list --debug to show
this information.
Test Plan: add debug output to the test
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3427791
Summary:
Before this patch we were returning the cache entries alphabetically.
This patch changes the behavior to return the entries sorted by date and
alphabetically. This will be used to simplify the code for pruning entries
in furter patches.
Test Plan: test output changes
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3427789
Summary:
The existing statprof uses signals to perform stack sampling. Those signals can
interfere with system calls though. This patch introduces a way of using python
threads to perform sampling from a background thread.
This also changes the default to be the threaded profiler instead of the
sampling one, since the sampling can crash the proces sometimes.
In some situations this produces a more accurate stack trace.
Test Plan:
```
sudo cp statprof.py /usr/lib64/python2.6/site-packages/statprof.py
hg pull --profile
# Verified I got output and that it was different from the old statprof (it was
# more accurate, as compared with time.time() measurements in the code)
```
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3402864
Signature: t1:3402864:1465978363:c8455d119cc03c2c475e190aee28587a9d434c4d
Summary:
Before this patch if the path was not accessible we would fail to
compute the limit.
Test Plan:
We should test systematically all these kind of issues, I am doing
a refactoring that will allow us to do that.
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3424996
Summary:
Before this patch we were stopping execution when the lock couldn't
be taken. This was problematic when running hg command for someone else's repo
(for example to debug problems). This patch allows us to continue the execution
of the command when that happen, except we won't use fastmanifest.
Test Plan: This is reflected in the test output
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3422458
Summary: This could break people using fastmanifest in someone else's repo
Test Plan: tests pass, not sure how we normally test permission stuff
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3420870
Summary:
Before this patch we would have error using fastmanifest on repos you
didn't own.
Test Plan: tests pass
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3420863
Summary: It can't find those extensions now that we removed the extensions from the system pythonpath. Therefore, we need to load it from the source directory.
Test Plan: pass PYTHONPATH=~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/mutable-history/hgext python ~/work/mercurial/facebook-hg-rpms/hg-crew/tests/run-tests.py -j32 test-fbhistedit-stop.t
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3419503
Signature: t1:3419503:1465590764:23e348efee30cabbbff232b7457d5820125bd6ad
Summary: printf appears to be more platform agnostic than echo -n, which is not supported on bsd/mac.
Test Plan: run PYTHONPATH=~/work/mercurial/facebook-hg-rpms/remotenames/:~/work/mercurial/facebook-hg-rpms/mutable-history/hgext python ~/work/mercurial/facebook-hg-rpms/hg-crew/tests/run-tests.py -j32 test-catnotate.t
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3419500
Signature: t1:3419500:1465590785:a34da9af73c64d9edef66a7b4a6ed3abd885e5c7
Summary: Newer versions of gcc (5.x) turn on `-Wstrict-prototype` by default. Turn this on in our setup.py file for older compilers, and fix the errors that crop up from this.
Test Plan: make local
Reviewers: tnardone, wez, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3414620
Signature: t1:3414620:1465514102:d91026062582e2a3ab3fc773bbf17686fa38609f
Summary:
This overrides the normal Mercurial statprof logic to allow us to pass custom
configuration to our customized version of statprof. In particular, it allows us
to choose between the thread and the signal profile methods, and between the
hotpath and json output formats.
Test Plan: hg.real log -r tip --config extensions.statprofext=../fb-hgext/statprofext.py --profile --config statprof.format='json' --config profiling.output=woooo.log
Reviewers: #mercurial, ttung, lcharignon
Reviewed By: lcharignon
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3408386
Signature: t1:3408386:1465485488:400da1f8e1787674c9ea3888166767a485f2fbdf
Summary:
As discussed, we want to make sure we have a good SocketServer for Python 2.6,
which is unsupported by PSF but still being widely used by our users.
This diff adds a new extension that tries to fix buggy Python standard modules,
namely the SocketServer. It fixes a critical race condition for Python 2.6
(https://bugs.python.org/issue21491) and the zombie process issue for both
Python 2.6 and 2.7 (https://bugs.python.org/issue11109).
A simple test is added to check zombie processes are no more.
Test Plan:
Run the newly added test, `test-patchpython.py` with both Python 2.6 and 2.7.
Comment out the `import patchpython` line in the test and make sure it fails
on Python 2.6 and 2.7 with `AssertionError: zombie process detected`.
Reviewers: #mercurial, ttung, mjpieters
Reviewed By: mjpieters
Subscribers: simonfar, durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3384909
Signature: t1:3384909:1465319491:0ba64b496186712651b7ee6bd502d1e2ccac57d1
Summary:
This adds a json display format that can dump the raw samples to json. This
allows other processes to read the data and log/analyze them.
Test Plan: hg.real log -r tip --config extensions.statprofext=../fb-hgext/statprofext.py --profile --config statprof.format='json' --config profiling.output=woooo.log
Reviewers: #mercurial, ttung, lcharignon
Reviewed By: lcharignon
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3408369
Signature: t1:3408369:1465436261:5fc344effdea46d2cf07c81cb707791ceea98f11
Summary:
We change pydoc to comment for the lock timeout and document the
remaining constants
Test Plan: documentation change, tests pass
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3398903
Summary: I forgot to add it to setup.py previously
Test Plan: How to test that? Is running the build enough?
Reviewers: ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3399075
Summary: Add code to log metrics for fastmanifest, also handles aggregate
metrics. We reuse ui.log for logtoprocess. We also refactor diffs and
filesnotin to make it easier to log cache hit and miss ratio.
Test Plan:
Replaced ui.log by ui.status and aggregated the field, check
that I see metrics getting logged.
Reviewers: ttung, durham
Differential Revision: https://phabricator.intern.facebook.com/D3368504
Summary:
There was a mix up between @ttung's diff and mine and this import
went missing.
Test Plan: Tests pass now
Reviewers: ttung, durham
Subscribers: mjpieters, ttung
Differential Revision: https://phabricator.intern.facebook.com/D3407832
Summary:
Mercurial stack traces often have a bunch of boiler plate function wrappers in
the stack. This change gives statprof a list of frames it can skip. As a side
benefit, this also means we can get rid of the logic that cuts off the top 2% of
the stack trace. It was meant to prevent the dispatch.py functions from showing
up in the stack, but that's now covered by the skip list.
Test Plan:
```
sudo cp statprof.py /usr/lib64/python2.6/site-packages/statprof.py
hg pull --profile
| 100.0% __init__.py: runcommand line 45: mercurial.dispatch.run()
| 100.0% cachemanager.py: runcommandtrigger line 325: return orig(lui, repo, *arg...
| 100.0% reflog.py: runcommand line 242: result = orig(*args, **kwargs)
| 100.0% dispatch.py: statprofile line 31: return orig(lui, repo, cmd,...
| 99.7% __init__.py: wrappull line 959: return func()
| 99.7% remotenames.py: expullcmd line 119: return orig(ui, repo, *args...
...
```
Reviewers: ttung, #mercurial, lcharignon
Reviewed By: lcharignon
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3402841
Signature: t1:3402841:1465407465:b0060496cd148eb6180cdf732bb7b8e6d3cd4db6
Summary: This mostly replicates the logic in `manifest.py::manifest::add(..)`, except it interfaces with the fastmanifest cache instead.
Test Plan:
used `hg debugcachedmanifest` to populate the cache, then did a commit with `pdb.set_trace()`.
added a correctness test that exercises the code path.
Reviewers: simonfar, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3397910
Tasks: 11294278
Signature: t1:3397910:1465407138:60c7b696b4bafa153c8076ab58c53bf71a9fcfd4
Summary:
We add the revisions actually cached by fastmanifest as it could be
useful to debug what happened after the fact. Since this is quite slow (makes
the command ~2x slower), then we gate that behind a config.
Test Plan:
hhg rage --config extensions.rage=/home/lcharignon/facebook-hg-rpms/fb-hgext/rage.py
>> Paste does not contain fastmanifest related stuff
hhg rage --config extensions.rage=/home/lcharignon/facebook-hg-rpms/fb-hgext/rage.py
--config rage.fastmanifestcached=True --config
>> Paste contains smartlog for entries cached in fastmanifest
Reviewers: quark, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3394646
Summary:
I ran flake8 and saw a bunch of things that we could fix to make the
code more succint and legible.
Test Plan: all tests pass
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3398906
Summary:
Before this patch, all the events logged by the sampling extensions
were dumped to the log file. With this patch, one can specify the relevant
events to log as well as a value to be added for each event. This will be
useful to specify what logging table to use for each of the logged events.
More information added to the documentation accordingly.
Test Plan:
The test changes accordingly and we add events that gets stopped
by the filter to make sure that the filter works.
Reviewers: ttung, durham
Differential Revision: https://phabricator.intern.facebook.com/D3387446
Summary:
We used to have test code to make the test deterministic. Now we
enforce the same constraint in code and that code is not necessary anymore.
We change the test output to match the new, deterministic output.
Test Plan: Ran the tests 80 times and checked that the output is stable
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3387259
Summary:
This will allow us to know what revisions are actually cached vs
revisions that would be cached by a trigger.
Test Plan: Add new tests
Reviewers: ttung, durham
Differential Revision: https://phabricator.intern.facebook.com/D3385955
Summary:
`lrucachedict`'s `get` method is a lower-level API that retrieves a _lrucachenode object, not what we want. the idiomatic construction is:
```
if key in lrucachedictobj:
do_something(lrucachedictobj[key])
```
Test Plan: hg amend with fastmanifest works. wrote a test that failed before this diff, passed after.
Reviewers: #mercurial, lcharignon, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3384502
Tasks: 11569523
Signature: t1:3384502:1465004304:8dfee6a4d0b2f6bb39262a310744458838cb0bf3
Summary: Noticed this while trying to debug something that since we call `_exit()`, we don't have the luxury of the standard flushing of output streams. If we have a silent worker that doesn't output enough to trigger an automatic flush, we might leave things in the buffer.
Test Plan:
pass unit tests.
output something small after the fork via ui.log, and it is now flushed to console.
```
diff --git a/fastmanifest/cachemanager.py b/fastmanifest/cachemanager.py
--- a/fastmanifest/cachemanager.py
+++ b/fastmanifest/cachemanager.py
@@ -175,6 +175,9 @@
return
cache = fastmanifestcache.getinstance(repo.store.opener, ui)
+ if background:
+ ui.debug("hi")
+
computedrevs = scmutil.revrange(repo, revset)
sortedrevs = sorted(computedrevs, key=lambda x:-x)
if ui.configbool("fastmanifest", "randomorder", True):
diff --git a/tests/test-fastmanifest.t b/tests/test-fastmanifest.t
--- a/tests/test-fastmanifest.t
+++ b/tests/test-fastmanifest.t
@@ -24,6 +24,7 @@
> [fastmanifest]
> cachecutoffdays=-1
> randomorder=False
+ > silentworker=False
> EOF
$ mkcommit a
@@ -52,7 +53,7 @@
$ sleep 1
$ hg debugcachemanifest -a --background
$ hg debugcachemanifest -a --background
- $ hg debugcachemanifest -a --background
+ $ hg debugcachemanifest -a --background --debug
$ sleep 1
$ hg debugcachemanifest --list
fast7ab5760d084a24168f7595c38c00f4bbc2e308d9 (size 328 bytes)
```
With this change, we see the "hi". Without this change, we do not see the "hi"
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3387112
Signature: t1:3387112:1464987504:dc44129afb36e27c84dd6d6e52db8d82533b4341
Summary:
This patch adds a new extension: sampling. It allows us to collect
aggregated data to a log file for each command that runs.
We will use it with our wrapper to collect data on sqldirstate and fastmanifest.
The workflow:
- Wrapper creates a log file, passes it to mercurial through an environment variable
- Mercurial writes data to this log file
- The wrapper parses the log file and upload its content to a data logging backend
Test Plan: Add a test
Reviewers: mitrandir, wez, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3360988
Summary: Since the `cacheallocation` already has the defaults, we only need to exclude the key from the config if we want the default.
Test Plan: run unit tests.
Reviewers: lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356592
Signature: t1:3356592:1464904353:303a7dd339f0c1d3a76e050cfe6e268a32d26403
Summary:
BSD version of head does not support -n0 and it looks like the only purpose of
using it here is to hide stdout output (and leave stderr).
Test Plan:
Ran test-tweakdefaults.t on both Linux and OSX, made sure that this particular
part of the test does not fail anymore.
That said, test-tweakdefaults.t still fails on OSX, but that requires more
investigation and my 5-minute look did not reveal the reason.
Reviewers: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3378544
Summary:
Before this patch, we were passing None as ui for hybridmanifest's
constructor in test-fastmanifest.py. With this patch, we pass a dummy ui object.
Therefore, it removes the need for None guard in hybridmanifest and simplifies
the code.
Test Plan: Tests pass
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3374069
Summary:
Before this patch we were not limiting the number of entries stored
in the in-memory cache. This patch adds a limit and makes it configurable.
As a followup we will need to add instrumentation.
Test Plan: Tests still pass
Reviewers: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3372517
Summary: We got all we wanted from this.
Test Plan: meh.
Reviewers: #mercurial, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3373068
Signature: t1:3373068:1464810628:ebfa809a0a02d6814c153c1b47354aae9a98dde1
Summary: This allow us to work regardless of order in which extensions are loaded.
Test Plan: ran some manual smoketests - there are no real integration tests yet
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3373452
Signature: t1:3373452:1464809280:0089c28818bab3acc4695b5c5411755a4505acf8
Summary:
"hg summary" is useful for hg developers. I forgot to add it while making
other changes to rage.py before. Let's add it now.
Test Plan: Run `hg rage --preview` and make sure it contains the `hg summary` section
Reviewers: #mercurial, simonfar, ttung
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3372197
Tasks: 10747644
Signature: t1:3372197:1464800008:ad6767b919bd4ae14cb2a6fdd814ba70df72a484
Summary:
For fbsource with sparse fbcode checkout this gets the conversion time
from 17s to 9.6s. Thanks @quark for showing me this option!
Test Plan: all tests passing
Reviewers: #mercurial, durham, ttung, quark
Reviewed By: quark
Subscribers: lcharignon, simonfar, mjpieters, quark
Differential Revision: https://phabricator.intern.facebook.com/D3365789
Signature: t1:3365789:1464712896:18ae59394b8e3591a8d55fed67f22921e1e4254f
Summary:
Removing the dirstate file from the filecache file list broke the dirstate
invalidation on non-sqldirsate repos while the extension is enabled. This is
turining the invalidation on only if we are really working with sqldirstate repo.
Test Plan:
tests didn't catch the broken behaviour - not sure when this would be a problem
in real life. Tests output didn't change.
Reviewers: #mercurial, quark, ttung, lcharignon, durham
Reviewed By: durham
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3365711
Signature: t1:3365711:1464738116:4c3446c6900f6b128cbcbd19811b743d3bd3ea06
Summary:
# We need to track the enlarged buffer capacity. If not, we will never enlarge the buffer again.
# The realloc was undersizing the buffer.
Test Plan:
successfully convert the opsfiles manifest using valgrind + test_convert_rt.
run `hg --config=extensions.fastmanifest= debugcache --all` on opsfiles-hg
Reviewers: wez, durham, lcharignon
Reviewed By: lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3369621
Signature: t1:3369621:1464792791:4bc83c39c0355644a0933b2ea3f544fd99894698
Summary:
As discussed, this will be an issues if we don't fix it before shipping
it to beta.
Test Plan: Test pass, lint is happy.
Reviewers: ttung
Differential Revision: https://phabricator.intern.facebook.com/D3368498
Summary:
@lcharignon hinted that relying on arguments position in format strings is
hard to read and confusing so I refactored it.
Also fixes one nit pointed by check-code
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters, lcharignon
Differential Revision: https://phabricator.intern.facebook.com/D3361466
Signature: t1:3361466:1464636235:5d979c3739255b4090ad73acae0eea2bd173b859
