Summary:
Strictly speaking, it doesn't care if it's the length is SHA1_BYTES * 2. It just has to be even.
Also, size_t comes from stddef.h.
Test Plan: make local
Reviewers: durham, lcharignon
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3626828
Summary:
We are going to use some `C11` features, namely unnamed union fields.
Sadly we still have to support gcc4.4, which does not support `-std=c11` while
its `-std=gnu99` compiles the code but with the warning
`ISO C doesn't support unnamed structs/unions`.
This patch adds a feature test function to help detect compiler differences.
Test Plan:
```
make local
```
Reviewers: #mercurial, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3649021
Signature: t1:3649021:1470082430:bced7bd155eb60cc8c3e1c2d950092e2abd39a00
Summary:
to debug the sql performance we need a to trace the sql queries. This hacky module
facilitates that need.
Test Plan:
[mitrandir@dev3277 fbsource (92e442f)]$ xhg --config sqldirstate.tracefile=trace st
? fbcode/eden/x.py
[mitrandir@dev3277 fbsource (92e442f)]$ cat trace |arc paste
Reading paste from stdin...
P56530356: https://phabricator.intern.facebook.com/P56530356
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3626545
Summary:
The low cardinality of the status column makes it basically
non-indexable (try to do a binsearch in a sorted array with
1m numbers from the range {1..5} and you will see why this approach
doesn't work).
There is a feature we could use in sqlite 3.9 called partial indexes - so
we could index only the files that have non-normal statuses.
The sqlite 3.6 that we are using on centos6 unfortunately doesn't support it.
IMO the cost of shipping and maintaining our separate version of sqlite is
too high so let's simulate it by keeping those entries in separate table.
Test Plan:
* tests are passing
* on fbsource/fbcode "hg st" went down from ~800ms to ~600ms.
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3559244
Summary: It adds basic support for upgrading the db schema in sqldirsate.
Test Plan: tests passing
Reviewers: #mercurial, durham, ttung, rmcelroy, quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3567032
Summary:
We need to get the `.py` file name from `__file__`, which could be `.pyc`
and `.pyo`. Previously only `.pyc` is handled. This diff handles `.pyo`
as well.
Test Plan: Run existing tests
Reviewers: ttung, mjpieters, #mercurial, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3627929
Signature: t1:3627929:1469625182:1c0f5f720fe7eb4cbe3e6abd8e9407ced617d2a0
Test Plan: run tree_dump on a fastmanifest file. did some spot checks against the text manifest with the same sha (`hg debugdata -m`)
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3585843
Tasks: 12119301
Signature: t1:3585843:1469050470:70729d5601f935f2de80f89a7ad83590bcb8dd00
Summary:
Because we now prune in parallel to priming the cache, we no longer need the prune at the end. The only scenario where we still need to prune is where we never enter the priming loop, i.e., when the revset is empty.
Depends on D3545267, D3544997
Test Plan: since we don't prune in many circumstances any more, the test output is slighty affected. otherwise, the tests pass.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: durham, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3547219
Signature: t1:3547219:1468373775:e4535e3a66fb362385a23b992749eb7c3fcf7ea1
Calling wrapfunction on the remotefilepeer(sshpeer) object in exchangepull
function introduces a reference cycle. Hence, this object will not be deleted
until the process dies. This is not a big issue for processes having a short
lifetime(e.g. lauched by command line.)
However, for persistent processes (e.g. TortoiseHg), this can lead to multiple
lingering ssh connections to the server(actually one by pull operation).
The fix is to not wrap the remotefilepeer._callstream. This method is defined
right into the remotefilepeer object. The required repo data is made available
in the remotefilepeer object by monkeypatching this object in the exchangepull
function.
In some situations the remotefilelog setup logic could be called, which will
wrap certain functions, and then later a call will happen to a repo that wasn't
remotefilelog which will run some remotefilelog code because of the wrapping.
Normally we take care of this by checking for the remotefilelog requirement. We
missed it in this one spot though.
Summary:
During chg pull or push over ssh, ssh is started by chgserver which does not
have a controlling tty. Therefore the ssh process won't be able to ask for
passwords interactively.
This is actually a hard issue because an unprivileged process without a ctty
cannot attach to a ctty of another process.
The discussion at upstream tends to make it clear it's part of limitations
of chg. Therefore if we decide to workaround it, it has to live outside core,
thus fb-hgext.
GUI ssh-askpass is actually a good and clean choice. See D3510178 and D3515604.
However, they are for OS X but not Linux.
This diff is a very hacky solution to make ssh-askpass works in terminal.
It starts a "tty server" providing tty I/O fds and set `SSH_ASKPASS` to use a
custom script talking to the "tty server".
Test Plan:
Run the new test. Start a sshd locally and try:
```
$ hg push ssh://root@localhost/tmp
pushing to ssh://root@localhost/tmp
root@localhost's password:
remote: Permission denied (publickey,password).
abort: no suitable response from remote hg!
$ chg push ssh://root@localhost/tmp
pushing to ssh://root@localhost/tmp
==== SSH Authenticating ====
root@localhost's password:
remote: Permission denied (publickey,password).
abort: no suitable response from remote hg!
```
Reviewers: #mercurial, ttung, mpm
Reviewed By: mpm
Subscribers: durham, mpm, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3577509
Tasks: 12029680
Signature: t1:3577509:1469467700:cd93565bd47e535bb4cb41fcdaa39e45dddfae28
Summary:
All of hg journal is now fully upstreamed to mercurial core and remotenames.
Remove the outdated copy here.
Test Plan: --
Reviewers: #mercurial, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3611123
Tasks: 10804988
Summary: This is more robust and keeps the style consistent throughout
Test Plan: ran tests against zsh and bash
Reviewers: #sourcecontrol, ttung, zamsden
Reviewed By: zamsden
Subscribers: zamsden, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3597346
Signature: t1:3597346:1469073711:de76132bd9c161242d8d8171da765ddb54a806a2
Summary:
The old scm-prompt uses this longer, less awesome name.
Let's be backwards compatible with this and all that jazz.
Test Plan: tests pass under bash and zsh
Reviewers: #sourcecontrol, ttung, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3597315
Signature: t1:3597315:1469153052:98600400bb3cabf561294fd62f7cc601c280a1bd
Summary:
Previously, aliases could screw up scm-prompt. We had seen earlier
issues with this (thus the --color=never), but this is a more generic and
robust solution, applied everywhere.
We can't use full paths because they differ on different hosts, so we still
rely on a reasonable $PATH, as we always have.
Test Plan: ran scm-prompt tests under bash and zsh
Reviewers: #sourcecontrol, ttung, zamsden
Reviewed By: zamsden
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3597300
Tasks: 12298139
Signature: t1:3597300:1469060969:e88be6bd79234167f039f2964117a738ffbb95f5
Summary:
Since statprof is not an extension, it needs to be treated like a normal python
module.
Test Plan:
```
~/local/fb-hgext> python setup.py build
~/local/fb-hgext> ls build/lib.linux-x86_64-2.6/
cfastmanifest.so hgext3rd/ sqldirstate/
fastmanifest/ phabricator/ statprof.py
```
Reviewers: ttung, #sourcecontrol, quark
Reviewed By: quark
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3607202
Signature: t1:3607202:1469216683:9aa08aee567425c239ddc5d1bb68fd892a8cf6a9
Summary: vfs.join(None) will return its base. doesn't matter if it's nested or whatever.
Test Plan:
cloned fbjava, then stripped a revision (with evolve off). then added a new revision, and ran `hg incoming`. this resulted in the same stacktrace!
then with the fix run `hg incoming` again. no crash this time.
Reviewers: #fastmanifest, simonfar
Reviewed By: simonfar
Subscribers: mitrandir, simonfar, mjpieters, lcharignon
Differential Revision: https://phabricator.intern.facebook.com/D3598348
Tasks: 12305684
Signature: t1:3598348:1469093045:440cf3d314589f46dd8083901b584d7cfd54a95a
Summary:
Let's use deltas between the subsequent samples instead of sample count
to count the time spent.
Rationale:
When the process is IO blocked the other thread doing the sampling can be waken
up much more often (GIL is not held) causing the profiler to collect much more
samples in that state.
Test Plan:
works in my sandbox
do we have any tests for statprof.py?
Reviewers: durham
Reviewed By: durham
Subscribers: quark, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D3551431
Signature: t1:3551431:1468543377:f6b86245c957dd59d0334dbb1898f6ad0bb8c617
Summary:
`import hgsubversion` can error out if demandimport is disabled and svn
bindings are not found. In that case, we should be able to continue and
just skip handling svn revisions.
Test Plan: Code review
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3594810
Signature: t1:3594810:1469042572:3f9ab69f5503f6925f455769a78dac42a47087de
Summary: Sometimes opener.vfs is an _fncachevfs, which has a .vfs member, not a .base member. When you get one of these, go deeper.
Test Plan:
Run testrunner.py flib/intern/sandcastle/vcs/ without this change, see it traceback.
Rerun testrunner.py flib/intern/sandcastle/vcs/ with this change, see it fail in the same way as the current released version.
Reviewers: durham, lcharignon, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D3593097
Tasks: 12305684
Signature: t1:3593097:1469033487:aec4c4a5de6f92a64f4730f03e9780e02f086819
Summary:
The `.py` tests are different from `.t` ones. They need special care about
`import`.
Also adds unlink `socketpath` to make sure the `.py` file runs directly
with python with additional effort to clean up the test dir.
Test Plan:
```
unset PYTHONPATH
run-tests.py test-patchpython.py
python2 test-patchpython.py
```
Reviewers: ttung, #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3579579
Signature: t1:3579579:1468848206:3711a4714080fbcc2d4360ded8316603be48fa25
In two place, we were checking if a revlog was an instance of
revlog.revlog and, I think, treating it as a
remotefilelog.remotefilelog otherwise. I noticed this when I created
another non-revlog.revlog revlog in narrowhg and remotefilelog thought
it was a remotefilelog.remotefilelog. Let's specifically check if it's
a remotefilelog.remotefilelog instead.
Summary:
From sqlite manual:
With synchronous OFF (0), SQLite continues without syncing as soon as it
has handed data off to the operating system. If the application running
SQLite crashes, the data will be safe, but the database might become
corrupted if the operating system crashes or the computer loses power
before that data has been written to the disk surface. On the other hand,
commits can be orders of magnitude faster with synchronous OFF.
I believe this is a proper tradeoff for dirstate as we can regenerate it easily if db becomes corrupted.
Test Plan: tests are passing
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3559167
Summary: We can be more lazy - it's propertycache so it will populate itself when needed.
Test Plan: tests are passing
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3558232
Signature: t1:3558232:1468460942:e5e03743ec9a33cdb5da6d5f7541af28316f6c34
Summary:
The tests that are modifying the hgrc are blacklisted for
sqldirstate because they are effectively switching exitension off.
Test Plan: tests are passing
Reviewers: #mercurial, ttung, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3559268
Signature: t1:3559268:1468453765:38140f2ab4c392dc2aeefec230cbd1d4bb1b7170
Summary:
It's a common mistake that our tests require foreign extensions (namely evolve
and remotenames) without checking them first.
This diff adds checks to catch these mistakes, adds missing checks, and unifies
our checking logic using `require-ext.sh`, which is aware of `hgext3rd` and
prints skip message.
This affects `arc lint` so hopefully our new testing code would be free of this
kind of mistakes.
Test Plan: `arc lint` would catch errors
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3550977
Signature: t1:3550977:1468455857:e849dfd9e3cbc446cc6e6c662050ee88a3366e6c
Summary: Rather than have fastmanifestcache as a singleton, we attach it to each opener.
Test Plan: pass existing unit tests.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3553542
Tasks: 12169797
Signature: t1:3553542:1468373683:b0a7fcdf5dbf12046e5b4be9aa0bc5d46e55ce06
Summary: `prune` is essentially `makeroomfor(0, set())`.
Test Plan: pass existing unit tests. output is slightly different, but that's just because we no longer output that debugging line.
Reviewers: #fastmanifest, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3547069
Signature: t1:3547069:1468352305:6185132648f871e26d3cfd449c059523b5eb6d6b
Summary:
When running large repack operations, the resident size of the process
could become quite large, since we're scanning in entire pack files. Linux/OSX
have api calls for telling the kernel it's ok to release some of that memory,
but those apis are not exposed to python.
So instead, let's unmap and remap the mmap's once a certain amount of data has
been read. I also tried changing the mmap accessors to use the file oriented api
(mmap.read(), mmap.seek(), etc) so we could switch to actual file handles during
repack, but it had a drastic affect on normal performance (repack took 1 hour
instead of a few minutes).
Long term we should move all of this logic to c++ so we can use the more
powerful APIs.
Test Plan:
Did a full repack on a laptop and verified memory capped out at 2GB
instead of exceeding 5GB.
Reviewers: #sourcecontrol, ttung
Differential Revision: https://phabricator.intern.facebook.com/D3545171
Summary: It's caught by the contbuild script. I forgot to change this test.
Test Plan: Run `test-patchpython.t`.
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3550459
Signature: t1:3550459:1468349937:07123651fb103d283a382d1323d9f69a5c5d81b7
Blame Revision: D3534311
Summary:
Be a better citizen under system python path.
Fix all tests issues and change setup.py to use glob pattern to include
all extensions.
Test Plan:
Run tests and `make local`.
Also build and install the package and run `hg sl` in major repos.
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3534311
Signature: t1:3534311:1468275426:fe122646c8bd6c541e1889e73e9df28f86747ff2
Summary:
It's kind of silly to write the FM cache entry, and then immediately blow away the entry if it doesn't fit in the limit. Instead, just calculate if we have enough space, and raise CacheFullException if we don't.
Depends on D3538944
Test Plan: pass existing tests.
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters, quark
Differential Revision: https://phabricator.intern.facebook.com/D3545446
Signature: t1:3545446:1468283437:7e4167e3b5a13bf3d01fca45a965d7aaa8843166
Summary:
There's two issues with `setwithlimit`:
1) It does not have a docblock, nor an API that fully covers all the possible conditions.
2) It returns None if the cache entry is already present.
This modifies `setwithlimit` to return True if the cache entry ultimately makes it (whether it previously existed or the write was successful), False if the write fails, and raises CacheFullException if the cache is full.
Test Plan: pass existing tests. the already-in-cache case is a race condition that is not practical to reproduce.
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3545407
Signature: t1:3545407:1468283390:64d6cdba7a17cca9bd4465c6e03b9143e4b4cc4b
Summary:
Instead of pruning at the end, we need to prune as we are writing entries.
Depends on D3545060
Test Plan: pass test in D3545060
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3545267
Signature: t1:3545267:1468279898:5332838efa52bb55733d0e6dcf613b27e71e9d3e
Summary: Currently, this test breaks! This is awful because that means for users, when the cache becomes full, we never accept new manifests. :( :( :(
Test Plan: it breaks. :( did i say :( ?
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3545060
Tasks: 12136039
Signature: t1:3545060:1468279919:ff8928557c8ad03e06370ee75f2386ffb46f54fa
Summary:
Previously, depending on the code path, the limit specified would not actually take effect. For instance, if we came in from debugmanifestcache, and attempted to populated the cache, we would use `systemawarecachelimit` when filling the cache, and the fixedsize limit specified by the user when pruning.
With this change, we unify the all the cache limit decisions to `fastmanifestcache`. If the user actually overrides the limit, we set the limit in `fastmanifestcache` and let that make the decisions.
We also change the definitions of limit in `hg debugcachemanifest` to:
1) >0 => it's the limit.
2) =0 => use systemawarecachelimit
3) <0 => no limit!
Test Plan: pass existing unit tests. there's a small change in the test output, because we always evaluate the limit now, plus we remove the test for limit=0, since it means something different now.
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: trunkagent, mitrandir, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3544997
Signature: t1:3544997:1468281604:8f78f00ebf2afd8f3f1fbefbd82316b97cc4b193
Summary:
We need a fastmanifest object in order to size it. Once we know its size, we can make room in the cache.
This slightly affects one of the tests, as we request the manifest text earlier than we previously did.
Depends on D3537904
Test Plan: used in later diff.
Reviewers: lcharignon, durham
Reviewed By: durham
Subscribers: mitrandir, mjpieters, quark
Differential Revision: https://phabricator.intern.facebook.com/D3538991
Signature: t1:3538991:1468280648:41c65d91529babe0559eac7b75509481adf2765f
