Summary:
After first sqldirstate tests on real www users we found out that the "hg update" has significant regression.
We need a knob to disable it for users.
Test Plan: ran `hg pull`
Reviewers: #mercurial, jeroenv
Reviewed By: jeroenv
Subscribers: mjpieters, akushner
Differential Revision: https://phabricator.intern.facebook.com/D3980897
Signature: t1:3980897:1475753604:14712c2786d1245ab2cab5297ebcba83efe8f6ae
Summary:
That will allow us to figure out the best one (or even configure it on
per-repo basis if we want)
Test Plan: ran tests
Reviewers: #mercurial, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3723215
Signature: t1:3723215:1471373384:e2e68369b0a1a3a5e7d630ea97b33061e24832bf
Summary:
This diff together with patches introducing the mentioned callback in upstream
will fix the journal under sqldirstate.
Test Plan: journal tests passed, the rest is currently running - will update the test plan soon
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3690878
Signature: t1:3690878:1470800077:3fe48e03bb65eaeed3a7326b6e7d392abe7efd8d
Summary: Maintains the lookupcache across edit operations and builds the cache fully after 10,000 reads. This helps cap the overhead of doing sqlreads.
Test Plan:
Ran a large hg update followed by hg status and verified that the status call took 10 seconds instead of 60 seconds.
```
~/local/fbsource> time hg.real up master --profile && sleep 60 && sync && time hg.real status --pager=off
62093 files updated, 0 files merged, 6740 files removed, 0 files unresolved
real 2m24.262s
user 0m52.124s
sys 0m13.462s
real 1m2.942s
user 0m48.617s
sys 0m12.631s
~/local/fbsource> time hg.real up master~50000 --profile && sleep 60 && sync && time hg.real status --pager=off --config extensions.sqldirstate=../fb-hg
ext/sqldirstate/
49029 files updated, 0 files merged, 19804 files removed, 0 files unresolved
real 1m29.619s
user 0m47.244s
sys 0m12.264s
real 0m10.269s
user 0m7.964s
sys 0m0.842s
```
Reviewers: mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3659071
Signature: t1:3659071:1470177313:9af93af70fdbd5929b97879b58553a3cea415c19
Summary:
to debug the sql performance we need a to trace the sql queries. This hacky module
facilitates that need.
Test Plan:
[mitrandir@dev3277 fbsource (92e442f)]$ xhg --config sqldirstate.tracefile=trace st
? fbcode/eden/x.py
[mitrandir@dev3277 fbsource (92e442f)]$ cat trace |arc paste
Reading paste from stdin...
P56530356: https://phabricator.intern.facebook.com/P56530356
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3626545
Summary:
The low cardinality of the status column makes it basically
non-indexable (try to do a binsearch in a sorted array with
1m numbers from the range {1..5} and you will see why this approach
doesn't work).
There is a feature we could use in sqlite 3.9 called partial indexes - so
we could index only the files that have non-normal statuses.
The sqlite 3.6 that we are using on centos6 unfortunately doesn't support it.
IMO the cost of shipping and maintaining our separate version of sqlite is
too high so let's simulate it by keeping those entries in separate table.
Test Plan:
* tests are passing
* on fbsource/fbcode "hg st" went down from ~800ms to ~600ms.
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3559244
Summary: It adds basic support for upgrading the db schema in sqldirsate.
Test Plan: tests passing
Reviewers: #mercurial, durham, ttung, rmcelroy, quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3567032
Summary:
From sqlite manual:
With synchronous OFF (0), SQLite continues without syncing as soon as it
has handed data off to the operating system. If the application running
SQLite crashes, the data will be safe, but the database might become
corrupted if the operating system crashes or the computer loses power
before that data has been written to the disk surface. On the other hand,
commits can be orders of magnitude faster with synchronous OFF.
I believe this is a proper tradeoff for dirstate as we can regenerate it easily if db becomes corrupted.
Test Plan: tests are passing
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3559167
Summary: We can be more lazy - it's propertycache so it will populate itself when needed.
Test Plan: tests are passing
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3558232
Signature: t1:3558232:1468460942:e5e03743ec9a33cdb5da6d5f7541af28316f6c34
Summary:
To fix sqldirstate perf in case of iterating through whole
dirstate we are introducing the possibility to cache it in memory when
neccessary.
Currently we are doing it only in the case when dirstate.walk() is called.
Test Plan:
./run-sqlitedirstate-test.py --hg ~/clowncopter/ -j 30 -t 1200
~/clowncopter/tests/run-tests.py -j 30
Reviewers: #mercurial, durham, ttung, lcharignon
Reviewed By: lcharignon
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3472402
Signature: t1:3472402:1466807592:a28bc4662cc157547b8b747c6be98faae35c63b5
Summary: That is neccesary for this to work with extorder.
Test Plan: ~/clowncopter/tests/run-tests.py -j 30
Reviewers: #mercurial, ttung, durham, quark
Reviewed By: quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3472367
Signature: t1:3472367:1466633447:8b5af3f546be9d1f3634882f9638cfe5feb6c181
Summary: The iteritems makes sure that we do one big query instead of a query per item.
Test Plan:
./run-sqlitedirstate-test.py --hg ~/clowncopter/ -j 30 -t 1200
~/clowncopter/tests/run-tests.py -j 30
Reviewers: #mercurial, durham, ttung
Reviewed By: ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3472343
Signature: t1:3472343:1466642757:d29c569b718fbfbf2c9b030467f9a262c0c4078b
Summary:
The upstream change has now been *properly* landed at
https://selenic.com/repo/hg//rev/6ee50bdeeb32.
Once again, Mea Culpa, yesterday I apparently was in clowntown mode with these
patches.
Test Plan:
Run all affected tests:
$HGSRC/tests/run-tests.py -l test-pull-createmarkers* test-pushrebase* test-pushvars*
Reviewers: #sourcecontrol, ttung, quark
Reviewed By: quark
Subscribers: mjpieters, durham
Differential Revision: https://phabricator.intern.facebook.com/D3462832
Signature: t1:3462832:1466513182:58d6187c04e0e567a026aba90bf753c76f4949a7
Blame Revision: D3460645
Summary:
e61bd9d5e8ca requires a change to core that has been accepted but not yet
pushed. This breaks the build, so let's revert it until upstream pushes the
dependency.
Test Plan: Ran the tests
Reviewers: ttung, mjpieters, #sourcecontrol, mitrandir
Reviewed By: mitrandir
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3460645
Signature: t1:3460645:1466465904:cd521a989e475f9987237307a7d4d672d58ba615
Summary:
Now that my upstream patch for filedescriptor class access has landed the helper function to bypass the descriptor protocol can go altogether.
Depends on D3454878
Test Plan:
Run affected extension tests:
$HGSRC/tests/run-tests.py -l test-sqldirstate-* test-reflog* test-sparse*
Reviewers: #sourcecontrol, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3456368
Summary:
There were 3 clowntowny versions of the wrapfiledescriptor function, 2 being
dangerously wrong about how they scan the class hierarchy. All because
`classobj.descriptorname` invokes the descriptor protocol and the filecache
descriptor is not set up for that access.
Instead of one function to do it all, use one utility function to retrieve
the descriptor object from the class __dict__ to avoid the descriptor protocol
kicking in, and use mercurial.extensions.wrapfunction to do the actual
wrapping.
I've sent a patch upstream to fix accessing the filecache descriptor on the
class; if accepted we can drop the getfilecache utility function altogether,
leaving only the wrapfunction calls.
Test Plan:
Run affected extension tests:
$HGSRC/tests/run-tests.py -l test-sqldirstate-* test-reflog* test-sparse*
Reviewers: #sourcecontrol, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3454878
Summary: This allow us to work regardless of order in which extensions are loaded.
Test Plan: ran some manual smoketests - there are no real integration tests yet
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3373452
Signature: t1:3373452:1464809280:0089c28818bab3acc4695b5c5411755a4505acf8
Summary:
For fbsource with sparse fbcode checkout this gets the conversion time
from 17s to 9.6s. Thanks @quark for showing me this option!
Test Plan: all tests passing
Reviewers: #mercurial, durham, ttung, quark
Reviewed By: quark
Subscribers: lcharignon, simonfar, mjpieters, quark
Differential Revision: https://phabricator.intern.facebook.com/D3365789
Signature: t1:3365789:1464712896:18ae59394b8e3591a8d55fed67f22921e1e4254f
Summary:
Removing the dirstate file from the filecache file list broke the dirstate
invalidation on non-sqldirsate repos while the extension is enabled. This is
turining the invalidation on only if we are really working with sqldirstate repo.
Test Plan:
tests didn't catch the broken behaviour - not sure when this would be a problem
in real life. Tests output didn't change.
Reviewers: #mercurial, quark, ttung, lcharignon, durham
Reviewed By: durham
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3365711
Signature: t1:3365711:1464738116:4c3446c6900f6b128cbcbd19811b743d3bd3ea06
Summary:
@lcharignon hinted that relying on arguments position in format strings is
hard to read and confusing so I refactored it.
Also fixes one nit pointed by check-code
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters, lcharignon
Differential Revision: https://phabricator.intern.facebook.com/D3361466
Signature: t1:3361466:1464636235:5d979c3739255b4090ad73acae0eea2bd173b859
Summary: For full fbsource population is now 1m 20s (was 3m)
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung, quark
Reviewed By: quark
Subscribers: quark, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356661
Signature: t1:3356661:1464703660:25be61b1170c52cf4e300490ea59afc21a575619
Summary:
This dirs datascture does a lot of queries about itself during its
construction. This speeds it up by doing the construction in memory
the uploading the whole dictionary to db at once.
Time for full population on fbsource went down form 5m to 3m.
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356658
Summary:
This uses the executemany for bulk INSERTS in db instead od doing many single
queries. This is also refactoring the sqldirstate conversion to use it.
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356654
Summary:
This is neccessary for scmprompt to work. I added a config knob because we
may want to disable it if we want to do paranoid mode at some point.
Test Plan: ran tests
Reviewers: #mercurial, durham
Differential Revision: https://phabricator.intern.facebook.com/D3356646
Summary: I based my code on the generaldelta upgrade which also occured on pull.
Test Plan: test will follow in next comit
Reviewers: #mercurial, durham, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356590
Summary: This is to make implementation of automatic upgrade easier.
Test Plan: Will add a test for it in one of the next diffs
Reviewers: #mercurial, durham, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356494
Summary: I've done it wrong in one place so let's centralize that.
Test Plan: ran tests.
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3354724
Summary:
shelve is woking thanks to the hack where it abandons the transaction but
preserves the dirstate anyway by copying the dirstate to the side and restoring
it. We can do something that works as well and is way faster.
Test Plan:
ran tests
tried it on fbsource - it was nice and fast
Reviewers: #mercurial, ttung, simonfar, durham
Reviewed By: durham
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3345280
Signature: t1:3345280:1464170069:cfb22b7034f8a8c44fd4284cd35ba17e85dabb99
Summary:
the transaction won't be commited anyway.
The journal will be rolled back on next database open.
Test Plan: ran mercurial tests with sqldirstate
Reviewers: #mercurial, durham, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3345277
Signature: t1:3345277:1464168844:69206055d24ae0349985f0038845122e7cea0a19
Summary:
In the dirstate world we don't maintain in-memory cache over dirstate - there
is no need to invalidate it then.
Test Plan: ran mercurial tests with sqldirstate
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3345268
Signature: t1:3345268:1464169114:e55b709c4acbc89c4e2239f7534c6e13ef9b4861
Summary: I discovered that mercurial has better mechanics for extensions to use
Test Plan: ran mercurial tests with sqldirstate
Reviewers: #mercurial, durham, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3345260
Signature: t1:3345260:1464168562:32b09a0f86bc19b8622f8fe51ecb727ef3c44bbc
Summary:
This causes the read-only dirstate operations to be readonly from
db standpoint so they can run while there is a pending transaction.
Test Plan: ran mercurial tests with sqldirstate
Reviewers: simonfar, durham
Reviewed By: simonfar, durham
Subscribers: simonfar, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D3345257
Signature: t1:3345257:1464168444:a98bc04c23e174d0c10ed07eb914930949a67b7d
Summary:
An extension replacing dirstate file with sqlite database so we can have incremental changes and we don't have to read the whole dirstate on every op. This makes sense only when hgwatchan/fsmonitor is on so we don't iterate through whole dirstate.
This is also using the sqlite transactions to handle dirstate transactions instead of copying db around. As a result of that "hg rollback" doesn't work anymore. You can fall back to copying things by setting sqldirstate.skipbackups to False.
Needs those to go to upstream to work: https://phabricator.intern.facebook.com/P56319612
(will send them once the freeze is over)
To use make sure that the extension is loaded *before* hgwatchman (watchman
should be outmost layer).
Test Plan:
Passing all but few mercurial tests (when running with skipbackups=False)
The failures are described in blacklist file.
Reviewers: lcharignon, wez, quark, durham
Reviewed By: durham
Subscribers: laurent, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D3242547
Signature: t1:3242547:1462577481:fdbfb5287fb8d3e58f7b4d587c01de79ce6b78df
