Summary: Upstream has added a new attribute to dirstate. Let's add it too.
Test Plan: Tests now pass
Reviewers: mitrandir, #mercurial, simonfar
Reviewed By: simonfar
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D4695923
Signature: t1:4695923:1489283439:c7876832af2764d2bbe9cd0f9e47199c84799609
Summary:
This diff was created when I was investigating hgamend exec not working
with histedit which resulted with [1].
This also brings some very small fixes for sqldirstate and fbamend that
I was too lazy to diff separately.
[1] https://patchwork.mercurial-scm.org/patch/16451/
Test Plan: ran tests
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3777784
Signature: t1:3777784:1472227075:f2f8a9cad22bf0c45d4554709846b8aafb61d9a0
Before this patch, if sqldirstate.fakedirstate configuration is false
value, (fake) dirstate isn't written out (or explicitly unlinked),
In such case, @filecache never detects change of dirstate by another
process/thread, because not-existing file never change its file stat.
To achieve checking cache validity without existence of .hg/dirstate
file, we should use another cache annotation instead of @filecache.
This patch removes code paths to avoid writing (or unlinking) fake
dirstate, and writes fake dirstate always to detect changes correctly.
Just removing these code paths seems reasonable enough, because it was
already confirmed at Mercurial 4.0 sprint that there is no concrete
usecase for sqldirstate.fakedirstate configuration.
With sqldirstate extension, size of .hg/dirstate is always fixed: node
ID x 2 + dummy fixed text.
This increases occurrence of file stat ambiguity, and
processes/threads running parallelly might overlook change of
dirstate, because reloading @filecache-ed property repo.dirstate is
avoided in such case. See wiki page below for detail about file stat
ambiguity.
https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan
This patch specifies checkambig=True at opening .hg/dirstate file for
writing, to avoid file stat ambiguity.
Strictly speaking, to enable sqldirstate with Mercurial earlier than
upcoming 4.0 (or ccbeace526ad), it should be examined whether
checkambig optional argument is available for opener or not, because
it has been available since 4.0 (or ccbeace526ad).
But current sqldirstate implementation depends on 2369a09e89de of
Mercurial, which has introduced dirstate._origpl after ccbeace526ad.
Therefor, changes in this patch are safe enough, unless removing
dependence on dirstate._origpl from sqldirstate.
Summary:
That will allow us to figure out the best one (or even configure it on
per-repo basis if we want)
Test Plan: ran tests
Reviewers: #mercurial, durham, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3723215
Signature: t1:3723215:1471373384:e2e68369b0a1a3a5e7d630ea97b33061e24832bf
Summary:
This diff together with patches introducing the mentioned callback in upstream
will fix the journal under sqldirstate.
Test Plan: journal tests passed, the rest is currently running - will update the test plan soon
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: durham, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3690878
Signature: t1:3690878:1470800077:3fe48e03bb65eaeed3a7326b6e7d392abe7efd8d
Summary:
to debug the sql performance we need a to trace the sql queries. This hacky module
facilitates that need.
Test Plan:
[mitrandir@dev3277 fbsource (92e442f)]$ xhg --config sqldirstate.tracefile=trace st
? fbcode/eden/x.py
[mitrandir@dev3277 fbsource (92e442f)]$ cat trace |arc paste
Reading paste from stdin...
P56530356: https://phabricator.intern.facebook.com/P56530356
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3626545
Summary:
The low cardinality of the status column makes it basically
non-indexable (try to do a binsearch in a sorted array with
1m numbers from the range {1..5} and you will see why this approach
doesn't work).
There is a feature we could use in sqlite 3.9 called partial indexes - so
we could index only the files that have non-normal statuses.
The sqlite 3.6 that we are using on centos6 unfortunately doesn't support it.
IMO the cost of shipping and maintaining our separate version of sqlite is
too high so let's simulate it by keeping those entries in separate table.
Test Plan:
* tests are passing
* on fbsource/fbcode "hg st" went down from ~800ms to ~600ms.
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3559244
Summary: It adds basic support for upgrading the db schema in sqldirsate.
Test Plan: tests passing
Reviewers: #mercurial, durham, ttung, rmcelroy, quark
Subscribers: quark, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3567032
Summary:
From sqlite manual:
With synchronous OFF (0), SQLite continues without syncing as soon as it
has handed data off to the operating system. If the application running
SQLite crashes, the data will be safe, but the database might become
corrupted if the operating system crashes or the computer loses power
before that data has been written to the disk surface. On the other hand,
commits can be orders of magnitude faster with synchronous OFF.
I believe this is a proper tradeoff for dirstate as we can regenerate it easily if db becomes corrupted.
Test Plan: tests are passing
Reviewers: #mercurial, ttung, durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3559167
Summary: We can be more lazy - it's propertycache so it will populate itself when needed.
Test Plan: tests are passing
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3558232
Signature: t1:3558232:1468460942:e5e03743ec9a33cdb5da6d5f7541af28316f6c34
Summary:
To fix sqldirstate perf in case of iterating through whole
dirstate we are introducing the possibility to cache it in memory when
neccessary.
Currently we are doing it only in the case when dirstate.walk() is called.
Test Plan:
./run-sqlitedirstate-test.py --hg ~/clowncopter/ -j 30 -t 1200
~/clowncopter/tests/run-tests.py -j 30
Reviewers: #mercurial, durham, ttung, lcharignon
Reviewed By: lcharignon
Subscribers: lcharignon, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3472402
Signature: t1:3472402:1466807592:a28bc4662cc157547b8b747c6be98faae35c63b5
Summary: This allow us to work regardless of order in which extensions are loaded.
Test Plan: ran some manual smoketests - there are no real integration tests yet
Reviewers: #mercurial, ttung, durham
Reviewed By: durham
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3373452
Signature: t1:3373452:1464809280:0089c28818bab3acc4695b5c5411755a4505acf8
Summary:
For fbsource with sparse fbcode checkout this gets the conversion time
from 17s to 9.6s. Thanks @quark for showing me this option!
Test Plan: all tests passing
Reviewers: #mercurial, durham, ttung, quark
Reviewed By: quark
Subscribers: lcharignon, simonfar, mjpieters, quark
Differential Revision: https://phabricator.intern.facebook.com/D3365789
Signature: t1:3365789:1464712896:18ae59394b8e3591a8d55fed67f22921e1e4254f
Summary: For full fbsource population is now 1m 20s (was 3m)
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung, quark
Reviewed By: quark
Subscribers: quark, simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356661
Signature: t1:3356661:1464703660:25be61b1170c52cf4e300490ea59afc21a575619
Summary:
This dirs datascture does a lot of queries about itself during its
construction. This speeds it up by doing the construction in memory
the uploading the whole dictionary to db at once.
Time for full population on fbsource went down form 5m to 3m.
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356658
Summary:
This uses the executemany for bulk INSERTS in db instead od doing many single
queries. This is also refactoring the sqldirstate conversion to use it.
Test Plan: ran tests
Reviewers: #mercurial, durham, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356654
Summary:
This is neccessary for scmprompt to work. I added a config knob because we
may want to disable it if we want to do paranoid mode at some point.
Test Plan: ran tests
Reviewers: #mercurial, durham
Differential Revision: https://phabricator.intern.facebook.com/D3356646
Summary: This is to make implementation of automatic upgrade easier.
Test Plan: Will add a test for it in one of the next diffs
Reviewers: #mercurial, durham, ttung
Subscribers: mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3356494
Summary:
the transaction won't be commited anyway.
The journal will be rolled back on next database open.
Test Plan: ran mercurial tests with sqldirstate
Reviewers: #mercurial, durham, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar, mjpieters
Differential Revision: https://phabricator.intern.facebook.com/D3345277
Signature: t1:3345277:1464168844:69206055d24ae0349985f0038845122e7cea0a19
Summary:
This causes the read-only dirstate operations to be readonly from
db standpoint so they can run while there is a pending transaction.
Test Plan: ran mercurial tests with sqldirstate
Reviewers: simonfar, durham
Reviewed By: simonfar, durham
Subscribers: simonfar, mjpieters, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D3345257
Signature: t1:3345257:1464168444:a98bc04c23e174d0c10ed07eb914930949a67b7d
Summary:
An extension replacing dirstate file with sqlite database so we can have incremental changes and we don't have to read the whole dirstate on every op. This makes sense only when hgwatchan/fsmonitor is on so we don't iterate through whole dirstate.
This is also using the sqlite transactions to handle dirstate transactions instead of copying db around. As a result of that "hg rollback" doesn't work anymore. You can fall back to copying things by setting sqldirstate.skipbackups to False.
Needs those to go to upstream to work: https://phabricator.intern.facebook.com/P56319612
(will send them once the freeze is over)
To use make sure that the extension is loaded *before* hgwatchman (watchman
should be outmost layer).
Test Plan:
Passing all but few mercurial tests (when running with skipbackups=False)
The failures are described in blacklist file.
Reviewers: lcharignon, wez, quark, durham
Reviewed By: durham
Subscribers: laurent, mjpieters, #sourcecontrol
Differential Revision: https://phabricator.intern.facebook.com/D3242547
Signature: t1:3242547:1462577481:fdbfb5287fb8d3e58f7b4d587c01de79ce6b78df
