facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2025-01-01 09:37:56 +03:00
Code Issues Projects Releases Wiki Activity
84cbc26b1e
sapling/eden/scm/tests/test-hgsql-sync-interval.t

130 lines
3.2 KiB
Perl
Raw Normal View History

tests: enable chg for 572 tests Summary: Add `#chg-compatible` to 572 tests that seem to pass with chg enabled. This should make them run faster. Reviewed By: xavierd Differential Revision: D18870507 fbshipit-source-id: fe895e733efffc9286cd3d17c7a156c803124395
2019-12-10 02:24:31 +03:00
#chg-compatible
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
Test hgsql tries to sync before entering the critical section
$ . "$TESTDIR/hgsql/library.sh"
tests: use disable treemanifest in place of setconfig Summary: Use `disable treemanifest` in place of `setconfig extensions.treemanifest=!` Reviewed By: quark-zju Differential Revision: D19457678 fbshipit-source-id: 47c8b3c4f4ed5cfc97275ca67afab4a86ffe7f0d
2020-01-20 13:42:49 +03:00
$ disable treemanifest
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
$ setconfig hgsql.verbose=1
$ newrepo state1
$ hg debugdrawdag << 'EOS'
> B
> |
> A
> EOS
$ newrepo state2
$ hg debugdrawdag << 'EOS'
> C
> |
> B
> |
> A
> EOS
$ newrepo state3
$ hg debugdrawdag << 'EOS'
> D
> |
> C
> |
> B
> |
> A
> EOS
$ newrepo state4
$ hg debugdrawdag << 'EOS'
> E
> |
> D
> |
> C
> |
> B
> |
> A
> EOS
$ cd $TESTTMP
$ initserver repo1 master
$ initserver repo2 master
$ initserver repo3 master
$ initserver repo4 master
Repo1: 2 commits. Sync them to the database.
$ cd $TESTTMP/repo1
$ hg pull -r tip $TESTTMP/state1
hgsql: log rows read and write Summary: This gives us more information about how large the hgsql operation is. As we're here, move `sqlwriteunlock` to an earlier place so the logging is outside the sql lock. Also document `unbundle` a bit so it's clear `repo.transaction()` only takes sql lock in `unbundle` context. Reviewed By: phillco Differential Revision: D10056307 fbshipit-source-id: 5d3361b4044e6fcf01e60409ef1ecb34da34ccac
2018-09-27 02:32:10 +03:00
[hgsql] got lock after * seconds (read 1 rows) (glob)
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
pulling from $TESTTMP/state1
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 2 files
drawdag: use bookmarks instead of tags Summary: A future diff will remove tags entirely, so let's move drawdag to be bookmark based. Reviewed By: quark-zju Differential Revision: D18995059 fbshipit-source-id: 70ef67259b37ef9821009d0145aa1e03c09b1309
2019-12-17 05:10:38 +03:00
adding remote bookmark A
adding remote bookmark B
[hgsql] held lock for * seconds (read 5 rows; write 10 rows) (glob)
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
Repo2: Emulate client push. Hold the lock for long.
$ cd $TESTTMP/repo2
$ hg --config hooks.pretxnclose.dely='sleep 6' pull -r tip $TESTTMP/state2 &>out &
$ disown
Repo 3: Emulate client push to sql, after repo2.
$ sleep 2
$ cd $TESTTMP/repo3
$ hg --config hooks.pretxnclose.dely='sleep 6' pull -r tip $TESTTMP/state3 &>out &
$ disown
Emulate writing to another repo when the lock was held elsewhere.
Explaination:
- The first "getting 2 commits from database" is because repo4 is empty, and the database has A,B synced from repo1.
- The second "getting 1 commits from database" is because repo2 push completes.
- The third "getting 1 commits from database" is because repo3 push completes.
$ cd $TESTTMP/repo4
$ setconfig hgsql.syncinterval=0.1 hgsql.debugminsqllockwaittime=13
$ hg pull -r tip $TESTTMP/state4
[hgsql] getting 2 commits from database
[hgsql] getting 1 commits from database
[hgsql] getting 1 commits from database
hgsql: log rows read and write Summary: This gives us more information about how large the hgsql operation is. As we're here, move `sqlwriteunlock` to an earlier place so the logging is outside the sql lock. Also document `unbundle` a bit so it's clear `repo.transaction()` only takes sql lock in `unbundle` context. Reviewed By: phillco Differential Revision: D10056307 fbshipit-source-id: 5d3361b4044e6fcf01e60409ef1ecb34da34ccac
2018-09-27 02:32:10 +03:00
[hgsql] got lock after * seconds (read * rows) (glob)
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
pulling from $TESTTMP/state4
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
drawdag: use bookmarks instead of tags Summary: A future diff will remove tags entirely, so let's move drawdag to be bookmark based. Reviewed By: quark-zju Differential Revision: D18995059 fbshipit-source-id: 70ef67259b37ef9821009d0145aa1e03c09b1309
2019-12-17 05:10:38 +03:00
adding remote bookmark E
[hgsql] held lock for * seconds (read * rows; write 8 rows) (glob)
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
Make sure repo2 and repo3 log looks sane.
$ cat $TESTTMP/repo2/out
[hgsql] getting 2 commits from database
hgsql: log rows read and write Summary: This gives us more information about how large the hgsql operation is. As we're here, move `sqlwriteunlock` to an earlier place so the logging is outside the sql lock. Also document `unbundle` a bit so it's clear `repo.transaction()` only takes sql lock in `unbundle` context. Reviewed By: phillco Differential Revision: D10056307 fbshipit-source-id: 5d3361b4044e6fcf01e60409ef1ecb34da34ccac
2018-09-27 02:32:10 +03:00
[hgsql] got lock after * seconds (read 1 rows) (glob)
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
pulling from $TESTTMP/state2
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
drawdag: use bookmarks instead of tags Summary: A future diff will remove tags entirely, so let's move drawdag to be bookmark based. Reviewed By: quark-zju Differential Revision: D18995059 fbshipit-source-id: 70ef67259b37ef9821009d0145aa1e03c09b1309
2019-12-17 05:10:38 +03:00
adding remote bookmark C
[hgsql] held lock for * seconds (read 9 rows; write 8 rows) (glob)
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
$ cat $TESTTMP/repo3/out
[hgsql] getting 2 commits from database
hgsql: log rows read and write Summary: This gives us more information about how large the hgsql operation is. As we're here, move `sqlwriteunlock` to an earlier place so the logging is outside the sql lock. Also document `unbundle` a bit so it's clear `repo.transaction()` only takes sql lock in `unbundle` context. Reviewed By: phillco Differential Revision: D10056307 fbshipit-source-id: 5d3361b4044e6fcf01e60409ef1ecb34da34ccac
2018-09-27 02:32:10 +03:00
[hgsql] got lock after * seconds (read 1 rows) (glob)
hgsql: sync repos before acquiring the SQL write lock Summary: When entering the critical section (creating a transaction), hgsql might wait for SQL lock for a long time (minutes). And it currently does nothing during that wait time. If the lock was acquired after a long time, the work of catching up with what the database already has might be too much. That is a waste within the precious hgsql lock held critical section. Instead of waiting for the lock for N seconds and doing nothing, wait for k (k < N) seconds and also try to sync with SQL periodically. This makes the repo closer to what the database has and can reduce work needed to pull from the database in the critical section. This adds CPU pressure to the MySQL tier. I added a config option to control it. Worse case, we can change `hgsql.syncinterval` to make the sync less frequent. As we're here, try document the configs. Reviewed By: phillco Differential Revision: D10002578 fbshipit-source-id: bd72d8225c919aa2bc62743de1e1d3f27cba606a
2018-09-26 02:04:12 +03:00
[hgsql] getting 1 commits from database
pulling from $TESTTMP/state3
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
drawdag: use bookmarks instead of tags Summary: A future diff will remove tags entirely, so let's move drawdag to be bookmark based. Reviewed By: quark-zju Differential Revision: D18995059 fbshipit-source-id: 70ef67259b37ef9821009d0145aa1e03c09b1309
2019-12-17 05:10:38 +03:00
adding remote bookmark D
[hgsql] held lock for * seconds (read 10 rows; write 8 rows) (glob)
Reference in New Issue Copy Permalink