Add rate limiting to read-only database syncs

Summary:
We've seen load issues on the mysql server in certain situations. This is caused
by large revision_reference query load from every read connection checking the
database.

To fix it, let's use Mercurial locks to enforce that only one reader is checking
and syncing at any given time.  All other will just serve the existing on disk
data immediately.

Writer clients are unaffected. They do not obey the rate limiting lock and do
exactly what they did before.

Test Plan: I'll try to add a test

Reviewers: #sourcecontrol, ttung

Differential Revision: https://phabricator.fb.com/D2756217

hgsql.py

@@ -441,8 +441,44 @@ def wraprepo(repo):
             outofsync = heads != sqlheads or bookmarks != sqlbookmarks or tip != len(self) - 1
             return outofsync, sqlheads, sqlbookmarks, tip
 
+        def synclimiter(self):
+            """Attempts to acquire the lock used to rate limit how many
+            read-only clients perform database syncs at the same time. If None
+            is returned, it means the limiter was not acquired, and readonly
+            clients should not attempt to perform a sync."""
+            try:
+                wait = False
+                return self._lock(self.svfs, "synclimiter", wait, None,
+                               None, _('repository %s') % self.origroot)
+            except error.LockHeld:
+                return None
+
         def syncdb(self, waitforlock=False):
-            ui = self.ui
+            """Attempts to sync the local repository with the latest bits in the
+            database.
+
+            If `waitforlock` is False, the sync is on a best effort basis,
+            and the repo may not actually be up-to-date afterwards. If
+            `waitforlock` is True, we guarantee that the repo is up-to-date when
+            this function returns, otherwise an exception will be thrown."""
+            if waitforlock:
+                return self._syncdb(waitforlock)
+            else:
+                # For operations that do not require the absolute latest bits,
+                # only let one process update the repo at a time.
+                limiter = self.synclimiter()
+                if not limiter:
+                    # Someone else is already checking and updating the repo
+                    self.ui.debug("skipping database sync because another "
+                                  "process is already syncing\n")
+                    return
+
+                try:
+                    return self._syncdb(waitforlock)
+                finally:
+                    limiter.release()
+
+        def _syncdb(self, waitforlock):
             if not self.needsync()[0]:
                 ui.debug("syncing not needed\n")
                 return

tests/test-race-conditions.t

@@ -26,9 +26,12 @@
 
   $ cd master2
   $ printf "[hooks]\npresyncdb.sleep = sleep 2\n" >> .hg/hgrc
-  $ hg log -l 2 --template "first:{rev}\n" &
+  $ hg log -l 2 --template "first:{rev}\n" --debug &
   $ sleep 0.2
-  $ hg log -l 2 --template "second:{rev}\n"
+  syncing with mysql
+  running hook presyncdb.sleep: sleep 2
+  $ hg log -l 2 --template "second:{rev}\n" --debug
+  skipping database sync because another process is already syncing
   second:0
   $ sleep 4
   first:1