c3369a4 · reshape · git.iko.soy

git.iko.soy/reshape

Set `lock_timeout` to limit blocking of other queries

When attempting to acquire a lock for DDL queries, Postgres will wait for any long-running queries to complete first. It will also block any other queries that need access, which causes a queue of queries until we release our lock. This has the effect of long-running queries blocking other queries when we perform migrations, which is equivalent to downtime if the query runs for long enough. To avoid this, we set the `lock_timeout` setting. This puts an upper bound on how long we will wait for a lock and also for how long other queries can be blocked by a long-running query. The next step will be to add automatic retries to handle timeouts gracefully. Reference: https://medium.com/paypal-tech/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680

commitc3369a4c12db74065cd56cf4e18e83d6399088b4 authorfabianlindfors <fabian@flapplabs.se> dateTue 25 Jan 2022, 10:45 (5 years ago) parentf754ac0

1 file changed+19−2

compare with main

unifiedsplit

src/db.rs+19−2

src/db.rs+19−2

@@ -1,4 +1,4 @@

1−use anyhow::anyhow;

1+use anyhow::{anyhow, Context};

22use postgres::{types::ToSql, NoTls, Row};

44// DbLocker wraps a regular DbConn, only allowing access using the

@@ -22,7 +22,24 @@ impl DbLocker {

2222 const LOCK_KEY: i64 = 4036779288569897133;

2323

2424 pub fn connect(config: &postgres::Config) -> anyhow::Result<Self> {

25− let pg = config.connect(NoTls)?;

25+ let mut pg = config.connect(NoTls)?;

26+

27+ // When running DDL queries that acquire locks, we risk causing a "lock queue".

28+ // When attempting to acquire a lock, Postgres will wait for any long running queries to complete.

29+ // At the same time, it will block other queries until the lock has been acquired and released.

30+ // This has the bad effect of the long-running query blocking other queries because of us, forming

31+ // a queue of other queries until we release our lock.

32+ //

33+ // We set the lock_timeout setting to avoid this. This puts an upper bound for how long Postgres will

34+ // wait to acquire locks and also the maximum amount of time a long-running query can block other queries.

35+ // We should also add automatic retries to handle these timeouts gracefully.

36+ //

37+ // Reference: https://medium.com/paypal-tech/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680

38+ //

39+ // TODO: Make lock_timeout configurable

40+ pg.simple_query("SET lock_timeout = '1s'")

41+ .context("failed to set lock_timeout")?;

42+

2643 Ok(Self {

2744 client: DbConn::new(pg),

2845 })