mirror of
https://github.com/digital-asset/daml.git
synced 2024-11-10 00:35:25 +03:00
clear shared memory segment on macOS (#6530)
For a while now we've had errors along the line of ``` FATAL: could not create shared memory segment: No space left on device DETAIL: Failed system call was shmget(key=5432001, size=56, 03600). HINT: This error does *not* mean that you have run out of disk space. It occurs either if all available shared memory IDs have been taken, in which case you need to raise the SHMMNI parameter in your kernel, or because the system's overall limit for shared memory has been reached. The PostgreSQL documentation contains more information about shared memory configuration. child process exited with exit code 1 ``` on macOS CI nodes, which we were not able to reproduce locally. Today I managed to, sort of by accident, and that allowed me to dig a bit further. The root cause seems to be that PostgreSQL, as run by Bazel, does not always seem to properly unlink the shared memory segment it uses to communicate with itself. On my machine, running: ``` bazel test -t- --runs_per_test=100 //ledger/sandbox:conformance-test-wall-clock-postgresql ``` and eyealling the results of ``` watch ipcs -mcopt ``` I would say about one in three runs leaks its memory segment. After much googling and some head scratching trying to figure out the C APIs for managing shared memory segments on macOS, I kind of stumbled on a reference to `pcirm` in a comment to some low-ranking StackOverflow answer. It looks like it's working very well on my machine, even if I run it while a test (and therefore an instance of pg) is running. I believe this is because the command does not actually remove the shared memory segments, but simply marks them for removal once the last process stops using it. (At least that's what the manpage describes.) CHANGELOG_BEGIN CHANGELOG_END
This commit is contained in:
parent
9c4daad473
commit
55776f92ba
@ -139,6 +139,9 @@ jobs:
|
||||
steps:
|
||||
- template: ci/report-start.yml
|
||||
- checkout: self
|
||||
- bash:
|
||||
for shmid in $(ipcs -m | sed 1,3d | awk '{print $2}' | sed '$d'); do ipcrm -m $shmid; done
|
||||
name: clear_shm
|
||||
- bash: |
|
||||
set -euo pipefail
|
||||
git checkout $(release_sha)
|
||||
@ -240,6 +243,9 @@ jobs:
|
||||
steps:
|
||||
- template: ci/report-start.yml
|
||||
- checkout: self
|
||||
- bash:
|
||||
for shmid in $(ipcs -m | sed 1,3d | awk '{print $2}' | sed '$d'); do ipcrm -m $shmid; done
|
||||
name: clear_shm
|
||||
- template: ci/compatibility.yml
|
||||
parameters:
|
||||
test_flags: '--quick'
|
||||
|
@ -48,6 +48,10 @@ jobs:
|
||||
demands: assignment -equals default
|
||||
steps:
|
||||
- checkout: self
|
||||
- ${{ if eq(variables['pool'], 'macos-pool') }}:
|
||||
- bash:
|
||||
for shmid in $(ipcs -m | sed 1,3d | awk '{print $2}' | sed '$d'); do ipcrm -m $shmid; done
|
||||
name: clear_shm
|
||||
- template: ../compatibility.yml
|
||||
- template: ../daily_tell_slack.yml
|
||||
|
||||
|
Loading…
Reference in New Issue
Block a user