Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Triaged
-
1
-
Yes
Description
Recent ASan-UBSan CV jobs running on the recently-added "large" CV machines have been failing due to the compilation running out of memory - for example:
05:40:02 [1261/1337] Linking CXX executable kv_engine/memcached_mcbp_test
|
05:40:02 FAILED: kv_engine/memcached_mcbp_test
|
...
|
05:40:02 clang: error: unable to execute command: Killed
|
05:40:02 clang: error: linker command failed due to signal (use -v to see invocation)
|
URL: http://cv.jenkins.couchbase.com/job/kv_engine.ASan-UBSan/job/master/17886/consoleFull
Looking at the machine in question (ubuntu18-cv-large-07), it has 16 cores but only 16GB RAM. As such, with PARALLELISM env var set to 16 there is potentially 16 ld link processes running concurrently. The oom-killer dmesg logs show mulitple instances where the killed linker processes had >1.5GB RSS at the point they were killed.
Attachments
| For Gerrit Dashboard: MB-45800 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 151637,3 | MB-45800: Use Ninja pools to limit concurrent link jobs | master | tlm | Status: MERGED | +2 | +1 |
| 151642,2 | MB-45800: Set CB_PARALLEL_LINK_JOBS to limit concurrent link jobs | master | server-cv | Status: MERGED | +2 | +1 |