Loading...

Details

Type: Bug
Resolution: Duplicate
Priority: Critical
Fix Version/s: 7.1.0
Affects Version/s: 7.1.0
Component/s: eventing
Labels:

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.106.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.106.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.106.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.151.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.74.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.106.134.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.106.136.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.106.137.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.58.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.73.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.74.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.75.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.81.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.120.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.121.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.24.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.26.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.31.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.123.33.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.14.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.243.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.254.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.96.48.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.105.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.110.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.149.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.150.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.151.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.241.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1628936959/collectinfo-2021-08-14T102921-ns_1%40172.23.97.74.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build : 7.1.0-1138
Test : -test tests/integration/neo/test_neo_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3_magma.yml -scope tests/integration/neo/scope_neo_with_backup_magma.yml
Scale : 3
Iteration : 1st

Seeing the following crash on 172.23.106.136

2021-08-14T02:00:53.722-07:00 [Info] [gocb] Will retry request. Backoff=500ms, OperationID=0x58de9. Reason=KV_ERROR_MAP_RETRY_INDICATED

2021-08-14T02:00:53.727-07:00 [Info] DCPT[eventing:20V03rv7-254:{eventing:20V03rv7-253:worker_timers_0_0_172.23.96.48:11210_172.23.106.136:8096}/0] ##2f DCP_STREAMEND for vb 47

2021-08-14T02:00:53.728-07:00 [Info] Consumer::dcpRequestStreamHandle [worker_sbm_0_0:/tmp/127.0.0.1:8091_0_3256247565.sock:16325] vb: 212 Updated checkpoint blob to indicate STREAMREQ was issued

2021-08-14T02:00:53.728-07:00 [Info] Consumer::processReqStreamMessages [worker_sbm_0_0:/tmp/127.0.0.1:8091_0_3256247565.sock:16325] vb: 212 DCP stream successfully requested

2021-08-14T02:00:53.744-07:00 [Info] DCPT[eventing:20V03rv7-256:{eventing:20V03rv7-255:worker_sbm_0_0_172.23.123.26:11210_172.23.106.136:8096}/0] ##d4 DCP_STREAMEND for vb 212

2021-08-14T02:00:53.746-07:00 [Info] [gocb] Will retry request. Backoff=500ms, OperationID=0x58dea. Reason=KV_ERROR_MAP_RETRY_INDICATED

fatal error: runtime: out of memory

runtime stack:

runtime.throw(0x14fa3dd, 0x16)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:774 +0x72

runtime.sysMap(0xc674000000, 0x104000000, 0x2141bb8)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mem_linux.go:169 +0xc5

runtime.(*mheap).sysAlloc(0x2129520, 0x104000000, 0x7f9cfa7fbc88, 0x964dd7)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/malloc.go:701 +0x1cd

runtime.(*mheap).grow(0x2129520, 0x82000, 0xffffffff)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1255 +0xa3

runtime.(*mheap).allocSpanLocked(0x2129520, 0x82000, 0x2141bc8, 0x0)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1170 +0x266

runtime.(*mheap).alloc_m(0x2129520, 0x82000, 0x7f9ca0df0101, 0x0)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1022 +0xc2

runtime.(*mheap).alloc.func1()

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1093 +0x4c

runtime.(*mheap).alloc(0x2129520, 0x82000, 0x7f9cfa010101, 0x7f9ca0dfa0a8)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1092 +0x8a

runtime.largeAlloc(0x103ffffff, 0x101, 0xe)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/malloc.go:1138 +0x97

runtime.mallocgc.func1()

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/malloc.go:1033 +0x46

runtime.systemstack(0x0)

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/asm_amd64.s:370 +0x66

runtime.mstart()

	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1146

Similar crashes are also seen on the other eventing nodes - 172.23.123.31 (at 2021-08-14T02:11:23) and 172.23.123.33 (at 2021-08-14T02:23:35)

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[System Test] Eventing crash "fatal error: runtime: out of memory" across multiple nodes

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty