Details
-
Bug
-
Resolution: Duplicate
-
Test Blocker
-
None
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Yes
Description
Build : 7.0.0-3216
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Iteration : 1 (seen within 15 mins of starting the test)
Memcached on multiple nodes (172.23.108.103, 172.23.97.239) starts crashing continuously. The following is seen in the info.log :
[ns_server:warn,2020-09-28T18:07:38.463-07:00,ns_1@172.23.108.103:<0.19444.6>:ns_memcached:connect:1167]Unable to connect: {throw,{error,{badmatch,{error,closed}}}}, retrying.
|
[user:info,2020-09-28T18:07:38.463-07:00,ns_1@172.23.108.103:<0.21439.0>:ns_log:crash_consumption_loop:69]Service 'memcached' exited with status 134. Restarting. Messages:
|
2020-09-28T18:07:38.356980-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fcb11fac000+0x11b496]
|
2020-09-28T18:07:38.357002-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fcb11fac000+0x135459]
|
2020-09-28T18:07:38.357023-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fcb11fac000+0x14a7cd]
|
2020-09-28T18:07:38.357052-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x4469c]
|
2020-09-28T18:07:38.357080-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x44c59]
|
2020-09-28T18:07:38.357101-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(_ZN9Couchbase6Thread12thread_entryEv+0xf) [0x7fcb10a74000+0x1b5df]
|
2020-09-28T18:07:38.357124-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fcb10a74000+0x10947]
|
2020-09-28T18:07:38.357137-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fcb0e2c2000+0x7dd5]
|
2020-09-28T18:07:38.357196-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fcb0def5000+0xfdead]
|
[*** LOG ERROR ***] [2020-09-28 18:07:38] [spdlog_file_logger] async log: thread pool doesn't exist anymore
|
Following is seen in the memcached logs :
2020-09-28T18:15:30.037514-07:00 INFO 107: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"19306739d306d12f/3e1f49349dfe4a9b"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.137","port":43840} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.037833-07:00 INFO 125: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"91e19e3de3a287e6/f682d603deeeebb1"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.61","port":34150} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.037842-07:00 INFO 115: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"d5ceed7f38da26bc/32bba2d0becb1739"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.87","port":48780} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.038100-07:00 INFO 122: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"dae5f4ed33003774/b85132e865456a92"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.61","port":34146} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.040097-07:00 INFO 82: Client {"ip":"172.23.99.23","port":33763} authenticated as <ud>Administrator</ud>
|
2020-09-28T18:15:30.040268-07:00 INFO 80: Client {"ip":"172.23.99.23","port":33760} authenticated as <ud>Administrator</ud>
|
2020-09-28T18:15:30.040401-07:00 INFO 84: Client {"ip":"172.23.104.67","port":51338} authenticated as <ud>@index</ud>
|
2020-09-28T18:15:30.040644-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-3216). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/496d2a5d-1b13-46fd-5d904c9d-b755de35.dmp before terminating.
|
2020-09-28T18:15:30.040672-07:00 CRITICAL Stack backtrace of crashed thread:
|
2020-09-28T18:15:30.040975-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x19422d]
|
2020-09-28T18:15:30.041022-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1aa5aa]
|
2020-09-28T18:15:30.041050-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x1aa8e8]
|
2020-09-28T18:15:30.041066-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f8c46626000+0xf5d0]
|
2020-09-28T18:15:30.041116-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7f8c46259000+0x36207]
|
2020-09-28T18:15:30.041156-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f8c46259000+0x378f8]
|
2020-09-28T18:15:30.041238-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f8c46d5b000+0x91195]
|
2020-09-28T18:15:30.041276-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1a4f72]
|
2020-09-28T18:15:30.041324-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c46d5b000+0x8ef86]
|
2020-09-28T18:15:30.041394-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c46d5b000+0x8efd1]
|
2020-09-28T18:15:30.041452-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8c46d5b000+0x8f213]
|
2020-09-28T18:15:30.041531-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x37) [0x7f8c46d5b000+0xb7867]
|
2020-09-28T18:15:30.041564-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x1f1887]
|
2020-09-28T18:15:30.041613-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x237202]
|
2020-09-28T18:15:30.041631-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x23d267]
|
2020-09-28T18:15:30.041648-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x23d7d2]
|
2020-09-28T18:15:30.041667-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x23d915]
|
2020-09-28T18:15:30.041709-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x191425]
|
2020-09-28T18:15:30.041729-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x1ac98a]
|
2020-09-28T18:15:30.041764-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x1f5b2f]
|
2020-09-28T18:15:30.041789-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x19d2de]
|
2020-09-28T18:15:30.041811-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x11b496]
|
2020-09-28T18:15:30.041834-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x135459]
|
2020-09-28T18:15:30.041861-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7f8c4a310000+0x14a7cd]
|
2020-09-28T18:15:30.041897-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x4469c]
|
2020-09-28T18:15:30.042159-07:00 INFO 89: Client {"ip":"172.23.104.69","port":51900} authenticated as <ud>@index</ud>
|
2020-09-28T18:15:30.042248-07:00 INFO 88: Client {"ip":"172.23.96.253","port":38836} authenticated as <ud>@index</ud>
|
2020-09-28T18:15:30.042382-07:00 INFO 78: Client {"ip":"172.23.99.23","port":33759} authenticated as <ud>Administrator</ud>
|
2020-09-28T18:15:30.042503-07:00 INFO 82: HELO [cbdatasource-sg:db_import_11d008cf60054d8b_7332d0c7-4bbe05b6] XATTR, XERROR [ {"ip":"172.23.99.23","port":33763} - {"ip":"172.23.108.103","port":11210} (<ud>Administrator</ud>) ]
|
2020-09-28T18:15:30.042676-07:00 INFO 81: Client {"ip":"172.23.99.23","port":33762} authenticated as <ud>Administrator</ud>
|
2020-09-28T18:15:30.042776-07:00 INFO 80: HELO [cbdatasource-sg:db_import_11d008cf60054d8b_f47365c5-5519f44b] XATTR, XERROR [ {"ip":"172.23.99.23","port":33760} - {"ip":"172.23.108.103","port":11210} (<ud>Administrator</ud>) ]
|
2020-09-28T18:15:30.042804-07:00 INFO 97: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"53d74f0725403ac4/acf136a5862e46b5"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.61","port":34120} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.042892-07:00 INFO 85: Client {"ip":"172.23.104.70","port":45790} authenticated as <ud>@index</ud>
|
2020-09-28T18:15:30.041918-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x44c59]
|
2020-09-28T18:15:30.042956-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(_ZN9Couchbase6Thread12thread_entryEv+0xf) [0x7f8c48dd8000+0x1b5df]
|
2020-09-28T18:15:30.042967-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f8c48dd8000+0x10947]
|
2020-09-28T18:15:30.042978-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f8c46626000+0x7dd5]
|
2020-09-28T18:15:30.043036-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f8c46259000+0xfdead]
|
2020-09-28T18:15:30.043107-07:00 INFO 78: HELO [cbdatasource-SG-v-2.7-commit--uuid-522850ba-01ef-11eb-ab73-82a0edc6c50c] XATTR, XERROR [ {"ip":"172.23.99.23","port":33759} - {"ip":"172.23.108.103","port":11210} (<ud>Administrator</ud>) ]
|
2020-09-28T18:15:30.043121-07:00 INFO 98: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"33f6002664afcfc4/54d1b1ba424959c7"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.137","port":43838} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.043424-07:00 INFO 103: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"b8dfca56153dba6f/aed5683ef3638181"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.87","port":48776} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.043884-07:00 INFO 119: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"0674c0b0c00785d8/5a790c435aa5db7c"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.61","port":34140} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.043896-07:00 INFO 120: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"6b7321df5375c64e/0621a5e5cc9b6a41"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.61","port":34142} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.044274-07:00 INFO 81: HELO [cbdatasource-sg:db_import_11d008cf60054d8b_80d40edf-34297e25] XATTR, XERROR [ {"ip":"172.23.99.23","port":33762} - {"ip":"172.23.108.103","port":11210} (<ud>Administrator</ud>) ]
|
2020-09-28T18:15:30.044290-07:00 INFO 126: HELO [{"a":"gocbcore/v9.0.6 gocb/v2.1.5","i":"23df07603cb3cd7a/47c7eb2361637ce6"}] Mutation seqno, XATTR, XERROR, Select bucket, JSON, Tracing, AltRequestSupport, SyncReplication, Collections, SubdocCreateAsDeleted [ {"ip":"172.23.104.87","port":48784} - {"ip":"172.23.108.103","port":11210} (not authenticated) ]
|
2020-09-28T18:15:30.047680-07:00 INFO 83: Client {"ip":"172.23.99.23","port":33764} authenticated as <ud>Administrator</ud>
|
2020-09-28T18:15:30.048867-07:00 INFO 62: Client {"ip":"172.23.104.137","port":43828} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.048882-07:00 INFO 121: Client {"ip":"172.23.104.61","port":34144} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.049021-07:00 INFO 74: Client {"ip":"172.23.104.87","port":48762} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.049707-07:00 INFO 101: Client {"ip":"172.23.104.87","port":48768} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.049877-07:00 INFO 115: Client {"ip":"172.23.104.87","port":48780} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.049899-07:00 INFO 100: Client {"ip":"172.23.104.87","port":48766} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.050117-07:00 INFO 112: Client {"ip":"172.23.104.61","port":34148} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.050815-07:00 INFO 94: Client {"ip":"172.23.104.137","port":43832} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.051500-07:00 INFO 99: Client {"ip":"172.23.104.87","port":48764} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.052068-07:00 INFO 103: Client {"ip":"172.23.104.87","port":48776} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.052105-07:00 INFO 77: Client {"ip":"172.23.104.137","port":43830} authenticated as <ud>@eventing</ud>
|
2020-09-28T18:15:30.054745-07:00 INFO ---------- Closing logfile
|
This is a regression from 7.0.0-3154 when the same test was run and this issue wasn't seen.
Attachments
Issue Links
- duplicates
-
-
- Closed
-