Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
Unknown
-
Magma: Jan 20 - Feb 2, Magma: Apr2-Apr16
Description
1. Create a 2 node(220,221) cluster
2. Create a magma bucket
3. Load 1M items
4. Stop persistence on node 220. Load 100k items on nodeA so that they are replicated and presisted on node 221
5. Kill memcached on Node 220 which will trigger rollback on Node 221
6. Repeat steps 4,5 multiple times.
7. Magma crashed -> memcached crashed in a loop on node .221
Logs:
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.565789-08:00 CRITICAL *** Fatal error encountered during exception handling ***
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.565858-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): FailoverTable::pruneEntries: cannot prune up to seqno 862 as it would result in less than one element in failover table
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725466-08:00 CRITICAL Breakpad caught a crash (Couchbase version 6.5.0-42). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/7dfd8c5a-2577-77c2-4d127aad-32396f37.dmp before terminating.
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725484-08:00 CRITICAL Stack backtrace of crashed thread:
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725668-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x13246d]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725679-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14a4ae]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725689-08:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14a7c4]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725696-08:00 CRITICAL /lib64/libpthread.so.0() [0x7f53ac060000+0xf5d0]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725720-08:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7f53abc93000+0x36207]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725739-08:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f53abc93000+0x378f8]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725771-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f53ac795000+0x91195]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725780-08:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x145da2]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725833-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f53ac795000+0x8ef86]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725849-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f53ac795000+0x8efd1]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725863-08:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f53ac795000+0x8f213]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725873-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f53a726d000+0x132157]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725889-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f53a726d000+0x18b5d3]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725898-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f53a726d000+0x15d768]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725958-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f53a726d000+0x8c68b]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725975-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f53a726d000+0x8c81a]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725982-08:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f53a726d000+0x12ed84]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725987-08:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f53ae63e000+0x8f17]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.725993-08:00 CRITICAL /lib64/libpthread.so.0() [0x7f53ac060000+0x7dd5]
|
/opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt:2020-02-03T20:31:36.726023-08:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f53abc93000+0xfdead]
|
If this looks like the magma bug please assign it to storage team.
NOTE: I have noticed this issue on a private build shared by Apaar Gupta present here: http://server.jenkins.couchbase.com/job/toy-unix/6411/