Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.1
-
7.1.1-3030
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
QE TEST
-test tests/integration/neo/test_neo_magma_milestone4.yml -scope tests/integration/neo/scope_neo_magma.yml
|
Day - 1
Cycle - 1
Scale - 3
TEST STEP
Perform failover and rebalance out pair of KV nodes.
[2022-05-02T17:31:05-07:00, sequoiatools/couchbase-cli:7.1:7e0802] failover -c 172.23.108.139:8091 --server-failover 172.23.108.138:8091 -u Administrator -p password |
[2022-05-02T17:32:28-07:00, sequoiatools/couchbase-cli:7.1:c01f33] failover -c 172.23.108.139:8091 --server-failover 172.23.108.143:8091 -u Administrator -p password --hard |
[2022-05-02T17:32:37-07:00, sequoiatools/couchbase-cli:7.1:dd033d] rebalance -c 172.23.108.139:8091 -u Administrator -p password |
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.108.139:8091 -u Administrator -p password] |
|
|
docker logs dd033d
|
docker start dd033d
|
|
|
*Unable to display progress bar on this os |
JERROR: Rebalance failed. See logs for detailed reason. You can try again. |
OBSERVATION
Rebalance failed due to memcached/magma crash.
grep CRITICAL memcached.log on 172.23.107.25
2022-05-03T16:51:19.307675-07:00 CRITICAL (default) WarmupBackfillTask::run(): caught exception while running backfill - aborting warmup: kvstore-194/rev-000000001: found keyIndex seqno < seqIndex seqno (2071646 < 2073699) |
2022-05-03T16:57:27.143860-07:00 CRITICAL *** Fatal error encountered during exception handling *** |
2022-05-03T16:57:27.143927-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): kvstore-194/rev-000000001: found keyIndex seqno < seqIndex seqno (2071646 < 2073699) |
2022-05-03T16:57:29.724023-07:00 CRITICAL Detected previous crash |
2022-05-03T16:57:29.724112-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.1-3030). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/3ea660d2-ba86-4918-5eb49695-3f215dcf.dmp before terminating. |
2022-05-03T16:57:29.724116-07:00 CRITICAL Stack backtrace of crashed thread: |
2022-05-03T16:57:29.724118-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x7280e8] |
2022-05-03T16:57:29.724120-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x7798ba] |
2022-05-03T16:57:29.724122-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x779bf8] |
2022-05-03T16:57:29.724123-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fc3bba2b000+0xf630] |
2022-05-03T16:57:29.724125-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7fc3b9275000+0x36387] |
2022-05-03T16:57:29.724127-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7fc3b9275000+0x37a78] |
2022-05-03T16:57:29.724128-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0x9963c] |
2022-05-03T16:57:29.724130-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x7325bb] |
2022-05-03T16:57:29.724131-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xa48f6] |
2022-05-03T16:57:29.724131-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xa4961] |
2022-05-03T16:57:29.724133-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xa4bf4] |
2022-05-03T16:57:29.724134-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0x10e709] |
2022-05-03T16:57:29.724135-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x5d563f] |
2022-05-03T16:57:29.724136-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x487863] |
2022-05-03T16:57:29.724137-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x469c61] |
2022-05-03T16:57:29.724138-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x4be0ed] |
2022-05-03T16:57:29.724139-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x4c1a02] |
2022-05-03T16:57:29.724140-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x4c3471] |
2022-05-03T16:57:29.724141-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x4c3c14] |
2022-05-03T16:57:29.724142-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x6a1049] |
2022-05-03T16:57:29.724143-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x69a70a] |
2022-05-03T16:57:29.724144-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x6a23ee] |
2022-05-03T16:57:29.724145-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x7f9af0] |
2022-05-03T16:57:29.724146-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x7e458a] |
2022-05-03T16:57:29.724148-07:00 CRITICAL #24 /opt/couchbase/bin/memcached() [0x400000+0x7fcaa9] |
2022-05-03T16:57:29.724150-07:00 CRITICAL #25 /opt/couchbase/bin/memcached() [0x400000+0x69a404] |
2022-05-03T16:57:29.724151-07:00 CRITICAL #26 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc3b9b5d000+0xcdd40] |
2022-05-03T16:57:29.724152-07:00 CRITICAL #27 /lib64/libpthread.so.0() [0x7fc3bba2b000+0x7ea5] |
2022-05-03T16:57:29.724154-07:00 CRITICAL #28 /lib64/libc.so.6(clone+0x6d) [0x7fc3b9275000+0xfeb0d] |
Attachments
| For Gerrit Dashboard: MB-52046 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 174581,5 | MB-52046 magma: Handle empty value in ValueFormat | master | magma | Status: MERGED | +2 | +1 |
| 174944,3 | MB-52046 magma: Fix target level selection for L0 compactions | master | magma | Status: MERGED | +2 | +1 |
| 175314,3 | MB-52046 [BP] magma: Fix target level selection for L0 compactions | neo | magma | Status: MERGED | +2 | +1 |
| 175338,2 | MB-52046 [BP] magma: Handle empty value in ValueFormat | neo | magma | Status: MERGED | +2 | +1 |
| 175340,2 | MB-52046 test: Assert ValueFormat state after decoding empty Slice | master | magma | Status: MERGED | +2 | +1 |