Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
1006.5.1-1121-enterprise: http://latestbuilds.service.couchbase.com/builds/latestbuilds/couchbase-server/magma-preview/1121/
-
Untriaged
-
1
-
Unknown
Description
Step 1: Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.220 | kv | Cluster node |
|
| 172.23.105.221 | None | <--- IN --- |
|
| 172.23.105.223 | None | <--- IN --- |
|
| 172.23.105.225 | None | <--- IN --- |
|
| 172.23.105.226 | None | <--- IN --- |
|
| 172.23.105.227 | None | <--- IN --- |
|
| 172.23.105.208 | None | <--- IN --- |
|
+----------------+----------+--------------+
|
Step 2: Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.225 | kv | Cluster node |
|
| 172.23.105.226 | kv | Cluster node |
|
| 172.23.105.220 | kv | Cluster node |
|
| 172.23.105.223 | kv | Cluster node |
|
| 172.23.105.227 | kv | Cluster node |
|
| 172.23.105.221 | kv | Cluster node |
|
| 172.23.105.208 | [u'kv'] | --- OUT ---> |
|
+----------------+----------+--------------+
|
Step 3: Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.225 | kv | Cluster node |
|
| 172.23.105.226 | [u'kv'] | --- OUT ---> |
|
| 172.23.105.220 | kv | Cluster node |
|
| 172.23.105.223 | kv | Cluster node |
|
| 172.23.105.227 | [u'kv'] | --- OUT ---> |
|
| 172.23.105.221 | kv | Cluster node |
|
+----------------+----------+--------------+
|
Step 4: Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.225 | [u'kv'] | --- OUT ---> |
|
| 172.23.105.220 | kv | Cluster node |
|
| 172.23.105.223 | [u'kv'] | --- OUT ---> |
|
| 172.23.105.221 | kv | Cluster node |
|
+----------------+----------+--------------+
|
CORE DUMPS SEEN ON SERVER 172.23.105.220
CORE DUMPS SEEN ON SERVER 172.23.105.221
memcached.log.000093.txt:2020-04-23T10:34:35.921561-07:00 CRITICAL *** Fatal error encountered during exception handling ***
|
memcached.log.000093.txt:2020-04-23T10:34:35.921614-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Checkpoint::expelItem: not found in index Item[0x7f1cbe64c060] with key:cid:0x1:set_vbucket_state
|
memcached.log.000093.txt:2020-04-23T10:34:36.366072-07:00 CRITICAL Breakpad caught a crash (Couchbase version 1006.5.1-1121). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/1a15f0b3-b8c6-dcd6-6b69d78f-182e7352.dmp before terminating.
|
memcached.log.000093.txt:2020-04-23T10:34:36.366110-07:00 CRITICAL Stack backtrace of crashed thread:
|
memcached.log.000093.txt:2020-04-23T10:34:36.366289-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1324cd]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366300-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14a50e]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366309-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14a824]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366316-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f1db85c6000+0xf5d0]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366339-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7f1db81f9000+0x36207]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366358-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7f1db81f9000+0x378f8]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366387-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f1db8cfb000+0x91195]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366396-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x145e02]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366412-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1db8cfb000+0x8ef86]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366429-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1db8cfb000+0x8efd1]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366443-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1db8cfb000+0x8f213]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366453-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x5ab5d]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366460-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x627b4]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366467-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x6d5f4]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366476-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f1db365f000+0x133cc4]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366480-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f1dbaba4000+0x8ee7]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366485-07:00 CRITICAL /lib64/libpthread.so.0() [0x7f1db85c6000+0x7dd5]
|
memcached.log.000093.txt:2020-04-23T10:34:36.366515-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7f1db81f9000+0xfdead]
|
Saw the crashes in couple of tests:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ritesh_magma.ini -p num_items=5000000,GROUP=P0,bucket_storage=magma,bucket_eviction_policy=fullEviction,randomize_value=True,vbuckets=32,rerun=false,enable_dp=False,infra_log_level=debug,log_level=debug -t rebalance_new.rebalance_out.RebalanceOutTests.incremental_rebalance_out_with_ops,nodes_init=7,replicas=1,num_items=50000,value_size=1024,GROUP=P0;non_dgm -m rest'
|
|
|
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/ritesh_magma.ini -p num_items=5000000,GROUP=P0,bucket_storage=magma,bucket_eviction_policy=fullEviction,randomize_value=True,vbuckets=32,rerun=false,enable_dp=False,infra_log_level=debug,log_level=debug -t rebalance_new.rebalance_out.RebalanceOutTests.rebalance_out_with_failover_full_addback_recovery,nodes_init=5,replicas=1,num_items=10000,GROUP=P0;SET1,skip_cleanup=True -m rest'
|