Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
Unknown
Description
1. Create a 7 node cluster:
+---------------+-----------------+--------------+
|
| Nodes | Services | Status |
|
+---------------+-----------------+--------------+
|
| 172.23.106.44 | index, kv, n1ql | Cluster node |
|
| 172.23.106.45 | None | <--- IN --- |
|
| 172.23.106.47 | None | <--- IN --- |
|
| 172.23.106.48 | None | <--- IN --- |
|
| 172.23.106.49 | None | <--- IN --- |
|
| 172.23.106.50 | None | <--- IN --- |
|
| 172.23.106.51 | None | <--- IN --- |
|
+---------------+-----------------+--------------+
|
2. Create a default bucket and load 1M items in the bucket
2020-01-05 22:54:39,520 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
|
+---------+---------+----------+-----+---------+-------------+------------+------------+
|
| Bucket | Type | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
|
+---------+---------+----------+-----+---------+-------------+------------+------------+
|
| default | membase | 2 | 0 | 1000000 | 10452205568 | 1321594488 | 3443407329 |
|
+---------+---------+----------+-----+---------+-------------+------------+------------+
|
3. Rebalance out 5 nodes in one shot:
+---------------+-----------------+--------------+
|
| Nodes | Services | Status |
|
+---------------+-----------------+--------------+
|
| 172.23.106.44 | index, kv, n1ql | Cluster node |
|
| 172.23.106.47 | [u'kv'] | --- OUT ---> |
|
| 172.23.106.50 | [u'kv'] | --- OUT ---> |
|
| 172.23.106.49 | [u'kv'] | --- OUT ---> |
|
| 172.23.106.48 | [u'kv'] | --- OUT ---> |
|
| 172.23.106.45 | kv | Cluster node |
|
| 172.23.106.51 | [u'kv'] | --- OUT ---> |
|
+---------------+-----------------+--------------+
|
4. Magma Crashed:
memcached.log.000026.txt:2020-01-06T06:55:09.338218+00:00 CRITICAL Stack backtrace of crashed thread:
|
memcached.log.000026.txt:2020-01-06T06:55:09.338391+00:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1324cd]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338403+00:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14a50e]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338412+00:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14a824]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338420+00:00 CRITICAL /lib64/libpthread.so.0() [0x7fef62c11000+0xf5d0]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338443+00:00 CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7fef62844000+0x36207]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338463+00:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7fef62844000+0x378f8]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338528+00:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fef63346000+0x91195]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338539+00:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x145e02]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338570+00:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fef63346000+0x8ef86]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338586+00:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fef63346000+0x8efd1]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338600+00:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fef63346000+0x8f213]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338606+00:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fef5d90c000+0x5e879]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338612+00:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fef5d90c000+0xc0601]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338617+00:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fef5d90c000+0xac1bd]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338622+00:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fef5d90c000+0xb47c8]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338625+00:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fef5d90c000+0xb6008]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338630+00:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libmagma.so() [0x7fef5d90c000+0x525cf]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338635+00:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fef651ef000+0x8ee7]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338641+00:00 CRITICAL /lib64/libpthread.so.0() [0x7fef62c11000+0x7dd5]
|
memcached.log.000026.txt:2020-01-06T06:55:09.338671+00:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fef62844000+0xfdead]
|
memcached.log.000027.txt:2020-01-06T06:55:18.738884+00:00 CRITICAL WALIterator corruption error:Corruption: 3:22351872
|
memcached.log.000027.txt:2020-01-06T06:55:19.038232+00:00 CRITICAL WALIterator corruption error:Corruption: 3:8888320
|
memcached.log.000027.txt:2020-01-06T06:55:19.355701+00:00 CRITICAL WALIterator corruption error:Corruption: 3:11124736
|
memcached.log.000027.txt:2020-01-06T06:55:19.632785+00:00 CRITICAL WALIterator corruption error:Corruption: 3:10862592
|
memcached.log.000027.txt:2020-01-06T06:55:19.729686+00:00 CRITICAL *** Fatal error encountered during exception handling ***
|
memcached.log.000027.txt:2020-01-06T06:55:19.729743+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): InsertTables: Tree Snapshot:
|
memcached.log.000027.txt:2020-01-06T06:55:19.755107+00:00 CRITICAL *** Fatal error encountered during exception handling ***
|
memcached.log.000027.txt:2020-01-06T06:55:19.759323+00:00 CRITICAL *** Fatal error encountered during exception handling ***
|
memcached.log.000027.txt:2020-01-06T06:55:19.893053+00:00 CRITICAL Breakpad caught a crash (Couchbase version 1006.5.1-1014). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/44e6ea43-8048-148e-18ebdb71-19ee4f46.dmp before terminating.
|