Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
7.1.0
-
7.1.0-1258
-
Untriaged
-
Centos 64-bit
-
1
-
No
Description
Script to Repro
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.4534.ini GROUP=swap_rebalance_P0_set1,rerun=False,disk_optimized_thread_settings=True,upgrade_version=7.1.0-1258 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=5,nodes_swap=2,bucket_spec=magma_dgm.5_percent_dgm.5_node_2_replica_magma_ttl_256,doc_size=256,randomize_value=True,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=swap_rebalance_P0_set1'
|
Steps to Repro
1. Create a 5 node cluster
2021-09-09 04:40:42,316 | test | INFO | pool-3-thread-8 | [table_view:display:72] Rebalance Overview
-----------------------------------------------------------------------
| Nodes | Services | Version | CPU | Status |
-----------------------------------------------------------------------
| 172.23.107.159 | kv | 7.1.0-1258-enterprise | 0.263322884013 | Cluster node |
| 172.23.107.163 | None | <--- IN — | ||
| 172.23.107.158 | None | <--- IN — | ||
| 172.23.107.161 | None | <--- IN — | ||
| 172.23.107.167 | None | <--- IN — |
-----------------------------------------------------------------------
2. Create buckets/scopes/collections/data
2021-09-09 05:01:35,138 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
----------------------------------------------------------------------------------------------------------
| Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR |
----------------------------------------------------------------------------------------------------------
| bucket1 | couchbase | couchstore | 2 | none | 1200 | 100000 | 9.77 GiB | 338.59 MiB | 408.72 MiB | 100 |
| bucket2 | couchbase | magma | 2 | none | 1200 | 50000 | 4.88 GiB | 513.72 MiB | 610.36 MiB | 100 |
| default | couchbase | magma | 2 | none | 1200 | 33134124 | 1.25 GiB | 1.10 GiB | 20.73 GiB | 0.393326639239 |
----------------------------------------------------------------------------------------------------------
3. Start a swap rebalance of 2 nodes. Nodes in ( 172.23.107.105 and 172.23.107.168) and Nodes out(172.23.107.161 and 172.23.107.167)
2021-09-09 05:01:49,507 | test | INFO | pool-3-thread-22 | [table_view:display:72] Rebalance Overview
-----------------------------------------------------------------------
| Nodes | Services | Version | CPU | Status |
-----------------------------------------------------------------------
| 172.23.107.159 | kv | 7.1.0-1258-enterprise | 12.1614781068 | Cluster node |
| 172.23.107.163 | kv | 7.1.0-1258-enterprise | 9.82199217271 | Cluster node |
| 172.23.107.161 | kv | 7.1.0-1258-enterprise | 10.0403632694 | — OUT ---> |
| 172.23.107.167 | kv | 7.1.0-1258-enterprise | 11.376866616 | — OUT ---> |
| 172.23.107.105 | kv | 7.1.0-1258-enterprise | 0 | Cluster node |
| 172.23.107.158 | kv | 7.1.0-1258-enterprise | 10.8714844243 | Cluster node |
| 172.23.107.168 | kv | 7.1.0-1258-enterprise | 0.537903427571 | Cluster node |
-----------------------------------------------------------------------
grep CRITICAL on 172.23.107.167:
Found ' CRITICAL ' logs - ['memcached<0.129.0>: 2021-09-09T06:35:21.667266-07:00 CRITICAL *** Fatal error encountered during exception handling ***\n', 'memcached<0.129.0>: 2021-09-09T06:35:21.667368-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): KVStore::CompactionContext::maybeUpdatePurgeSeqno(): Unable to get vbucket ptr for vb:157\n', '[ns_server:info,2021-09-09T06:35:22.319-07:00,babysitter_of_ns_1@cb.local:<0.129.0>:ns_port_server:log:221]memcached<0.129.0>: CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1258). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/44c64a25-738c-4a8f-12ca4aa3-1607539d.dmp before terminating.\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377712-07:00 CRITICAL Detected previous crash\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377777-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1258). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/44c64a25-738c-4a8f-12ca4aa3-1607539d.dmp before terminating.\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377788-07:00 CRITICAL Stack backtrace of crashed thread:\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377790-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x6a6dd8]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377791-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x6ff32a]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377793-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x6ff668]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377795-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fe7d6202000+0xf630]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377797-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7fe7d3a34000+0x36387]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377798-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7fe7d3a34000+0x37a78]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377800-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe7d431c000+0x9963c]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377801-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x6b73fb]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377802-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe7d431c000+0xa48f6]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377803-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe7d431c000+0xa4961]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377804-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe7d431c000+0xa4bf4]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377805-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xc4a2d]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377806-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x46277c]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377808-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x5136ed]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377809-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x5bec05]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377811-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x5960c2]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377812-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x56c2c3]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377813-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x56ed1a]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377814-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x56efac]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377815-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x588f28]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377816-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x557e8d]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377817-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x559681]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377819-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x714a09]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377820-07:00 CRITICAL #23 /lib64/libpthread.so.0() [0x7fe7d6202000+0x7ea5]\n', 'memcached<0.1258.0>: 2021-09-09T06:35:22.377822-07:00 CRITICAL #24 /lib64/libc.so.6(clone+0x6d) [0x7fe7d3a34000+0xfe8dd]\n']
|
Minidump of interest :- 44c64a25-738c-4a8f-12ca4aa3-1607539 on 172.23.107.167:
cbcollect_info attached. This is being run for the first time , so we don't have a baseline.
Attachments
Issue Links
- duplicates
-
-
- Closed
-