Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Cheshire-Cat
-
7.0.0-4578
-
Untriaged
-
Centos 64-bit
-
1
-
Yes
Description
Script to Repro
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.14263.ini GROUP=durability_persist_to_majority,rerun=False,upgrade_version=7.0.0-4578 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority'
|
Steps to Repro
1) Create a 4 node cluster
2021-03-02 04:45:51,967 | test | INFO | pool-4-thread-7 | [table_view:display:72] Rebalance Overview
----------------------------------------------------------------------
Nodes | Services | Version | CPU | Status |
----------------------------------------------------------------------
172.23.121.214 | kv | 7.0.0-4578-enterprise | 1.97648236177 | Cluster node |
172.23.123.99 | None | <--- IN — | ||
172.23.123.96 | None | <--- IN — | ||
172.23.123.97 | None | <--- IN — |
----------------------------------------------------------------------
2) Create buckets/scopes/collections/data.
2021-03-02 04:51:02,661 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-------------------------------------------------------------------------
Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-------------------------------------------------------------------------
bucket1 | couchbase | 2 | none | 0 | 30000 | 419430400 | 112321568 | 307977480 |
bucket2 | couchbase | 2 | none | 0 | 30000 | 419430400 | 112827184 | 194371822 |
default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 417488272 | 419289625 |
-------------------------------------------------------------------------
4)Add 172.23.123.98 and 172.23.121.213 , remove(172.23.123.97 and 172.23.123.96) and start a swap rebalance of 2 nodes.
2021-03-02 04:51:13,318 | test | INFO | pool-4-thread-1 | [table_view:display:72] Rebalance Overview
-----------------------------------------------------------------------
Nodes | Services | Version | CPU | Status |
-----------------------------------------------------------------------
172.23.123.98 | kv | 7.0.0-4578-enterprise | 0.326387145368 | Cluster node |
172.23.123.97 | kv | 7.0.0-4578-enterprise | 7.98383021728 | — OUT ---> |
172.23.121.213 | kv | 7.0.0-4578-enterprise | 0.551240290654 | Cluster node |
172.23.123.96 | kv | 7.0.0-4578-enterprise | 7.07427993936 | — OUT ---> |
172.23.123.99 | kv | 7.0.0-4578-enterprise | 6.18191031163 | Cluster node |
172.23.121.214 | kv | 7.0.0-4578-enterprise | 8.39195979899 | Cluster node |
-----------------------------------------------------------------------
We see ERROR messages like below on 172.23.121.213
2021-03-02 04:55:28,926 | test | INFO | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt
|
2021-03-02 04:55:28,927 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:791] 172.23.121.213: Found ' ERROR ' logs - ['2021-03-02T04:55:17.490262-08:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:118 - Details: CouchKVStore::compactDBInternal finalising dropped collections, container should be emptyEraserContext: removed:true, ScanContext: startSeqno:0, endSeqno:1416 dropped:[0x62, 0x61, 0x63, 0x64, 0x65, 0x5e, 0x5d, 0x5f, 0x60, 0x58, 0x57, 0x59, 0x5a, 0x5b, 0x5c, 0x55, 0x54, 0x23, 0x21, 0x22, 0x27, 0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x1c, 0x1b, 0x1a, 0x19, 0x17, 0x18, 0x1d, 0x1e, 0x1f, 0x20, 0xb, 0xa, 0x9, 0x4f, 0x11, 0xd, 0x2d, 0x8, 0xc, 0x12, 0x10, 0xe, 0xf, 0x16, 0x15, 0x13, 0x14, 0x24, 0x25, 0x26, 0x32, 0x31, 0x30, 0x2e, 0x2f, 0x37, 0x36, 0x35, 0x33, 0x34, 0x3b, 0x3a, 0x38, 0x39, 0x41, 0x40, 0x3f, 0x3e, 0x3c, 0x3d, 0x45, 0x44, 0x42, 0x43, 0x4b, 0x4a, 0x49, 0x48, 0x46, 0x47, 0x50, 0x4e, 0x4c, 0x4d, 0x53, 0x51, 0x52, 0x56, ]\n']
|
cbcollect_info attached. This was not seen on 7.0.0-4554.
Attachments
For Gerrit Dashboard: MB-44694 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
147710,4 | MB-44590: ScanContext startSeqno should not always be 0 | master | kv_engine | Status: MERGED | +2 | +1 |