Details
-
Bug
-
Resolution: Known Error
-
Critical
-
Cheshire-Cat
-
7.0.0-4834-enterprise
-
Untriaged
-
Centos 64-bit
-
1
-
Yes
-
KV-Engine 2021-March
Description
Script to Repro
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.4521.ini GROUP=rebalance_with_collection_crud_dgm,rerun=False,upgrade_version=7.0.0-4834 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=dgm.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,dgm=55,num_items=32000,skip_validations=False,GROUP=rebalance_with_collection_crud_dgm'
|
Steps to Repro
1) Create a 5 node cluster.
2021-04-01 01:05:14,032 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview
-----------------------------------------------------------------------
| Nodes | Services | Version | CPU | Status |
-----------------------------------------------------------------------
| 172.23.106.13 | kv | 7.0.0-4834-enterprise | 0.376695128076 | Cluster node |
| 172.23.106.16 | None | <--- IN — | ||
| 172.23.106.15 | None | <--- IN — | ||
| 172.23.106.11 | None | <--- IN — | ||
| 172.23.104.251 | None | <--- IN — |
-----------------------------------------------------------------------
2) Create bucket/scope/collections.data
2021-04-01 01:07:43,092 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
----------------------------------------------------------------------------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
----------------------------------------------------------------------------
| default | couchbase | 2 | none | 0 | 6400000 | 4194304000 | 2390737096 | 4158093834 |
----------------------------------------------------------------------------
3) Push the bucket to DGM
2021-04-01 01:07:51,966 | test | INFO | pool-3-thread-4 | [task:_load_bucket_into_dgm:2082] DGM doc loading for 'default' to atleast 55%
|
4) Rebalance out 2 nodes (172.23.106.11 and 172.23.104.251)
2021-04-01 01:07:55,461 | test | INFO | pool-3-thread-5 | [table_view:display:72] Rebalance Overview
----------------------------------------------------------------------
| Nodes | Services | Version | CPU | Status |
----------------------------------------------------------------------
| 172.23.106.16 | kv | 7.0.0-4834-enterprise | 9.55882352941 | Cluster node |
| 172.23.106.13 | kv | 7.0.0-4834-enterprise | 9.50695322377 | Cluster node |
| 172.23.106.15 | kv | 7.0.0-4834-enterprise | 6.45079686314 | Cluster node |
| 172.23.106.11 | kv | 7.0.0-4834-enterprise | 5.90759909114 | — OUT ---> |
| 172.23.104.251 | kv | 7.0.0-4834-enterprise | 6.08040201005 | — OUT ---> |
----------------------------------------------------------------------
Following error messages are seen on 172.23.106.15
[basetestcase:check_coredump_exist:795] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt
|
2021-04-01 01:09:46,092 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:797] 172.23.106.15: Found ' ERROR ' logs - ['2021-04-01T01:08:00.924756-07:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:241 - Details: cb::couchstore::replay() - precommit hook Failed: error reading file\n', '2021-04-01T01:08:01.002835-07:00 ERROR (default) CouchKVStore::compactDB: exception while performing compaction for vb:245 - Details: cb::couchstore::replay() - precommit hook Failed: error reading file\n']
|
cbcollect_info attached. This was not seen on 7.0.0-4797.