Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Centos 7 64 bit; Couchbase Enterprise Build 7.0.0-3291
-
Untriaged
-
Centos 64-bit
-
-
1
-
Yes
-
KV Sprint 2020-Oct
Description
Script to Repo
./testrunner -i /tmp/durability_volume.ini sdk_client_pool=True,rerun=False,get-cbcollect-info=True,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY |
Steps to Reproduce
1. Create a 3 node cluster
2020-10-04 05:09:50,635 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.105.211 | kv | Cluster node |
| 172.23.105.212 | None | <--- IN — |
| 172.23.105.213 | None | <--- IN — |
-----------------------++-------------
2. Create buckets and initial data load
2020-10-04 05:21:57,029 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-----------------+----------------------------------------------------+----------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-----------------+----------------------------------------------------+----------
| bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 186185152 | 348621350 |
| bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 186140784 | 217369177 |
| default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 473287136 | 396946078 |
-----------------+----------------------------------------------------+----------
3. Rebalance-in with crud on collections in parallel
2020-10-04 05:22:05,269 | test | INFO | pool-1-thread-16 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.105.212 | kv | Cluster node |
| 172.23.105.213 | kv | Cluster node |
| 172.23.105.211 | kv | Cluster node |
| 172.23.105.215 | None | <--- IN — |
| 172.23.105.217 | None | <--- IN — |
-----------------------++-------------
rebalance op fails
Observations
grep WARN memcached.log | grep -v Slow | grep -v "The stream closed early because the conn was disconnected" on .211
2020-10-04T05:25:31.141474-07:00 WARNING (default) CouchKVStore::compactDB openDB error:error opening file, vb:292, fileRev:15
|
2020-10-04T05:25:53.246033-07:00 WARNING (default) VBucket::addStats: Exception caught during getDbFileInfo for vb:0 - what(): CouchKVStore::getDbInfo: failed to open database file for vb:0 rev = 16 with error:error opening file: No such file or directory
|
2020-10-04T05:25:53.247064-07:00 WARNING (default) VBucket::addStats: Exception caught during getDbFileInfo for vb:1 - what(): CouchKVStore::getDbInfo: failed to open database file for vb:1 rev = 16 with error:error opening file: No such file or directory
|
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-
| For Gerrit Dashboard: MB-41857 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 137569,5 | MB-41857: Increase cache limit on close of old db during compaction | master | kv_engine | Status: MERGED | +2 | +1 |
| 137574,3 | MB-41857: Use given ops in openDbForRead | master | kv_engine | Status: MERGED | +2 | +1 |
| 137576,3 | MB-41857: Correct exception error type in getDbInfo | master | kv_engine | Status: MERGED | +2 | +1 |
| 137662,2 | MB-41857: Add debug logging to CouchKVStore::openDbForRead | master | kv_engine | Status: MERGED | +2 | +1 |
| 137663,3 | MB-41857: Don't log CouchKVStoreFileCache resize on every read | master | kv_engine | Status: MERGED | +2 | +1 |
| 137784,2 | MB-41857: Rename OpenResult to OpenForWriteResult | master | kv_engine | Status: MERGED | +2 | +1 |
| 137785,3 | MB-41857: Rename OpenForReadResult to OpenResult | master | kv_engine | Status: MERGED | +2 | +1 |
| 137786,3 | MB-41857: Remove CouchKVStore::openDb | master | kv_engine | Status: ABANDONED | 0 | -1 |
| 137787,5 | MB-41857: Correct test name | master | kv_engine | Status: MERGED | +2 | +1 |
| 137788,7 | MB-41857: Only adjust file cache limit if open db successful | master | kv_engine | Status: MERGED | +2 | +1 |
| 137789,4 | MB-41857: Track the .compact file against the file cache limit | master | kv_engine | Status: ABANDONED | 0 | -1 |
| 137994,2 | MB-41857: Make rollback use openDbForWrite | master | kv_engine | Status: ABANDONED | 0 | -1 |
| 138403,1 | MB-41857: Track file usage in openDb | master | kv_engine | Status: ABANDONED | 0 | -1 |
| 138448,4 | MB-41857: Make rollback use openDb | master | kv_engine | Status: MERGED | +2 | +1 |
| 138452,1 | MB-41857: Make rollback use openDb | master | kv_engine | Status: ABANDONED | -1 | +1 |