Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Centos 7 asan; Couchbase EE 7.0.0 3816
-
Triaged
-
Centos 64-bit
-
-
1
-
Yes
Description
Script to Repo:
./testrunner -i /tmp/durability_volume.ini rerun=False,get-cbcollect-info=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,GROUP=rebalance_with_collection_crud_durability_MAJORITY |
Steps to Reproduce
1. Create 5 node cluster
2020-11-21 00:42:10,214 | test | INFO | pool-11-thread-7 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.120.137 | kv | Cluster node |
| 172.23.120.138 | None | <--- IN — |
| 172.23.120.139 | None | <--- IN — |
| 172.23.120.140 | None | <--- IN — |
| 172.23.120.142 | None | <--- IN — |
-----------------------++-------------
2. Create buckets and initial data load
2020-11-21 00:53:21,908 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-----------------+-----------------------------------------------------+----------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-----------------+-----------------------------------------------------+----------
| bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 16506673 | 241853540 |
| bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 133167122 | 170 |
| default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 355172604 | 745223956 |
-----------------+-----------------------------------------------------+----------
3. Rebalance out with CRUD on collections
2020-11-21 00:53:26,644 | test | INFO | pool-11-thread-2 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.120.142 | [u'kv'] | — OUT ---> |
| 172.23.120.138 | kv | Cluster node |
| 172.23.120.139 | kv | Cluster node |
| 172.23.120.137 | kv | Cluster node |
| 172.23.120.140 | [u'kv'] | — OUT ---> |
-----------------------++-------------
Rebalance fails
Observations
On .137
grep WARN ns_server.babysitter.log | grep -v Slow | grep -v "The stream closed early because the conn was disconnected"
818-08:00 WARNING (bucket1) CouchKVStore::rollback: couchstore_rewind_db_header error:this db handle could have its file reopened and must be closed [Success], vb:254, latestSeqno:425, rollbackSeqno:420 |
memcached<0.1474.0>: 2020-11-21T00:56:32.224998-08:00 WARNING (bucket1) CouchKVStore::rollback: couchstore_rewind_db_header error:this db handle could have its file reopened and must be closed [Success], vb:258, latestSeqno:425, rollbackSeqno:420 |
memcached<0.1474.0>: 2020-11-21T00:56:32.225828-08:00 WARNING (bucket1) CouchKVStore::rollback: couchstore_rewind_db_header error:this db handle could have its file reopened and must be closed [Success], vb:308, latestSeqno:425, rollbackSeqno:422 |
memcached<0.1474.0>: 2020-11-21T00:56:32.241266-08:00 WARNING (bucket1) CouchKVStore::rollback: couchstore_rewind_db_header error:this db handle could have its file reopened and must be closed [Success], vb:262, latestSeqno:425, rollbackSeqno:420 |
[ns_server:info,2020-11-21T00:56:37.213-08:00,babysitter_of_ns_1@cb.local:<0.1474.0>:ns_port_server:log:224]memcached<0.1474.0>: 2020-11-21T00:56:37.008089-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:76 - Details: ThrowExceptionUnderflowPolicy current:534 arg:537 |
[ns_server:info,2020-11-21T00:56:37.506-08:00,babysitter_of_ns_1@cb.local:<0.1474.0>:ns_port_server:log:224]memcached<0.1474.0>: 2020-11-21T00:56:37.306260-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:74 - Details: ThrowExceptionUnderflowPolicy current:355 arg:358 |
memcached<0.1474.0>: 2020-11-21T00:56:37.315167-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:217 - Details: ThrowExceptionUnderflowPolicy current:360 arg:363 |
memcached<0.1474.0>: 2020-11-21T00:56:37.326651-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:211 - Details: ThrowExceptionUnderflowPolicy current:540 arg:543 |
memcached<0.1474.0>: 2020-11-21T00:56:37.455689-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:88 - Details: ThrowExceptionUnderflowPolicy current:723 arg:726 |
memcached<0.1474.0>: 2020-11-21T00:56:37.497595-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:96 - Details: ThrowExceptionUnderflowPolicy current:540 arg:543 |
[ns_server:info,2020-11-21T00:56:37.871-08:00,babysitter_of_ns_1@cb.local:<0.1474.0>:ns_port_server:log:224]memcached<0.1474.0>: 2020-11-21T00:56:37.671217-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:94 - Details: ThrowExceptionUnderflowPolicy current:534 arg:537 |
[ns_server:info,2020-11-21T00:56:39.624-08:00,babysitter_of_ns_1@cb.local:<0.1474.0>:ns_port_server:log:224]memcached<0.1474.0>: 2020-11-21T00:56:39.423858-08:00 WARNING (default) CouchKVStore::compactDB: exception while performing compaction for vb:261 - Details: ThrowExceptionUnderflowPolicy current:539 arg:542 |
Attachments
Issue Links
- relates to
-
-
- Closed
-