Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
Cheshire-Cat
-
Centos 7; Couchbase EE 7.0.0-4008
-
Untriaged
-
Centos 64-bit
-
-
1
-
Yes
Description
Script to Repo
./testrunner -i /tmp/durability_volume.ini rerun=False,GROUP=P1_failover_and_rebalance_out -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,skip_validations=False |
Steps to Repro
1. Create a 5 node cluster
2020-12-14 21:59:42,983 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.105.215 | kv | Cluster node |
| 172.23.105.217 | None | <--- IN — |
| 172.23.105.219 | None | <--- IN — |
| 172.23.105.220 | None | <--- IN — |
| 172.23.106.237 | None | <--- IN — |
-----------------------++-------------
2. Create buckets
2020-12-14 22:07:17,236 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-----------------+----------------------------------------------------+----------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-----------------+----------------------------------------------------+----------
| bucket1 | couchbase | 3 | none | 0 | 30000 | 524288000 | 150185280 | 167196233 |
| bucket2 | ephemeral | 3 | none | 0 | 30000 | 524288000 | 97168000 | 170 |
| default | couchbase | 3 | none | 0 | 500000 | 7864320000 | 707622208 | 508271011 |
-----------------+----------------------------------------------------+----------
3. Graceful failover and rebalance out
2020-12-14 22:11:47,124 | test | INFO | pool-1-thread-26 | [table_view:display:72] Rebalance Overview
-----------------------++-------------
| Nodes | Services | Status |
-----------------------++-------------
| 172.23.105.215 | kv | Cluster node |
| 172.23.106.237 | [u'kv'] | — OUT ---> |
| 172.23.105.220 | [u'kv'] | — OUT ---> |
| 172.23.105.217 | kv | Cluster node |
| 172.23.105.219 | kv | Cluster node |
-----------------------++-------------
Observations
on .215
|
2020-12-14 22:12:13,969 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:793] 172.23.105.215: Found 'ERROR' logs - ['2020-12-14T22:11:48.232693-08:00 ERROR 89: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.105.215->ns_1@172.23.105.217:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":301,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.105.215->ns_1@172.23.105.217:default, vb:146, state:in-memory\n', '2020-12-14T22:11:48.232829-08:00 ERROR 89: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.105.215->ns_1@172.23.105.217:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":301,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.105.215->ns_1@172.23.105.217:default, vb:146, state:in-memory\n', '2020-12-14T22:11:48.232890-08:00 ERROR 89: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.105.215->ns_1@172.23.105.217:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype": |
|
|
|
["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":301,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.105.215->ns_1@172.23.105.217:default, vb:146, state:in-memory\n'] |
Attachments
Issue Links
- duplicates
-
-
- Closed
-