Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Centos 7; Couchbase EE 7.0.0-3800
-
Untriaged
-
Centos 64-bit
-
-
1
-
Yes
Description
/testrunner -i /tmp/durability_volume.ini rerun=False,get-cbcollect-info=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_rebalance |
Steps to Reproduce
1. Create a 4 node cluster
2020-11-19 20:57:35,454 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview
------------------------------------
| Nodes | Services | Status |
------------------------------------
| 172.23.120.137 | kv | Cluster node |
| 172.23.120.138 | None | <--- IN — |
| 172.23.120.139 | None | <--- IN — |
| 172.23.120.140 | None | <--- IN — |
------------------------------------
2. Create buckets, collections, and load initial data
2020-11-19 21:04:16,400 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
-------------------------------------------------------------------------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
-------------------------------------------------------------------------
| bucket1 | couchbase | 3 | none | 0 | 30000 | 419430400 | 140125424 | 177848907 |
| bucket2 | ephemeral | 3 | none | 0 | 30000 | 419430400 | 98180608 | 136 |
| default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 715935408 | 533339876 |
-------------------------------------------------------------------------
3. Swap rebalance with CRUD on data (.142,.147-> IN ; .139,.140-> OUT)
2020-11-19 21:04:24,628 | test | INFO | pool-1-thread-15 | [table_view:display:72] Rebalance Overview
------------------------------------
| Nodes | Services | Status |
------------------------------------
| 172.23.120.142 | kv | Cluster node |
| 172.23.120.138 | kv | Cluster node |
| 172.23.120.139 | [u'kv'] | — OUT ---> |
| 172.23.120.147 | kv | Cluster node |
| 172.23.120.137 | kv | Cluster node |
| 172.23.120.140 | [u'kv'] | — OUT ---> |
------------------------------------
This rebalance fails
Observations
on .137
grep WARN memcached.log | grep -v Slow | grep -v "The stream closed early because the conn was disconnected"
2020-11-19T20:32:39.848985-08:00 WARNING 54: (bucket2) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.137->ns_1@172.23.120.138:bucket2 - (vb:614) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 0. Client requested seqnos:{19,18446744073709551615} snapshot:{19,19} uuid:104078053479004 |
2020-11-19T20:32:39.851080-08:00 WARNING 56: (bucket2) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.137->ns_1@172.23.120.142:bucket2 - (vb:626) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 0. Client requested seqnos:{25,18446744073709551615} snapshot:{25,25} uuid:152408331017118 |
2020-11-19T20:32:39.851123-08:00 WARNING 56: (bucket2) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.137->ns_1@172.23.120.142:bucket2 - (vb:627) Stream request requires rollback to seqno:0 because consumer ahead of producer - producer upper at 0. Client requested seqnos:{25,18446744073709551615} snapshot:{25,25} uuid:60127240046347 |
Attachments
Issue Links
- relates to
-
-
- Closed
-