Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
6.5.0
-
Triaged
-
Centos 64-bit
-
-
No
-
KV-Engine Mad-Hatter Beta
Description
Steps to reproduce
1. Create a 2 node cluster - Both KV nodes
Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.155 | [u'kv'] | Cluster node |
|
| 172.23.105.159 | None | <--- IN --- |
|
+----------------+----------+--------------+
|
2. Create a default bucket with replica=1
3. Create 50000 docs with durability=Majority
Bucket statistics
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
| Bucket | Type | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
|
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
| default | membase | 1 | 0 | 50000 | 4477419520 | 153456256 | 205281449 |
|
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
4. Rebalance In 2 new nodes and in parallel start another load of new 50k docs with durability=majority.
Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.155 | [u'kv'] | Cluster node |
|
| 172.23.105.159 | [u'kv'] | Cluster node |
|
| 172.23.105.205 | None | <--- IN --- |
|
| 172.23.105.206 | None | <--- IN --- |
|
+----------------+----------+--------------+
|
NOTE: As soon as the rebalance starts we see a lot of com.couchbase.client.core.error.DurabilityImpossibleException for the cruds
5. Rebalance Fails
2019-07-11 05:12:03,252 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.105.155', u'tstamp': 1562847115523L, u'shortText': u'message', u'serverTime': u'2019-07-11T05:11:55.523Z', u'text': u'Rebalance exited with reason bad_replicas.\nRebalance Operation Id = 6ba504bc2511623413681e58d6ed8d7a'}
|
2019-07-11 05:12:03,253 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2620] {u'code': 2, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.105.155', u'tstamp': 1562847115521L, u'shortText': u'message', u'serverTime': u'2019-07-11T05:11:55.521Z', u'text': u"Bad replicators after rebalance:\nMissing = [{'ns_1@172.23.105.206','ns_1@172.23.105.155',256},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',257},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',258},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',259},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',260},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',261},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',262},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',263},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',264},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',265},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',266},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',267},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',268},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',269},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',270},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',271},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',272},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',273},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',274},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',275},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',276},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',277},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',278},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',279},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',280},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',281},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',282},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',283},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',284},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',285},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',286},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',287},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',288},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',289},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',290},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',291},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',292},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',293},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',294},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',295},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',296},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',297},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',298},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',299},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',300},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',301},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',302},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',303},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',304},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',305},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',306},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',307},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',308},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',309},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',310},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',311},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',312},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',313},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',314},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',315},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',316},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',317},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',318},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',319},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',320},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',321},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',322},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',323},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',324},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',325},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',326},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',327},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',328},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',329},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',330},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',331},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',332},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',333},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',334},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',335},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',336},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',337},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',338},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',339},\n {'ns_1@172.23.105.206','ns_1@172.23.105.155',340}]\nExtras = []"}
|
2019-07-11 05:12:03,253 | test | ERROR | pool-2-thread-2 | [rest_client:print_UI_logs:2620] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.105.155', u'tstamp': 1562847094919L, u'shortText': u'message', u'serverTime': u'2019-07-11T05:11:34.919Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'}
|